Summer Milestone 10 Reputation System outline and review

From crowdresearch
Jump to: navigation, search

Welcome to the wiki for the Daemo Reputation System!


Factors affecting the reputation system

For workers:

  1. length of time on the platform
  2. number of tasks completed
  3. number of re-hires
  4. content analysis
  5. communication
  6. skills
  7. evidence of qualifications such as certificates
  8. evidence of skill level such as an internal testing service and badges to identify mastery of particular skills
  9. timeliness
  10. linked reputation evidence from other networks such as github, LinkedIn, etc.
  11. ethics

For requesters:

  1. number of tasks posted
  2. length of time on the platform
  3. feedback
  4. clarity of the task posted
  5. clarity of expectations
  6. communication
  7. amount he/she pays compared with other requesters ???
  8. accuracy
  9. consistency
  10. ethics

Types of reputation systems

  1. content-driven and endorsement networks
  2. reputation tied to various skills - provide an aggregate rating for all of the task completed (for both workers and requesters) and also a breakdown of reputation for each skill
  3. star ratings 1-5
  4. worker ratings of requesters (and vice versa) place them into A, B (good), C (fair), D (poor)

Elements of reputation systems

  1. ongoing vs end-of-task feedback/ratings
  2. social network referrals rather than star ratings
  3. based on skills rather than everything bundled into one rating?
  4. video verification


Ideas to address the cold start problem (new workers with no reputation)

  1. Workers can do work/tasks for NGOs and charities to earn their first ratings (Charities post what they need to be done, and workers could choose work/tasks from their favorite charities).
  2. Provide a supervised incubator space for newbies to work on tasks with other newbies under the guidance of workers with expert skills in order to gain their first reputation star ratings (if we end up using star ratings)

Summer Milestone 10 Reputation Paper Summaries

  1. Outline of the Trello Milestone task, DRIs, what to do, Reputation wiki link and google doc to keep track of who is doing what [1]
  2. Paper Summary Wiki- summary, pros and cons for all reputation-related research papers. [2]
  3. Evaluations of other crowdsourcing platform reputation systems [3]

Work previously completed by research project participants

  1. Link to all the documents with the word "Reputation" in the Crowd Research Wikis [4]
  2. Paper that Neil wrote in the Spring on the Reputation System [5]
  3. AllOurIdeas Fair & Effective Reputation System: Voting Results (Workers) [6]
  4. AllOurIdeas Fair & Effective Reputation System: Voting Results (Researchers) [7]
  5. AllOurIdeas Fair & Effective Reputation System: Voting Results (Requesters) [8]
  6. Reputation system options Google Doc (140-character ideas: AllOurIdeas basis) [9]
  7. AllOurIdeas Combined Voting Results (spreadsheet with bar chart - Draft version) [10]

Resources

  1. Relevant Work Wiki [11]
  2. All Crowd Research Reputation resources listed on main resource page at [12]

Top 10 ideas for requesters on http://allourideas.org/reputationsystem3r/results?all=true identified by Spring participants

172 votes on 42 ideas

Which feature will best support a fair and effective reputation system?(3r)

Score (0 - 100)

90 Rate worker on your desire to work with them on future projects.

86 Workers involved in work evaluation and work dispute moderation get higher reputation and/or pay.

85 Guilds: more senior workers with same skills review your work periodically and promote you (e.g., Level 5 C++ programmer)

83 Badges for high-performing and poor-performing workers.

80 More senior experts in your same area review your work before sending it back to requesters; their judgments determine your reputation.

75 Random sampling of worker output to determine promotion.

73 Traditional approach: workers and requesters can rate each other on a five-star scale after work is completed

73 Incentive to rate honestly: if I give a high rating, the system makes it likely I get that person in the future. Low ratings → less likely.

71 In addition to level based on skill, Reputation also includes levels for types of tasks, content and knowledge domains.

71 Guilds of experts and levels within each guild (e.g., Level 3 C++). When you want to be promoted, you apply and the guild reviews your work.

Top 10 ideas from identified by researchers on http://allourideas.org/reputationsystem1r by Spring participants

236 votes on 43 ideas

Which feature will best support a fair and effective reputation system? (1R)

Ideas

Score (0 - 100)

90 Reputation based on public+private/anonymous client/worker feedback. The system would create a Rep. Score tied to privileges/incentives

81 Offer workers option to include professional online profiles such as GitHub repositories or LinkedIn entries

75 In addition to level based on skill, Reputation also includes levels for types of tasks, content and knowledge domains.

73 Five level Social Rank worker+client Public Leaderboard: Hall of Fame to Hall of Shame. Reputation tied to privileges, fair pay & work quality

71 Reputation levels (e.g.,Level 3 C++). Incentive to rate honestly: require % of task cost to bet if they'll hit a level (e.g.,Level 5) later

70 Badges for high-performing and poor-performing workers.

69 Worker rating/profile including job ratings (star ratings: public/human & private/anon/algo feedback), forum posting, and dispute moderation

69 Incentive to rate honestly: if I give a high rating, the system makes it likely I get that person in the future. Low ratings → less likely.

68 Guilds: more senior workers with same skills review your work periodically and promote you (e.g., Level 5 C++ programmer)

67 Requester should also be evaluated on their participation on the platform like dispute solving, rating workers, fellow requester, etc

Top 10 workers' preferred feedback from http://allourideas.org/reputationsystem2w/results by spring participants

3318 votes on 49 ideas

75 Indicate when a worker/requester has joined the system: the calendar date or the number of days,weeks,months, and/or years since inception.

68 Workers get promoted with good work reviews.

65 Traditional approach: workers and requesters can rate each other on a five-star scale after work is completed

63 Provide ability to test up to various levels of a skill set. (i.e. Level 5 photoshop, Level 2 Python)

62 Reputation based on job rating(5stars), private feedback, and platform participation (work evaluation, dispute moderation,forum interaction)

62 In addition to level based on skill, reputation also includes levels by types of tasks, content and knowledge domains.

60 Rate worker on your desire to work with them on future projects.

59 Rate requester on your desire to work with them on future projects

59 Requester rating/profile criteria: accuracy/consistency, communication skills, ethics, hiring rate, cooperation, public/private feedback

59 Provide public access to ratings and reviews for both workers and requesters

Types of reputation systems

  1. content-driven and endorsement networks
  2. reputation tied to various skills - provide an aggregate rating for all of the task completed (for both workers and requesters) and also a breakdown of reputation for each skill
  3. star ratings 1-5
  4. worker ratings of requesters (and vice versa) place them into A, B (good), C (fair), D (poor)

Elements of reputation systems

  1. ongoing vs end-of-task feedback/ratings
  2. social network referrals rather than star ratings
  3. based on skills rather than everything bundled into one rating?
  4. video verification


Retrieved from "http://crowdresearch.stanford.edu/w/index.php?title=Summer_Milestone_10_Reputation_System_outline_and_review&oldid=11235"