Summer Milestone 10 Reputation System outline and review

From crowdresearch
Jump to: navigation, search

Welcome to the wiki for the Daemo Reputation System!

Factors affecting the reputation system

For workers:

  1. length of time on the platform
  2. number of tasks completed
  3. content analysis
  4. communication
  5. skills

For requesters:

  1. number of tasks posted
  2. length of time on the platform
  3. feedback
  4. clarity of the task posted
  5. communication
  6. amount he/she pays compared with other requesters ???

Top 10 ideas for requesters identified by Spring participants

Which feature will best support a fair and effective reputation system?(3r) Ideas

Score (0 - 100)

Rate worker on your desire to work with them on future projects. 90 Workers involved in work evaluation and work dispute moderation get higher reputation and/or pay. 86 Guilds: more senior workers with same skills review your work periodically and promote you (e.g., Level 5 C++ programmer) 85 Badges for high-performing and poor-performing workers. 83 More senior experts in your same area review your work before sending it back to requesters; their judgments determine your reputation. 80 Random sampling of worker output to determine promotion. 75 Traditional approach: workers and requesters can rate each other on a five-star scale after work is completed 73 Incentive to rate honestly: if I give a high rating, the system makes it likely I get that person in the future. Low ratings → less likely. 73 In addition to level based on skill, Reputation also includes levels for types of tasks, content and knowledge domains. 71 Guilds of experts and levels within each guild (e.g., Level 3 C++). When you want to be promoted, you apply and the guild reviews your work. 71

Types of reputation systems

  1. content-driven and endorsement networks
  2. reputation tied to various skills - provide an aggregate rating for all of the task completed (for both workers and requesters) and also a breakdown of reputation for each skill
  3. star ratings 1-5
  4. worker ratings of requesters (and vice versa) place them into A, B (good), C (fair), D (poor)

Elements of reputation systems

  1. ongoing vs end-of-task feedback/ratings
  2. social network referrals rather than star ratings
  3. based on skills rather than everything bundled into one rating?
  4. video verification

Summer Milestone 10 Reputation Paper Summaries

  1. Outline of the Trello Milestone task, DRIs, what to do, Reputation wiki link and google doc to keep track of who is doing what [1]
  2. Paper Summary Wiki- summary, pros and cons for all reputation-related research papers. [2]

Work previously completed by research project participants

  1. Link to all the documents with the word "Reputation" in the Crowd Research Wikis [3]
  2. AllOurIdeas Fair & Effective Reputation System: Voting Results (Workers) [4]
  3. AllOurIdeas Fair & Effective Reputation System: Voting Results (Researchers) [5]
  4. AllOurIdeas Fair & Effective Reputation System: Voting Results (Requesters) [6]
  5. Reputation options Google Doc (140-character ideas: AllOurIdeas basis) [7]
  6. Reputation system ideas generation document from Spring session document They used this to have 140 character ideas which were added on


  1. Relevant Work Wiki [8]
  2. All Crowd Research Reputation resources listed on main resource page at [9]