Summer Milestone 10 Reputation System outline and review
Welcome to the wiki for the Daemo Reputation System!
Contents
- 1 Factors affecting the reputation system
- 2 Types of reputation systems
- 3 Elements of reputation systems
- 4 Ideas to address the cold start problem (new workers with no reputation)
- 5 Summer Milestone 10 Reputation Paper Summaries
- 6 Work previously completed by research project participants
- 7 Resources
- 8 Top 10 ideas for requesters on http://allourideas.org/reputationsystem3r/results?all=true identified by Spring participants
- 9 Top 10 ideas from identified by researchers on http://allourideas.org/reputationsystem1r by Spring participants
- 10 Top 10 workers' preferred feedback from http://allourideas.org/reputationsystem2w/results by spring participants
- 11 Types of reputation systems
- 12 Elements of reputation systems
Factors affecting the reputation system
For workers:
- length of time on the platform
- number of tasks completed
- number of re-hires
- content analysis
- communication
- skills
- evidence of qualifications such as certificates
- evidence of skill level such as an internal testing service and badges to identify mastery of particular skills
- timeliness
- linked reputation evidence from other networks such as github, LinkedIn, etc.
- ethics
For requesters:
- number of tasks posted
- length of time on the platform
- feedback
- clarity of the task posted
- clarity of expectations
- communication
- amount he/she pays compared with other requesters ???
- accuracy
- consistency
- ethics
Types of reputation systems
- content-driven and endorsement networks
- reputation tied to various skills - provide an aggregate rating for all of the task completed (for both workers and requesters) and also a breakdown of reputation for each skill
- star ratings 1-5
- worker ratings of requesters (and vice versa) place them into A, B (good), C (fair), D (poor)
Elements of reputation systems
- ongoing vs end-of-task feedback/ratings
- social network referrals rather than star ratings
- based on skills rather than everything bundled into one rating?
- video verification
Ideas to address the cold start problem (new workers with no reputation)
- Workers can do work/tasks for NGOs and charities to earn their first ratings (Charities post what they need to be done, and workers could choose work/tasks from their favorite charities).
- Provide a supervised incubator space for newbies to work on tasks with other newbies under the guidance of workers with expert skills in order to gain their first reputation star ratings (if we end up using star ratings)
Summer Milestone 10 Reputation Paper Summaries
- Outline of the Trello Milestone task, DRIs, what to do, Reputation wiki link and google doc to keep track of who is doing what [1]
- Paper Summary Wiki- summary, pros and cons for all reputation-related research papers. [2]
- Evaluations of other crowdsourcing platform reputation systems [3]
Work previously completed by research project participants
- Link to all the documents with the word "Reputation" in the Crowd Research Wikis [4]
- Paper that Neil wrote in the Spring on the Reputation System [5]
- AllOurIdeas Fair & Effective Reputation System: Voting Results (Workers) [6]
- AllOurIdeas Fair & Effective Reputation System: Voting Results (Researchers) [7]
- AllOurIdeas Fair & Effective Reputation System: Voting Results (Requesters) [8]
- Reputation system options Google Doc (140-character ideas: AllOurIdeas basis) [9]
- AllOurIdeas Combined Voting Results (spreadsheet with bar chart - Draft version) [10]
Resources
- Relevant Work Wiki [11]
- All Crowd Research Reputation resources listed on main resource page at [12]
Top 10 ideas for requesters on http://allourideas.org/reputationsystem3r/results?all=true identified by Spring participants
172 votes on 42 ideas
Which feature will best support a fair and effective reputation system?(3r)
Score (0 - 100)
90 Rate worker on your desire to work with them on future projects.
86 Workers involved in work evaluation and work dispute moderation get higher reputation and/or pay.
85 Guilds: more senior workers with same skills review your work periodically and promote you (e.g., Level 5 C++ programmer)
83 Badges for high-performing and poor-performing workers.
80 More senior experts in your same area review your work before sending it back to requesters; their judgments determine your reputation.
75 Random sampling of worker output to determine promotion.
73 Traditional approach: workers and requesters can rate each other on a five-star scale after work is completed
73 Incentive to rate honestly: if I give a high rating, the system makes it likely I get that person in the future. Low ratings → less likely.
71 In addition to level based on skill, Reputation also includes levels for types of tasks, content and knowledge domains.
71 Guilds of experts and levels within each guild (e.g., Level 3 C++). When you want to be promoted, you apply and the guild reviews your work.
Top 10 ideas from identified by researchers on http://allourideas.org/reputationsystem1r by Spring participants
236 votes on 43 ideas
Which feature will best support a fair and effective reputation system? (1R)
Ideas
Score (0 - 100)
90 Reputation based on public+private/anonymous client/worker feedback. The system would create a Rep. Score tied to privileges/incentives
81 Offer workers option to include professional online profiles such as GitHub repositories or LinkedIn entries
75 In addition to level based on skill, Reputation also includes levels for types of tasks, content and knowledge domains.
73 Five level Social Rank worker+client Public Leaderboard: Hall of Fame to Hall of Shame. Reputation tied to privileges, fair pay & work quality
71 Reputation levels (e.g.,Level 3 C++). Incentive to rate honestly: require % of task cost to bet if they'll hit a level (e.g.,Level 5) later
70 Badges for high-performing and poor-performing workers.
69 Worker rating/profile including job ratings (star ratings: public/human & private/anon/algo feedback), forum posting, and dispute moderation
69 Incentive to rate honestly: if I give a high rating, the system makes it likely I get that person in the future. Low ratings → less likely.
68 Guilds: more senior workers with same skills review your work periodically and promote you (e.g., Level 5 C++ programmer)
67 Requester should also be evaluated on their participation on the platform like dispute solving, rating workers, fellow requester, etc
Top 10 workers' preferred feedback from http://allourideas.org/reputationsystem2w/results by spring participants
3318 votes on 49 ideas
75 Indicate when a worker/requester has joined the system: the calendar date or the number of days,weeks,months, and/or years since inception.
68 Workers get promoted with good work reviews.
65 Traditional approach: workers and requesters can rate each other on a five-star scale after work is completed
63 Provide ability to test up to various levels of a skill set. (i.e. Level 5 photoshop, Level 2 Python)
62 Reputation based on job rating(5stars), private feedback, and platform participation (work evaluation, dispute moderation,forum interaction)
62 In addition to level based on skill, reputation also includes levels by types of tasks, content and knowledge domains.
60 Rate worker on your desire to work with them on future projects.
59 Rate requester on your desire to work with them on future projects
59 Requester rating/profile criteria: accuracy/consistency, communication skills, ethics, hiring rate, cooperation, public/private feedback
59 Provide public access to ratings and reviews for both workers and requesters
Types of reputation systems
- content-driven and endorsement networks
- reputation tied to various skills - provide an aggregate rating for all of the task completed (for both workers and requesters) and also a breakdown of reputation for each skill
- star ratings 1-5
- worker ratings of requesters (and vice versa) place them into A, B (good), C (fair), D (poor)
Elements of reputation systems
- ongoing vs end-of-task feedback/ratings
- social network referrals rather than star ratings
- based on skills rather than everything bundled into one rating?
- video verification