Difference between revisions of "Summer Milestone 10 Reputation System outline and review"
From crowdresearch
| (31 intermediate revisions by 3 users not shown) | |||
| Line 6: | Line 6: | ||
#length of time on the platform | #length of time on the platform | ||
#number of tasks completed | #number of tasks completed | ||
| + | #number of re-hires | ||
#content analysis | #content analysis | ||
#communication | #communication | ||
#skills | #skills | ||
| + | #evidence of qualifications such as certificates | ||
| + | #evidence of skill level such as an internal testing service and badges to identify mastery of particular skills | ||
| + | #timeliness | ||
| + | #linked reputation evidence from other networks such as Github, LinkedIn, etc. | ||
| + | #ethics | ||
'''For requesters:''' | '''For requesters:''' | ||
| Line 15: | Line 21: | ||
#feedback | #feedback | ||
#clarity of the task posted | #clarity of the task posted | ||
| + | #clarity of expectations | ||
#communication | #communication | ||
#amount he/she pays compared with other requesters ??? | #amount he/she pays compared with other requesters ??? | ||
| + | #accuracy | ||
| + | #consistency | ||
| + | #ethics | ||
| + | == Types of reputation systems == | ||
| + | #content-driven and endorsement networks | ||
| + | #reputation tied to various skills - provide an aggregate rating for all of the task completed (for both workers and requesters) and also a breakdown of reputation for each skill | ||
| + | #star ratings 1-5 | ||
| + | #worker ratings of requesters (and vice versa) place them into A, B (good), C (fair), D (poor) | ||
| + | |||
| + | == Elements of reputation systems == | ||
| + | #ongoing vs end-of-task feedback/ratings | ||
| + | #social network referrals rather than star ratings | ||
| + | #based on skills rather than everything bundled into one rating? A question has been asked saying "How do you do skills-based ratings, etc., without hindering tasks with a requirement to categorize them?" | ||
| + | #video verification | ||
| + | #external quality ratings | ||
| + | #Metaphor of credit ratings: rather than just people rating each other, have an (external?) authority or algorithm responsible for credit ratings (A, B, C, etc.)
| ||
| + | #Group review of worker to be accepted onto the platform? Issues to address such as payment for reviews and who would do the reviews? | ||
| + | #Accept only highly skilled workers onto the platform? Interviews to onboard newbies? | ||
| + | |||
| + | == Ideas to address the cold start problem (new workers with no reputation) == | ||
| + | #Workers can do work/tasks for NGOs and charities to earn their first ratings (Charities post what they need to be done, and workers could choose work/tasks from their favorite charities). | ||
| + | #Provide a supervised incubator space for newbies to work on tasks with other newbies under the guidance of workers with expert skills in order to gain their first reputation star ratings (if we end up using star ratings) | ||
| + | |||
| + | == Summer Milestone 10 Reputation Paper Summaries == | ||
| + | #Outline of the Trello Milestone task, DRIs, what to do, Reputation wiki link and Google doc to keep track of who is doing what [https://docs.google.com/presentation/d/1A2WxfCNJn2Uu0_Y5un9eZddSGR6O1-m8s9rxkJosT-Y/edit#slide=id.p] | ||
| + | #'''Paper Summary Wiki'''- summary, pros and cons for all reputation-related research papers. [http://crowdresearch.stanford.edu/w/index.php?title=Summer_Milestone_9_Reputation_Systems_research_and_exploration] | ||
| + | #'''Evaluations of other crowdsourcing platform reputation systems''' [http://crowdresearch.stanford.edu/w/index.php?title=Summer_Milestone_9_Evaluations_of_reputation_systems_on_other_crowdsourcing_platforms#] | ||
| + | |||
| + | == Summer Milestone 12 Reputation == | ||
| + | Summary of Reputation Ideas [http://crowdresearch.stanford.edu/w/index.php?title=Reputation_System_Ideas_Summary] | ||
| + | |||
| + | == Work previously completed by research project participants == | ||
| + | #Link to all the documents with the word "Reputation" in the Crowd Research Wikis [http://crowdresearch.stanford.edu/w/index.php?search=reputation&title=Special%3ASearch&go=Go] | ||
| + | #Paper that Neil wrote in the Spring on the Reputation System [http://crowdresearch.stanford.edu/w/index.php?title=Milestone_7_TuringMachine] | ||
| + | #AllOurIdeas Fair & Effective Reputation System: Voting Results (Workers) [http://allourideas.org/reputationsystem2w/results?all=true] | ||
| + | #AllOurIdeas Fair & Effective Reputation System: Voting Results (Researchers) [http://allourideas.org/reputationsystem1r/results?all=true] | ||
| + | #AllOurIdeas Fair & Effective Reputation System: Voting Results (Requesters) [http://allourideas.org/reputationsystem3r/results?all=true] | ||
| + | #Reputation system options Google Doc (140-character ideas: AllOurIdeas basis) [https://docs.google.com/document/d/1Hl9Ui3Qmebz2h7MLsPHm1b5ry1b9NUQPq1atZRLj0hU/edit?pli=1] | ||
| + | #AllOurIdeas Combined Voting Results (spreadsheet with bar chart - Draft version; Jsilver) [https://docs.google.com/spreadsheets/d/14d1J0RRmSzZQu2YEkc2jgmdDuICrxvl6xoPaLjO2UjU/edit?usp=sharing] | ||
| + | |||
| + | == Resources == | ||
| + | #Relevant Work Wiki [http://crowdresearch.stanford.edu/w/index.php?title=Relevant_Work] | ||
| + | #All Crowd Research Reputation resources listed on main resource page at [http://crowdresearch.stanford.edu/w/index.php?title=Resources] | ||
| + | #Building Web Reputation Systems (Randy Farmer; GoogleTechTalks video; Time 58:20) [https://www.youtube.com/watch?v=Yn7e0J9m6rE] | ||
| + | #Article on Machine Learning And Human Bias: An Uneasy Pair [http://techcrunch.com/2015/08/02/machine-learning-and-human-bias-an-uneasy-pair/] | ||
| + | |||
| + | ==Self-organised Hangout links == | ||
| + | July 30, 2015 Hangout with @Dilrukshi, @RCompton and @Arichmondfuller (34 minutes) [https://www.youtube.com/watch?v=yJ-WzDrUEHM] | ||
| + | |||
| + | == Top 10 ideas for requesters on http://allourideas.org/reputationsystem3r/results?all=true identified by Spring participants == | ||
| + | 172 votes on 42 ideas | ||
| − | |||
Which feature will best support a fair and effective reputation system?(3r) | Which feature will best support a fair and effective reputation system?(3r) | ||
| − | |||
Score (0 - 100) | Score (0 - 100) | ||
| − | 90 Rate worker on your desire to work with them on future projects. | + | 90 Rate worker on your desire to work with them on future projects. |
86 Workers involved in work evaluation and work dispute moderation get higher reputation and/or pay. | 86 Workers involved in work evaluation and work dispute moderation get higher reputation and/or pay. | ||
| − | + | ||
85 Guilds: more senior workers with same skills review your work periodically and promote you (e.g., Level 5 C++ programmer) | 85 Guilds: more senior workers with same skills review your work periodically and promote you (e.g., Level 5 C++ programmer) | ||
| − | + | ||
| − | 83 Badges for high-performing and poor-performing workers. | + | 83 Badges for high-performing and poor-performing workers. |
80 More senior experts in your same area review your work before sending it back to requesters; their judgments determine your reputation. | 80 More senior experts in your same area review your work before sending it back to requesters; their judgments determine your reputation. | ||
| − | + | ||
| − | 75 Random sampling of worker output to determine promotion. | + | 75 Random sampling of worker output to determine promotion. |
73 Traditional approach: workers and requesters can rate each other on a five-star scale after work is completed | 73 Traditional approach: workers and requesters can rate each other on a five-star scale after work is completed | ||
73 Incentive to rate honestly: if I give a high rating, the system makes it likely I get that person in the future. Low ratings → less likely. | 73 Incentive to rate honestly: if I give a high rating, the system makes it likely I get that person in the future. Low ratings → less likely. | ||
| − | + | ||
71 In addition to level based on skill, Reputation also includes levels for types of tasks, content and knowledge domains. | 71 In addition to level based on skill, Reputation also includes levels for types of tasks, content and knowledge domains. | ||
71 Guilds of experts and levels within each guild (e.g., Level 3 C++). When you want to be promoted, you apply and the guild reviews your work. | 71 Guilds of experts and levels within each guild (e.g., Level 3 C++). When you want to be promoted, you apply and the guild reviews your work. | ||
| − | == Top 10 ideas from | + | == Top 10 ideas from researchers on http://allourideas.org/reputationsystem1r identified by Spring participants == |
236 votes on 43 ideas | 236 votes on 43 ideas | ||
| Line 55: | Line 111: | ||
90 Reputation based on public+private/anonymous client/worker feedback. The system would create a Rep. Score tied to privileges/incentives | 90 Reputation based on public+private/anonymous client/worker feedback. The system would create a Rep. Score tied to privileges/incentives | ||
| + | |||
81 Offer workers option to include professional online profiles such as GitHub repositories or LinkedIn entries | 81 Offer workers option to include professional online profiles such as GitHub repositories or LinkedIn entries | ||
| + | |||
75 In addition to level based on skill, Reputation also includes levels for types of tasks, content and knowledge domains. | 75 In addition to level based on skill, Reputation also includes levels for types of tasks, content and knowledge domains. | ||
| − | 73 Five level Social Rank worker+client Public Leaderboard: Hall of Fame to Hall of Shame. Reputation tied to privileges, fair pay & work quality | + | |
| + | 73 Five level Social Rank worker+client Public Leaderboard: Hall of Fame to Hall of Shame. Reputation tied to privileges, fair pay & work quality | ||
| + | |||
71 Reputation levels (e.g.,Level 3 C++). Incentive to rate honestly: require % of task cost to bet if they'll hit a level (e.g.,Level 5) later | 71 Reputation levels (e.g.,Level 3 C++). Incentive to rate honestly: require % of task cost to bet if they'll hit a level (e.g.,Level 5) later | ||
| + | |||
70 Badges for high-performing and poor-performing workers. | 70 Badges for high-performing and poor-performing workers. | ||
| + | |||
69 Worker rating/profile including job ratings (star ratings: public/human & private/anon/algo feedback), forum posting, and dispute moderation | 69 Worker rating/profile including job ratings (star ratings: public/human & private/anon/algo feedback), forum posting, and dispute moderation | ||
| + | |||
69 Incentive to rate honestly: if I give a high rating, the system makes it likely I get that person in the future. Low ratings → less likely. | 69 Incentive to rate honestly: if I give a high rating, the system makes it likely I get that person in the future. Low ratings → less likely. | ||
| + | |||
68 Guilds: more senior workers with same skills review your work periodically and promote you (e.g., Level 5 C++ programmer) | 68 Guilds: more senior workers with same skills review your work periodically and promote you (e.g., Level 5 C++ programmer) | ||
| − | |||
| + | 67 Requester should also be evaluated on their participation on the platform like dispute solving, rating workers, fellow requester, etc | ||
| + | == Top 10 workers' preferred feedback from http://allourideas.org/reputationsystem2w/results by spring participants == | ||
| + | 3318 votes on 49 ideas | ||
| − | + | 75 Indicate when a worker/requester has joined the system: the calendar date or the number of days,weeks,months, and/or years since inception. | |
| − | + | ||
| − | + | 68 Workers get promoted with good work reviews. | |
| − | + | ||
| − | + | ||
| − | + | 65 Traditional approach: workers and requesters can rate each other on a five-star scale after work is completed | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| + | 63 Provide ability to test up to various levels of a skill set. (i.e. Level 5 photoshop, Level 2 Python) | ||
| − | + | 62 Reputation based on job rating(5stars), private feedback, and platform participation (work evaluation, dispute moderation,forum interaction) | |
| − | + | 62 In addition to level based on skill, reputation also includes levels by types of tasks, content and knowledge domains. | |
| − | + | ||
| − | + | ||
| − | + | 60 Rate worker on your desire to work with them on future projects. | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | 59 Rate requester on your desire to work with them on future projects | |
| − | + | ||
| − | + | 59 Requester rating/profile criteria: accuracy/consistency, communication skills, ethics, hiring rate, cooperation, public/private feedback | |
| + | |||
| + | 59 Provide public access to ratings and reviews for both workers and requesters | ||
Latest revision as of 20:43, 10 August 2015
Welcome to the wiki for the Daemo Reputation System!
Contents
- 1 Factors affecting the reputation system
- 2 Types of reputation systems
- 3 Elements of reputation systems
- 4 Ideas to address the cold start problem (new workers with no reputation)
- 5 Summer Milestone 10 Reputation Paper Summaries
- 6 Summer Milestone 12 Reputation
- 7 Work previously completed by research project participants
- 8 Resources
- 9 Self-organised Hangout links
- 10 Top 10 ideas for requesters on http://allourideas.org/reputationsystem3r/results?all=true identified by Spring participants
- 11 Top 10 ideas from researchers on http://allourideas.org/reputationsystem1r identified by Spring participants
- 12 Top 10 workers' preferred feedback from http://allourideas.org/reputationsystem2w/results by spring participants
Factors affecting the reputation system
For workers:
- length of time on the platform
- number of tasks completed
- number of re-hires
- content analysis
- communication
- skills
- evidence of qualifications such as certificates
- evidence of skill level such as an internal testing service and badges to identify mastery of particular skills
- timeliness
- linked reputation evidence from other networks such as Github, LinkedIn, etc.
- ethics
For requesters:
- number of tasks posted
- length of time on the platform
- feedback
- clarity of the task posted
- clarity of expectations
- communication
- amount he/she pays compared with other requesters ???
- accuracy
- consistency
- ethics
Types of reputation systems
- content-driven and endorsement networks
- reputation tied to various skills - provide an aggregate rating for all of the task completed (for both workers and requesters) and also a breakdown of reputation for each skill
- star ratings 1-5
- worker ratings of requesters (and vice versa) place them into A, B (good), C (fair), D (poor)
Elements of reputation systems
- ongoing vs end-of-task feedback/ratings
- social network referrals rather than star ratings
- based on skills rather than everything bundled into one rating? A question has been asked saying "How do you do skills-based ratings, etc., without hindering tasks with a requirement to categorize them?"
- video verification
- external quality ratings
- Metaphor of credit ratings: rather than just people rating each other, have an (external?) authority or algorithm responsible for credit ratings (A, B, C, etc.)
- Group review of worker to be accepted onto the platform? Issues to address such as payment for reviews and who would do the reviews?
- Accept only highly skilled workers onto the platform? Interviews to onboard newbies?
Ideas to address the cold start problem (new workers with no reputation)
- Workers can do work/tasks for NGOs and charities to earn their first ratings (Charities post what they need to be done, and workers could choose work/tasks from their favorite charities).
- Provide a supervised incubator space for newbies to work on tasks with other newbies under the guidance of workers with expert skills in order to gain their first reputation star ratings (if we end up using star ratings)
Summer Milestone 10 Reputation Paper Summaries
- Outline of the Trello Milestone task, DRIs, what to do, Reputation wiki link and Google doc to keep track of who is doing what [1]
- Paper Summary Wiki- summary, pros and cons for all reputation-related research papers. [2]
- Evaluations of other crowdsourcing platform reputation systems [3]
Summer Milestone 12 Reputation
Summary of Reputation Ideas [4]
Work previously completed by research project participants
- Link to all the documents with the word "Reputation" in the Crowd Research Wikis [5]
- Paper that Neil wrote in the Spring on the Reputation System [6]
- AllOurIdeas Fair & Effective Reputation System: Voting Results (Workers) [7]
- AllOurIdeas Fair & Effective Reputation System: Voting Results (Researchers) [8]
- AllOurIdeas Fair & Effective Reputation System: Voting Results (Requesters) [9]
- Reputation system options Google Doc (140-character ideas: AllOurIdeas basis) [10]
- AllOurIdeas Combined Voting Results (spreadsheet with bar chart - Draft version; Jsilver) [11]
Resources
- Relevant Work Wiki [12]
- All Crowd Research Reputation resources listed on main resource page at [13]
- Building Web Reputation Systems (Randy Farmer; GoogleTechTalks video; Time 58:20) [14]
- Article on Machine Learning And Human Bias: An Uneasy Pair [15]
Self-organised Hangout links
July 30, 2015 Hangout with @Dilrukshi, @RCompton and @Arichmondfuller (34 minutes) [16]
Top 10 ideas for requesters on http://allourideas.org/reputationsystem3r/results?all=true identified by Spring participants
172 votes on 42 ideas
Which feature will best support a fair and effective reputation system?(3r)
Score (0 - 100)
90 Rate worker on your desire to work with them on future projects.
86 Workers involved in work evaluation and work dispute moderation get higher reputation and/or pay.
85 Guilds: more senior workers with same skills review your work periodically and promote you (e.g., Level 5 C++ programmer)
83 Badges for high-performing and poor-performing workers.
80 More senior experts in your same area review your work before sending it back to requesters; their judgments determine your reputation.
75 Random sampling of worker output to determine promotion.
73 Traditional approach: workers and requesters can rate each other on a five-star scale after work is completed
73 Incentive to rate honestly: if I give a high rating, the system makes it likely I get that person in the future. Low ratings → less likely.
71 In addition to level based on skill, Reputation also includes levels for types of tasks, content and knowledge domains.
71 Guilds of experts and levels within each guild (e.g., Level 3 C++). When you want to be promoted, you apply and the guild reviews your work.
Top 10 ideas from researchers on http://allourideas.org/reputationsystem1r identified by Spring participants
236 votes on 43 ideas
Which feature will best support a fair and effective reputation system? (1R)
Ideas
Score (0 - 100)
90 Reputation based on public+private/anonymous client/worker feedback. The system would create a Rep. Score tied to privileges/incentives
81 Offer workers option to include professional online profiles such as GitHub repositories or LinkedIn entries
75 In addition to level based on skill, Reputation also includes levels for types of tasks, content and knowledge domains.
73 Five level Social Rank worker+client Public Leaderboard: Hall of Fame to Hall of Shame. Reputation tied to privileges, fair pay & work quality
71 Reputation levels (e.g.,Level 3 C++). Incentive to rate honestly: require % of task cost to bet if they'll hit a level (e.g.,Level 5) later
70 Badges for high-performing and poor-performing workers.
69 Worker rating/profile including job ratings (star ratings: public/human & private/anon/algo feedback), forum posting, and dispute moderation
69 Incentive to rate honestly: if I give a high rating, the system makes it likely I get that person in the future. Low ratings → less likely.
68 Guilds: more senior workers with same skills review your work periodically and promote you (e.g., Level 5 C++ programmer)
67 Requester should also be evaluated on their participation on the platform like dispute solving, rating workers, fellow requester, etc
Top 10 workers' preferred feedback from http://allourideas.org/reputationsystem2w/results by spring participants
3318 votes on 49 ideas
75 Indicate when a worker/requester has joined the system: the calendar date or the number of days,weeks,months, and/or years since inception.
68 Workers get promoted with good work reviews.
65 Traditional approach: workers and requesters can rate each other on a five-star scale after work is completed
63 Provide ability to test up to various levels of a skill set. (i.e. Level 5 photoshop, Level 2 Python)
62 Reputation based on job rating(5stars), private feedback, and platform participation (work evaluation, dispute moderation,forum interaction)
62 In addition to level based on skill, reputation also includes levels by types of tasks, content and knowledge domains.
60 Rate worker on your desire to work with them on future projects.
59 Rate requester on your desire to work with them on future projects
59 Requester rating/profile criteria: accuracy/consistency, communication skills, ethics, hiring rate, cooperation, public/private feedback
59 Provide public access to ratings and reviews for both workers and requesters
