Difference between revisions of "Summer Milestone 10 Reputation System outline and review"

From crowdresearch
Jump to: navigation, search
(Top 10 ideas for requesters identified by Spring participants)
 
(34 intermediate revisions by 3 users not shown)
Line 6: Line 6:
 
#length of time on the platform
 
#length of time on the platform
 
#number of tasks completed
 
#number of tasks completed
 +
#number of re-hires
 
#content analysis
 
#content analysis
 
#communication
 
#communication
 
#skills
 
#skills
 +
#evidence of qualifications such as certificates
 +
#evidence of skill level such as an internal testing service and badges to identify mastery of particular skills
 +
#timeliness
 +
#linked reputation evidence from other networks such as Github, LinkedIn, etc.
 +
#ethics
  
 
'''For requesters:'''
 
'''For requesters:'''
Line 15: Line 21:
 
#feedback
 
#feedback
 
#clarity of the task posted
 
#clarity of the task posted
 +
#clarity of expectations
 
#communication
 
#communication
 
#amount he/she pays compared with other requesters ???
 
#amount he/she pays compared with other requesters ???
 
+
#accuracy
 
+
#consistency
== Top 10 ideas for requesters identified by Spring participants ==
+
#ethics
Which feature will best support a fair and effective reputation system?(3r)
+
Ideas
+
 
+
Score (0 - 100)
+
 
+
+
90 Rate worker on your desire to work with them on future projects.
+
86 Workers involved in work evaluation and work dispute moderation get higher reputation and/or pay.
+
85 Guilds: more senior workers with same skills review your work periodically and promote you (e.g., Level 5 C++ programmer)
+
83 Badges for high-performing and poor-performing workers.
+
80 More senior experts in your same area review your work before sending it back to requesters; their judgments determine your reputation.
+
75 Random sampling of worker output to determine promotion.
+
73 Traditional approach: workers and requesters can rate each other on a five-star scale after work is completed
+
73 Incentive to rate honestly: if I give a high rating, the system makes it likely I get that person in the future. Low ratings → less likely.
+
71 In addition to level based on skill, Reputation also includes levels for types of tasks, content and knowledge domains.
+
71 Guilds of experts and levels within each guild (e.g., Level 3 C++). When you want to be promoted, you apply and the guild reviews your work.
+
  
 
== Types of reputation systems ==
 
== Types of reputation systems ==
Line 46: Line 37:
 
#ongoing vs end-of-task feedback/ratings
 
#ongoing vs end-of-task feedback/ratings
 
#social network referrals rather than star ratings
 
#social network referrals rather than star ratings
#based on skills rather than everything bundled into one rating?
+
#based on skills rather than everything bundled into one rating? A question has been asked saying "How do you do skills-based ratings, etc., without hindering tasks with a requirement to categorize them?"
#video verification  
+
#video verification
 +
#external quality ratings
 +
#Metaphor of credit ratings: rather than just people rating each other, have an (external?) authority or algorithm responsible for credit ratings (A, B, C, etc.)

 +
#Group review of worker to be accepted onto the platform? Issues to address such as payment for reviews and who would do the reviews?
 +
#Accept only highly skilled workers onto the platform? Interviews to onboard newbies?
  
 
+
== Ideas to address the cold start problem (new workers with no reputation) ==
----
+
#Workers can do work/tasks for NGOs and charities to earn their first ratings (Charities post what they need to be done, and workers could choose work/tasks from their favorite charities).
 +
#Provide a supervised incubator space for newbies to work on tasks with other newbies under the guidance of workers with expert skills in order to gain their first reputation star ratings (if we end up using star ratings)
  
 
== Summer Milestone 10 Reputation Paper Summaries ==
 
== Summer Milestone 10 Reputation Paper Summaries ==
#Outline of the Trello Milestone task, DRIs, what to do, Reputation wiki link and google doc to keep track of who is doing what [https://docs.google.com/presentation/d/1A2WxfCNJn2Uu0_Y5un9eZddSGR6O1-m8s9rxkJosT-Y/edit#slide=id.p]
+
#Outline of the Trello Milestone task, DRIs, what to do, Reputation wiki link and Google doc to keep track of who is doing what [https://docs.google.com/presentation/d/1A2WxfCNJn2Uu0_Y5un9eZddSGR6O1-m8s9rxkJosT-Y/edit#slide=id.p]
 
#'''Paper Summary Wiki'''- summary, pros and cons for all reputation-related research papers. [http://crowdresearch.stanford.edu/w/index.php?title=Summer_Milestone_9_Reputation_Systems_research_and_exploration]
 
#'''Paper Summary Wiki'''- summary, pros and cons for all reputation-related research papers. [http://crowdresearch.stanford.edu/w/index.php?title=Summer_Milestone_9_Reputation_Systems_research_and_exploration]
 +
#'''Evaluations of other crowdsourcing platform reputation systems''' [http://crowdresearch.stanford.edu/w/index.php?title=Summer_Milestone_9_Evaluations_of_reputation_systems_on_other_crowdsourcing_platforms#]
 +
 +
== Summer Milestone 12 Reputation ==
 +
Summary of Reputation Ideas [http://crowdresearch.stanford.edu/w/index.php?title=Reputation_System_Ideas_Summary]
  
 
== Work previously completed by research project participants ==
 
== Work previously completed by research project participants ==
 
#Link to all the documents with the word "Reputation" in the Crowd Research Wikis [http://crowdresearch.stanford.edu/w/index.php?search=reputation&title=Special%3ASearch&go=Go]
 
#Link to all the documents with the word "Reputation" in the Crowd Research Wikis [http://crowdresearch.stanford.edu/w/index.php?search=reputation&title=Special%3ASearch&go=Go]
 +
#Paper that Neil wrote in the Spring on the Reputation System [http://crowdresearch.stanford.edu/w/index.php?title=Milestone_7_TuringMachine]
 
#AllOurIdeas Fair & Effective Reputation System: Voting Results (Workers) [http://allourideas.org/reputationsystem2w/results?all=true]
 
#AllOurIdeas Fair & Effective Reputation System: Voting Results (Workers) [http://allourideas.org/reputationsystem2w/results?all=true]
 
#AllOurIdeas Fair & Effective Reputation System: Voting Results (Researchers) [http://allourideas.org/reputationsystem1r/results?all=true]
 
#AllOurIdeas Fair & Effective Reputation System: Voting Results (Researchers) [http://allourideas.org/reputationsystem1r/results?all=true]
 
#AllOurIdeas Fair & Effective Reputation System: Voting Results (Requesters) [http://allourideas.org/reputationsystem3r/results?all=true]  
 
#AllOurIdeas Fair & Effective Reputation System: Voting Results (Requesters) [http://allourideas.org/reputationsystem3r/results?all=true]  
#Reputation options Google Doc (140-character ideas: AllOurIdeas basis) [https://docs.google.com/document/d/1Hl9Ui3Qmebz2h7MLsPHm1b5ry1b9NUQPq1atZRLj0hU/edit?pli=1]
+
#Reputation system options Google Doc (140-character ideas: AllOurIdeas basis) [https://docs.google.com/document/d/1Hl9Ui3Qmebz2h7MLsPHm1b5ry1b9NUQPq1atZRLj0hU/edit?pli=1]
#Reputation system ideas generation document from Spring session [https://docs.google.com/document/d/1Hl9Ui3Qmebz2h7MLsPHm1b5ry1b9NUQPq1atZRLj0hU/edit?usp=sharing '''document'''] They used this to have 140 character ideas which were added on AllOurIdeas.org.
+
#AllOurIdeas Combined Voting Results (spreadsheet with bar chart - Draft version; Jsilver) [https://docs.google.com/spreadsheets/d/14d1J0RRmSzZQu2YEkc2jgmdDuICrxvl6xoPaLjO2UjU/edit?usp=sharing]
  
 
== Resources ==
 
== Resources ==
 
#Relevant Work Wiki [http://crowdresearch.stanford.edu/w/index.php?title=Relevant_Work]
 
#Relevant Work Wiki [http://crowdresearch.stanford.edu/w/index.php?title=Relevant_Work]
 
#All Crowd Research Reputation resources listed on main resource page at [http://crowdresearch.stanford.edu/w/index.php?title=Resources]
 
#All Crowd Research Reputation resources listed on main resource page at [http://crowdresearch.stanford.edu/w/index.php?title=Resources]
 +
#Building Web Reputation Systems (Randy Farmer; GoogleTechTalks video; Time 58:20) [https://www.youtube.com/watch?v=Yn7e0J9m6rE]
 +
#Article on Machine Learning And Human Bias: An Uneasy Pair [http://techcrunch.com/2015/08/02/machine-learning-and-human-bias-an-uneasy-pair/]
 +
 +
==Self-organised Hangout links ==
 +
July 30, 2015 Hangout with @Dilrukshi, @RCompton and @Arichmondfuller (34 minutes) [https://www.youtube.com/watch?v=yJ-WzDrUEHM]
 +
 +
== Top 10 ideas for requesters on http://allourideas.org/reputationsystem3r/results?all=true identified by Spring participants ==
 +
172 votes on 42 ideas
 +
 +
Which feature will best support a fair and effective reputation system?(3r)
 +
 +
Score (0 - 100)
 +
 +
90 Rate worker on your desire to work with them on future projects.
 +
 +
86 Workers involved in work evaluation and work dispute moderation get higher reputation and/or pay.
 +
 +
85 Guilds: more senior workers with same skills review your work periodically and promote you (e.g., Level 5 C++ programmer)
 +
 +
83 Badges for high-performing and poor-performing workers.
 +
 +
80 More senior experts in your same area review your work before sending it back to requesters; their judgments determine your reputation.
 +
 +
75 Random sampling of worker output to determine promotion.
 +
 +
73 Traditional approach: workers and requesters can rate each other on a five-star scale after work is completed
 +
 +
73 Incentive to rate honestly: if I give a high rating, the system makes it likely I get that person in the future. Low ratings → less likely.
 +
 +
71 In addition to level based on skill, Reputation also includes levels for types of tasks, content and knowledge domains.
 +
 +
71 Guilds of experts and levels within each guild (e.g., Level 3 C++). When you want to be promoted, you apply and the guild reviews your work.
 +
 +
== Top 10 ideas from researchers on http://allourideas.org/reputationsystem1r identified by Spring participants ==
 +
236 votes on 43 ideas
 +
 +
Which feature will best support a fair and effective reputation system? (1R)
 +
 +
Ideas
 +
 +
Score (0 - 100)
 +
 +
90 Reputation based on public+private/anonymous client/worker feedback. The system would create a Rep. Score tied to privileges/incentives
 +
 +
81 Offer workers option to include professional online profiles such as GitHub repositories or LinkedIn entries
 +
 +
75 In addition to level based on skill, Reputation also includes levels for types of tasks, content and knowledge domains.
 +
 +
73 Five level Social Rank worker+client Public Leaderboard: Hall of Fame to Hall of Shame. Reputation tied to privileges, fair pay & work quality
 +
 +
71 Reputation levels (e.g.,Level 3 C++). Incentive to rate honestly: require % of task cost to bet if they'll hit a level (e.g.,Level 5) later
 +
 +
70 Badges for high-performing and poor-performing workers.
 +
 +
69 Worker rating/profile including job ratings (star ratings: public/human & private/anon/algo feedback), forum posting, and dispute moderation
 +
 +
69 Incentive to rate honestly: if I give a high rating, the system makes it likely I get that person in the future. Low ratings → less likely.
 +
 +
68 Guilds: more senior workers with same skills review your work periodically and promote you (e.g., Level 5 C++ programmer)
 +
 +
67 Requester should also be evaluated on their participation on the platform like dispute solving, rating workers, fellow requester, etc
 +
 +
== Top 10 workers' preferred feedback from http://allourideas.org/reputationsystem2w/results by spring participants ==
 +
3318 votes on 49 ideas
 +
 +
75 Indicate when a worker/requester has joined the system: the calendar date or the number of days,weeks,months, and/or years since inception.
 +
 +
68 Workers get promoted with good work reviews.
 +
 +
65 Traditional approach: workers and requesters can rate each other on a five-star scale after work is completed
 +
 +
63 Provide ability to test up to various levels of a skill set. (i.e. Level 5 photoshop, Level 2 Python)
 +
 +
62 Reputation based on job rating(5stars), private feedback, and platform participation (work evaluation, dispute moderation,forum interaction)
 +
 +
62 In addition to level based on skill, reputation also includes levels by types of tasks, content and knowledge domains.
 +
 +
60 Rate worker on your desire to work with them on future projects.
 +
 +
59 Rate requester on your desire to work with them on future projects
 +
 +
59 Requester rating/profile criteria: accuracy/consistency, communication skills, ethics, hiring rate, cooperation, public/private feedback
 +
 +
59 Provide public access to ratings and reviews for both workers and requesters

Latest revision as of 21:43, 10 August 2015

Welcome to the wiki for the Daemo Reputation System!


Factors affecting the reputation system

For workers:

  1. length of time on the platform
  2. number of tasks completed
  3. number of re-hires
  4. content analysis
  5. communication
  6. skills
  7. evidence of qualifications such as certificates
  8. evidence of skill level such as an internal testing service and badges to identify mastery of particular skills
  9. timeliness
  10. linked reputation evidence from other networks such as Github, LinkedIn, etc.
  11. ethics

For requesters:

  1. number of tasks posted
  2. length of time on the platform
  3. feedback
  4. clarity of the task posted
  5. clarity of expectations
  6. communication
  7. amount he/she pays compared with other requesters ???
  8. accuracy
  9. consistency
  10. ethics

Types of reputation systems

  1. content-driven and endorsement networks
  2. reputation tied to various skills - provide an aggregate rating for all of the task completed (for both workers and requesters) and also a breakdown of reputation for each skill
  3. star ratings 1-5
  4. worker ratings of requesters (and vice versa) place them into A, B (good), C (fair), D (poor)

Elements of reputation systems

  1. ongoing vs end-of-task feedback/ratings
  2. social network referrals rather than star ratings
  3. based on skills rather than everything bundled into one rating? A question has been asked saying "How do you do skills-based ratings, etc., without hindering tasks with a requirement to categorize them?"
  4. video verification
  5. external quality ratings
  6. Metaphor of credit ratings: rather than just people rating each other, have an (external?) authority or algorithm responsible for credit ratings (A, B, C, etc.)

  7. Group review of worker to be accepted onto the platform? Issues to address such as payment for reviews and who would do the reviews?
  8. Accept only highly skilled workers onto the platform? Interviews to onboard newbies?

Ideas to address the cold start problem (new workers with no reputation)

  1. Workers can do work/tasks for NGOs and charities to earn their first ratings (Charities post what they need to be done, and workers could choose work/tasks from their favorite charities).
  2. Provide a supervised incubator space for newbies to work on tasks with other newbies under the guidance of workers with expert skills in order to gain their first reputation star ratings (if we end up using star ratings)

Summer Milestone 10 Reputation Paper Summaries

  1. Outline of the Trello Milestone task, DRIs, what to do, Reputation wiki link and Google doc to keep track of who is doing what [1]
  2. Paper Summary Wiki- summary, pros and cons for all reputation-related research papers. [2]
  3. Evaluations of other crowdsourcing platform reputation systems [3]

Summer Milestone 12 Reputation

Summary of Reputation Ideas [4]

Work previously completed by research project participants

  1. Link to all the documents with the word "Reputation" in the Crowd Research Wikis [5]
  2. Paper that Neil wrote in the Spring on the Reputation System [6]
  3. AllOurIdeas Fair & Effective Reputation System: Voting Results (Workers) [7]
  4. AllOurIdeas Fair & Effective Reputation System: Voting Results (Researchers) [8]
  5. AllOurIdeas Fair & Effective Reputation System: Voting Results (Requesters) [9]
  6. Reputation system options Google Doc (140-character ideas: AllOurIdeas basis) [10]
  7. AllOurIdeas Combined Voting Results (spreadsheet with bar chart - Draft version; Jsilver) [11]

Resources

  1. Relevant Work Wiki [12]
  2. All Crowd Research Reputation resources listed on main resource page at [13]
  3. Building Web Reputation Systems (Randy Farmer; GoogleTechTalks video; Time 58:20) [14]
  4. Article on Machine Learning And Human Bias: An Uneasy Pair [15]

Self-organised Hangout links

July 30, 2015 Hangout with @Dilrukshi, @RCompton and @Arichmondfuller (34 minutes) [16]

Top 10 ideas for requesters on http://allourideas.org/reputationsystem3r/results?all=true identified by Spring participants

172 votes on 42 ideas

Which feature will best support a fair and effective reputation system?(3r)

Score (0 - 100)

90 Rate worker on your desire to work with them on future projects.

86 Workers involved in work evaluation and work dispute moderation get higher reputation and/or pay.

85 Guilds: more senior workers with same skills review your work periodically and promote you (e.g., Level 5 C++ programmer)

83 Badges for high-performing and poor-performing workers.

80 More senior experts in your same area review your work before sending it back to requesters; their judgments determine your reputation.

75 Random sampling of worker output to determine promotion.

73 Traditional approach: workers and requesters can rate each other on a five-star scale after work is completed

73 Incentive to rate honestly: if I give a high rating, the system makes it likely I get that person in the future. Low ratings → less likely.

71 In addition to level based on skill, Reputation also includes levels for types of tasks, content and knowledge domains.

71 Guilds of experts and levels within each guild (e.g., Level 3 C++). When you want to be promoted, you apply and the guild reviews your work.

Top 10 ideas from researchers on http://allourideas.org/reputationsystem1r identified by Spring participants

236 votes on 43 ideas

Which feature will best support a fair and effective reputation system? (1R)

Ideas

Score (0 - 100)

90 Reputation based on public+private/anonymous client/worker feedback. The system would create a Rep. Score tied to privileges/incentives

81 Offer workers option to include professional online profiles such as GitHub repositories or LinkedIn entries

75 In addition to level based on skill, Reputation also includes levels for types of tasks, content and knowledge domains.

73 Five level Social Rank worker+client Public Leaderboard: Hall of Fame to Hall of Shame. Reputation tied to privileges, fair pay & work quality

71 Reputation levels (e.g.,Level 3 C++). Incentive to rate honestly: require % of task cost to bet if they'll hit a level (e.g.,Level 5) later

70 Badges for high-performing and poor-performing workers.

69 Worker rating/profile including job ratings (star ratings: public/human & private/anon/algo feedback), forum posting, and dispute moderation

69 Incentive to rate honestly: if I give a high rating, the system makes it likely I get that person in the future. Low ratings → less likely.

68 Guilds: more senior workers with same skills review your work periodically and promote you (e.g., Level 5 C++ programmer)

67 Requester should also be evaluated on their participation on the platform like dispute solving, rating workers, fellow requester, etc

Top 10 workers' preferred feedback from http://allourideas.org/reputationsystem2w/results by spring participants

3318 votes on 49 ideas

75 Indicate when a worker/requester has joined the system: the calendar date or the number of days,weeks,months, and/or years since inception.

68 Workers get promoted with good work reviews.

65 Traditional approach: workers and requesters can rate each other on a five-star scale after work is completed

63 Provide ability to test up to various levels of a skill set. (i.e. Level 5 photoshop, Level 2 Python)

62 Reputation based on job rating(5stars), private feedback, and platform participation (work evaluation, dispute moderation,forum interaction)

62 In addition to level based on skill, reputation also includes levels by types of tasks, content and knowledge domains.

60 Rate worker on your desire to work with them on future projects.

59 Rate requester on your desire to work with them on future projects

59 Requester rating/profile criteria: accuracy/consistency, communication skills, ethics, hiring rate, cooperation, public/private feedback

59 Provide public access to ratings and reviews for both workers and requesters