Difference between revisions of "Milestone 11"

From crowdresearch
Jump to: navigation, search
Line 1: Line 1:
'''This week we will be collectively discussing and contributing to External quality ratings'''. People interested in contributing to infrastructure should check the #infra slack channel, and [[Infrastructure]] wiki page.
+
'''This week we will be collectively discussing and contributing to external quality ratings/reputation systems'''. People interested in contributing to infrastructure should check the #infra slack channel, and [[Infrastructure]] wiki page.
  
== External quality ratings ==  
+
== Reputation systems ==  
 
=== Questions ===  
 
=== Questions ===  
 
Is it algorithmic or human? Who exactly rates who on what dimensions?  
 
Is it algorithmic or human? Who exactly rates who on what dimensions?  
Line 9: Line 9:
 
As written up previously, we have multiple promotion tiers for skill area (e.g., Photoshop Level 1-6). After each task, we ask the requester to optionally provide feedback: for example, if they're Photoshop Level 3, we can ask: "given what you’ve seen, is this 1) below, 2) at, or 3) above the level of a Photoshop Level 3?” The results are delayed and aggregated to be shown in batches of, say, 5 jobs. Once you get enough upvotes to the next level from trusted requesters, Photoshop Level 4s (or 5s?) can look at your portfolio of past work and vote whether to promote you. We start by asking people to do these reviews as volunteers, but am open to workers paying to get reviewed (like taking the SAT). We have the same ranking levels for requesters.
 
As written up previously, we have multiple promotion tiers for skill area (e.g., Photoshop Level 1-6). After each task, we ask the requester to optionally provide feedback: for example, if they're Photoshop Level 3, we can ask: "given what you’ve seen, is this 1) below, 2) at, or 3) above the level of a Photoshop Level 3?” The results are delayed and aggregated to be shown in batches of, say, 5 jobs. Once you get enough upvotes to the next level from trusted requesters, Photoshop Level 4s (or 5s?) can look at your portfolio of past work and vote whether to promote you. We start by asking people to do these reviews as volunteers, but am open to workers paying to get reviewed (like taking the SAT). We have the same ranking levels for requesters.
  
== Deliverable ==
+
== Deliverable and Submitting ==
  
Teams will not be delivering individual reports this week. Instead you will all be working in a group to explore the idea of External Quality Rating in depth by listing research questions that haven't been answered yet. You will collectively then work on answering those questions.  
+
* [Reputation System] Teams will not be delivering individual reports this week. Instead you will all be working in a group to explore the idea of external quality ratings/reputation systems in depth by listing research questions that haven't been answered yet. You will collectively then work on answering those questions. The document that you will be contributing to is [https://docs.google.com/document/d/1Hl9Ui3Qmebz2h7MLsPHm1b5ry1b9NUQPq1atZRLj0hU/edit?usp=sharing here]. Please make your contributions by 11:59 pm 13th May 2015
 +
 +
* [Open Gov] We have been working on this [https://docs.google.com/document/d/1UVvLdPZELOsopMlmVOVHZQF-Y8XaKzrPZwTUliPPAZI/edit document]. Please write 140 character summaries of ideas and post them on [http://www.allourideas.org/opengov allourideas.org/opengov]. Let's aim for the deadline to be Sunday midnight PST? 
  
You will also be listing out very specific features that you want to have in our platform to enable External Quality Rating. You will also be considering the various implementation details for these features and writing down how to design them.
+
* [Collecting research questions and feature ideas]  We’ll use this info when the new batch joins in, and they can pick up the ideas, research questions and features depending on their interests. Link [https://docs.google.com/document/d/17ZJ60T8qB9eCUX56n2kqJv88CzD-2adX-BVOPsJGV2s/edit?usp=sharing here]. No deadline, its an ongoing effort. Whenever an idea strikes, paste it here.  
  
Please remember that this is a collective effort. Use the slack channels to communicate your thoughts with the group. Also ensure that you visit the google document often, read what others have written and help extend or refine those ideas further.
+
Please remember that this is a collective effort. Use the slack channels to communicate your thoughts with the group. Also ensure that you visit the google document often, read what others have written and help extend or refine those ideas further. Since teams are not creating individual submissions but collectively working together, we highly encourage all the teams to participate and help us shape how we move forward.
  
== Submitting ==
 
The document that you will be contributing to is [https://docs.google.com/document/d/1Hl9Ui3Qmebz2h7MLsPHm1b5ry1b9NUQPq1atZRLj0hU/edit?usp=sharing here].
 
 
Since teams are not creating individual submissions but collectively working together, we highly encourage all the teams to participate and help us shape how we move forward.
 
 
<strong>Please make your contributions by 11:59 pm 14th May 2015</strong>
 
  
 
== Weekly Survey ==
 
== Weekly Survey ==
  
 
Please fill out the [https://docs.google.com/forms/d/1bhO5NRf5eaE6Bjl0B011WTVkkwFUtootCtADL_0T-fQ/edit# weekly survey] so we can continue to improve your crowd research experience.
 
Please fill out the [https://docs.google.com/forms/d/1bhO5NRf5eaE6Bjl0B011WTVkkwFUtootCtADL_0T-fQ/edit# weekly survey] so we can continue to improve your crowd research experience.

Revision as of 10:09, 9 May 2015

This week we will be collectively discussing and contributing to external quality ratings/reputation systems. People interested in contributing to infrastructure should check the #infra slack channel, and Infrastructure wiki page.

Reputation systems

Questions

Is it algorithmic or human? Who exactly rates who on what dimensions?

Comments

Michael's note is that we can't base it entirely on accept/rejects or 1-5 stars, since there's a major positive bias in these scores on oDesk and AMT. Interesting suggestion was to make it like AirBnB, where feedback cannot be later linked to a single job.

Suggestions

As written up previously, we have multiple promotion tiers for skill area (e.g., Photoshop Level 1-6). After each task, we ask the requester to optionally provide feedback: for example, if they're Photoshop Level 3, we can ask: "given what you’ve seen, is this 1) below, 2) at, or 3) above the level of a Photoshop Level 3?” The results are delayed and aggregated to be shown in batches of, say, 5 jobs. Once you get enough upvotes to the next level from trusted requesters, Photoshop Level 4s (or 5s?) can look at your portfolio of past work and vote whether to promote you. We start by asking people to do these reviews as volunteers, but am open to workers paying to get reviewed (like taking the SAT). We have the same ranking levels for requesters.

Deliverable and Submitting

  • [Reputation System] Teams will not be delivering individual reports this week. Instead you will all be working in a group to explore the idea of external quality ratings/reputation systems in depth by listing research questions that haven't been answered yet. You will collectively then work on answering those questions. The document that you will be contributing to is here. Please make your contributions by 11:59 pm 13th May 2015
  • [Open Gov] We have been working on this document. Please write 140 character summaries of ideas and post them on allourideas.org/opengov. Let's aim for the deadline to be Sunday midnight PST?
  • [Collecting research questions and feature ideas] We’ll use this info when the new batch joins in, and they can pick up the ideas, research questions and features depending on their interests. Link here. No deadline, its an ongoing effort. Whenever an idea strikes, paste it here.

Please remember that this is a collective effort. Use the slack channels to communicate your thoughts with the group. Also ensure that you visit the google document often, read what others have written and help extend or refine those ideas further. Since teams are not creating individual submissions but collectively working together, we highly encourage all the teams to participate and help us shape how we move forward.


Weekly Survey

Please fill out the weekly survey so we can continue to improve your crowd research experience.