This week we will be collectively discussing and contributing to external quality ratings/reputation systems. People interested in contributing to infrastructure should check the #infra slack channel, and Infrastructure wiki page.
Is it algorithmic or human? Who exactly rates who on what dimensions?
Michael's note is that we can't base it entirely on accept/rejects or 1-5 stars, since there's a major positive bias in these scores on oDesk and AMT. Interesting suggestion was to make it like AirBnB, where feedback cannot be later linked to a single job.
As written up previously, we have multiple promotion tiers for skill area (e.g., Photoshop Level 1-6). After each task, we ask the requester to optionally provide feedback: for example, if they're Photoshop Level 3, we can ask: "given what you’ve seen, is this 1) below, 2) at, or 3) above the level of a Photoshop Level 3?” The results are delayed and aggregated to be shown in batches of, say, 5 jobs. Once you get enough upvotes to the next level from trusted requesters, Photoshop Level 4s (or 5s?) can look at your portfolio of past work and vote whether to promote you. We start by asking people to do these reviews as volunteers, but am open to workers paying to get reviewed (like taking the SAT). We have the same ranking levels for requesters.
Deliverable and Submitting
- [Reputation System] Teams will not be delivering individual reports this week. Instead you will all be working in a group to explore the idea of external quality ratings/reputation systems in depth by listing research questions that haven't been answered yet. You will collectively then work on answering those questions. The document that you will be contributing to is here. Please make your contributions by 11:59 pm 13th May 2015
- [Open Gov] We have been working on this document. Please vote for them here for researchers, here for requesters, here for workers, and here for others.
- [Collecting research questions and feature ideas] We’ll use this info when the new batch joins in, and they can pick up the ideas, research questions and features depending on their interests. Link here. No deadline, its an ongoing effort. Whenever an idea strikes, paste it here.
Please remember that this is a collective effort. Use the slack channels to communicate your thoughts with the group. Also ensure that you visit the google document often, read what others have written and help extend or refine those ideas further. Since teams are not creating individual submissions but collectively working together, we highly encourage all the teams to participate and help us shape how we move forward.
Please fill out the weekly survey so we can continue to improve your crowd research experience.