This week we will be collectively discussing and contributing to External quality ratings. People interested in contributing to infrastructure should check the #infra slack channel, and Infrastructure wiki page.
External quality ratings
Is it algorithmic or human? Who exactly rates who on what dimensions?
Michael's note is that we can't base it entirely on accept/rejects or 1-5 stars, since there's a major positive bias in these scores on oDesk and AMT. Interesting suggestion was to make it like AirBnB, where feedback cannot be later linked to a single job.
As written up previously, we have multiple promotion tiers for skill area (e.g., Photoshop Level 1-6). After each task, we ask the requester to optionally provide feedback: for example, if they're Photoshop Level 3, we can ask: "given what you’ve seen, is this 1) below, 2) at, or 3) above the level of a Photoshop Level 3?” The results are delayed and aggregated to be shown in batches of, say, 5 jobs. Once you get enough upvotes to the next level from trusted requesters, Photoshop Level 4s (or 5s?) can look at your portfolio of past work and vote whether to promote you. We start by asking people to do these reviews as volunteers, but am open to workers paying to get reviewed (like taking the SAT). We have the same ranking levels for requesters.
Teams will not be delivering individual reports this week. Instead you will all be working in a group to explore the idea of External Quality Rating in depth by listing research questions that haven't been answered yet. You will collectively then work on answering those questions.
You will also be listing out very specific features that you want to have in our platform to enable External Quality Rating. You will also be considering the various implementation details for these features and writing down how to design them.
Please remember that this is a collective effort. Use the slack channels to communicate your thoughts with the group. Also ensure that you visit the google document often, read what others have written and help extend or refine those ideas further.
The document that you will be contributing to is here.
Since teams are not creating individual submissions but collectively working together, we highly encourage all the teams to participate and help us shape how we move forward.
Please make your contributions by 11:59 pm 14th May 2015
Please fill out the weekly survey so we can continue to improve your crowd research experience.