Difference between revisions of "Milestone 11"

From crowdresearch
Jump to: navigation, search
(Created page with "'''This week we will be collectively discussing and contributing to External quality ratings'''. People interested in contributing to infrastructure should check the #infra sl...")
(No difference)

Revision as of 14:57, 8 May 2015

This week we will be collectively discussing and contributing to External quality ratings. People interested in contributing to infrastructure should check the #infra slack channel, and Infrastructure wiki page.

External quality ratings

Questions

Is it algorithmic or human? Who exactly rates who on what dimensions?

Comments

Michael's note is that we can't base it entirely on accept/rejects or 1-5 stars, since there's a major positive bias in these scores on oDesk and AMT. Interesting suggestion was to make it like AirBnB, where feedback cannot be later linked to a single job.

Suggestions

As written up previously, we have multiple promotion tiers for skill area (e.g., Photoshop Level 1-6). After each task, we ask the requester to optionally provide feedback: for example, if they're Photoshop Level 3, we can ask: "given what you’ve seen, is this 1) below, 2) at, or 3) above the level of a Photoshop Level 3?” The results are delayed and aggregated to be shown in batches of, say, 5 jobs. Once you get enough upvotes to the next level from trusted requesters, Photoshop Level 4s (or 5s?) can look at your portfolio of past work and vote whether to promote you. We start by asking people to do these reviews as volunteers, but am open to workers paying to get reviewed (like taking the SAT). We have the same ranking levels for requesters.

Deliverable

Teams will not be delivering individual

Submitting

The document that you will be contributing to is here.

Since teams are not creating individual submissions but collectively working together, we highly encourage all the teams to participate and help us shape how we move forward.

Please make your contributions by 11:59 pm 30th April 2015, and DO NOT vote/comment until 1st May 12:05 am

Weekly Survey

Please fill out the weekly survey so we can continue to improve your crowd research experience.