Milestone 5 Peer Review Wire Frame

From crowdresearch
Jump to: navigation, search

What Would Peer Review look like?

alt = The steps that could be followed in setting up a peer review system for Amazon Mechanical Turk.

Why is this a good idea?

We described some of our ideas in this particular page: [[1]] - the long and short of which is that there is a necessity to ensure good work is ratified by someone other than the person who puts up a task to ensure that it is done in an unbiased manner.

How does this prototype process make sense?

The workers need to ensure that they provide a good monetary incentive for the reviewer or they could choose to not review the task. No task will be allowed to be posted without a definite vote for whether there is a need for a reviewer or not. Also, a reason should be given for why it is important for a review to be conducted - this is where the requester could go above and beyond the task details and provide information about what they may be looking for, and what may be obvious errors.

There is a lot missing from this!

We agree.

The best way to improve this model is to add in a social interaction aspect, and a way to use reputation to select the reviewers.

Also we assumed the latter part of the process was clear, after the reviewer has commented upon the task. After a task is deemed reviewer-worthy, they are assigned to it and look into the work that is submitted.

There are also problems with this method:

  • there may be ways for the requester to just avoid using a reviewer - deciding which situation needs reviewers and which don't is an important problem to be solved
  • There is anonymity among who reviews the work making it easier for them to remain unbiased.
  • Another problem is the reviewers could always decide to say that the monetary amount is insufficient, but there are two caveats to this problem:
    • The first is that many of the workers on AMT are quite idealistic and as we have been reading many actually want the work to succeed
    • Secondly, this problem can be cleverly offset by reducing reputation by some amount every time a reviewer says that the monetary compensation is insufficient - this is an important concept for actual prototyping