Milestone 4 Triple Clicks Results: Bounty for Task Quality Pre-Check

From crowdresearch
Jump to: navigation, search


Problem

How might we incentivize workers to help requestors craft HITs and tasks in order to gain better task instructions?

Inspiration

Peer Review System idea, bug bounty programs, Assembly (https://assembly.com)

Concept

Building on the Peer Review System idea and borrowing from the concept of bug bounties (as found in information security communities) and other “payouts based on contributions” sites like Assembly, requestors pay a fee that is proportionate to the complexity of their task that serves as a pre-check quality bounty.

During a limited phase before the task “goes live” (pre-check), workers can support the requestor in improving the task and document clarifications that a typical worker might ask (like requirements gathering). Highly insightful and helpful clarifications and revisions/edits to the task receive a portion of the pre-check bounty.

Once the task is reviewed approved by workers and the requestor, the task is made live. Additional questions and clarifications may be added as the task progresses and are posted to a FAQ page for the task. If a question/clarification affects multiple workers, additional payout is made from the bounty with a bonus or multiplier paid out to the pre-flight check workers if no additional questions are asked during the course of the task.

Limitations

  • Determining complexity of a task and how to translate that into a number for the bounty - and how to break up the reward. Some possible suggestions: measure deviation from default task templates, required prerequisites in skills - certain task types and skill types require premiums, number of “check points” or steps in task or places that a worker has to stop off at such as websites or documents, etc.
  • Guidelines or rules for setting an appropriate period of time, number of workers to review, or amount of feedback for a task during pre-check before the task is live.
  • How to avoid having requestors get “lazy” and having workers build tasks for them in entirety? Possible to block or flag requestors that repeatedly exploit.