Milestone 8 TuringMachine Input and output transducers

From crowdresearch
Jump to: navigation, search

Input and output transducers

Tasks get vetted or improved by people on the platform immediately after getting submitted, and before workers are exposed to them. Results are likewise vetted and tweaked. For example, peer-review.


Cost: who pays for this? In other words, can this be done without hugely increasing the cost of crowdsourcing?

Speed: is it possible to do this quickly enough to give near-immediate feedback to requesters? Like, 2–4 minutes? As spamgirl says, "The #1 thing that requesters love about AMT from her recent survey of requesters, is that the moment that I post tasks, they start getting done."

From Edwin: What happens when I have a task that I know is hard but I want workers to just try their best and submit? I’m OK with it being subjective, but the panel would just reject my task, which would have been frustrating.

From Edwin: Could this help deal with people feeling bad when rejecting work? Maybe we need a new metaphor, like revision.


Hypothesis: Well designed tasks have better quality submissions
Task Improvement Solution 1: Fixing the TASK with aggregated feedback In this scenario we leverage the one to many relationship in the task graph i.e. one Requestor many Workers

  • Hypothesis: Emergency Break: The tasks that have serious design flaws are easy to fix:
    • If a task has serious design flaws then majority of workers will face issues understanding it. At present there is no mechanism to capture this feedback.
    • We propose an interface UI control that will allow workers immediately report the issues related to the task flaws. This is similar to the Emergency Break used in the Trains.
    • Once the task is posted it will have timestamp associated with it. If 45% workers, who are working on the task raise the Emergency Break to report the unclear instructions; the task will be then placed on hold and the requestor will be notified.
    • Workers can provide feedback on the task instructions.
    • We can learn how workers use Alerts and add penalty for misusing it.
    • This is a low cost and faster mechanism.
Emergency Break

Hypothesis 1: Over the period of time the workers will get experience in the marketplace and would participate in review process for monetary volunteering benefits. Ranking system will help select the expert from the crowd
Hypothesis 2: Amount of time and money spent in reviewing the bad quality results will be more than amount of time spent improving the task and getting immediate feedback. This extra money can be used to pay the experts.
Task Improvement Solution 2: Trust Circle Design Process: Requestor - Expert Worker - Supervisors - Workers

  • The Figure below highlights the detailed review process
  • Select Expert Workers using automated algorithms
  • Select supervisors from the set of Expert Workers. Expert workers are paid higher and selected from pool of high accomplished individuals. We have also designed the ranking mechanism that can be integrated with the system to motivate workers to perform well and get into the class of Expert Workers.

Motivation for becoming an Expert:

  • Intrinsic motivation: Bad quality submission or cheating behavior affects entire crowdsourcing community. Most of the workers want to stop the bad guys and volunteer their time. However, the current system does not have any mechanism that will involve workers in filtering out bad submissions. Pool of experts is a motivated group of individuals who want to maximize the social welfare.
  • Extrinsic motivation: The expert workers are paid higher for the experience & managerial skills they bring in. In addition, we propose the reward system that will encourage the workers to do well and move into the class of socially recognized experts.

Realtime Feedback & Peer Pressure to Perform Better

  • The Figure below highlights the worker A's dashboard. Please read the diagram from #0 to #5 i.e. from the bottom to top
  • The worker A receives real time feedback, motivational messages.
  • The worker A can see the live task statistics and performance of his colleagues can motivate him to participate and do well in the task.

Over the period of time the system can build a network graph of workers and requestors who work well together. This can be further extended to build the teams that can work together on highly complex tasks.