Winter Milestone 2 AtinMittra
- 1 Attend a Panel to Hear from Workers and Requesters
- 2 Reading Others' Insights
- 3 Synthesize the Needs You Found
Attend a Panel to Hear from Workers and Requesters
- Is able to conduct test hypothesis on MTurk within one day
- Acknowledges the importance of well designed HIT request
- Will release "small batch" of request instructions to gain feedback on their wording
- Can be difficult to work with non-native English speakers as their e-mails and feedback may be difficult to understand
- Talks about inhumane symbolism of identifying workers with serial numbers instead of names
- Will run threshold - if worker gets multiple choice answer right only 25% of the time, he knows they are randomly guessing. "It gets tricky when they're doing better than chance but worse than doing tasks conscientiously."
- Puts a lot of effort into designing HIT instruction because he can't pay a lot of money
- Instead of rejecting work of rouge workers, he instituted qualifications. He mentioned rejecting work will effect long-term engagement of even productive workers.
- Something to grapple with is attention span of workers
- Goes to TurkerNation to find good workers
Reading Others' Insights
Worker perspective: Being a Turker
Worker perspective: Turkopticon
Requester perspective: Crowdsourcing User Studies with Mechanical Turk
Requester perspective: The Need for Standardization in Crowdsourcing
Both perspectives: A Plea to Amazon: Fix Mechanical Turk
Synthesize the Needs You Found
List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.
A set of bullet points summarizing the needs of workers.
- Example: Workers need to be respected by their employers. Evidence: Sanjay said in the worker panel that he wrote an angry email to a requester who mass-rejected his work. Interpretation: this wasn't actually about the money; it was about the disregard for Sanjay's work ethic.
A set of bullet points summarizing the needs of requesters.
- Example: requesters need to trust the results they get from workers. Evidence: In this thread on Reddit (linked), a requester is struggling to know which results to use and which ones to reject or re-post for more data. Interpretation: it's actually quite difficult for requesters to know whether 1) a worker tried hard but the question was unclear or very difficult or an edge case, or 2) a worker wasn't really putting in a best effort.