WinterMilestone 2 CarpeNoctem

From crowdresearch
Jump to: navigation, search

Attend a Panel to Hear from Workers and Requesters

Deliverable

Report on some of the observations you gathered during the panel.

Reading Others' Insights

Worker perspective: Being a Turker

Summary

  • An ethnomethodological analysis of publicly available content on Turker Nation, a general forum for Amazon Mechanical Turk (AMT) users.
  • Provided novel depth and detail on how the Turker Nation members operate as economic actors, working out which Requesters and jobs are worthwhile to them

What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Workers use AMT as a “safety net” to compensate for their everyday low wages or as a safety net when they are not able to find jobs
    • Although Turkers often get paid below minimum wage, and view $4-6 hours as an acceptable average salary for a Turker, they still are willing to work for AMT because they have a lack of choice. Many have low wages in their full time jobs, or are unable to find jobs, hoping that AMT can help them with their daily expenses.
    • “apparently half a million people find work at pay rates they’re entirely happy with but pay rates that are below minimum wage”
    • Another example is a worker whose boss actually doesn’t mind him turking during his full-time job because he is underpaid for his current job
    • These group of people are in the “digital underclass” where AMT serves as a way for them to live on the “breadline”
  • Some turkers prefer working for AMT than finding a full-time job because of the flexibility they get
    • For example, one worker mentioned that he liked the flexibility of sitting in front of the sofa watching TV while doing these mindless tasks to earn money
  • In order to optimize their work - pay ratio, many turkers form a strong community on Turker Nation to share insights on the pay of each HIT request, the best and worst Requesters etc.
    • Turkers believe that their private collective efforts can influence the Turker community, and that government regulations will do more harm than good
    • This is considered “invisible work” because turkers have to spend these extra efforts themselves without pay.
    • Solutions or recommendations can focus more on easing the “invisible work” of the turkers, improving the transparency of work, and the communication channels between turkers and requesters instead of enforcing government regulations or minimum wages
  • Novice and experienced turkers have different goals:
    • Novice: have to concentrate on getting their numbers up for both their approval rating (>90%) and their HIT count (1000 or even 5000)
    • Experienced: not so concerned with HIT count, as it only increases, approval rating can rise and fall and is something that is of key importance to them. High approval rating is crucial in getting access to a wider selection of and better paying HITs.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

Summary

  • Investigates the usefulness of micro-tasks markets like AMT for collecting user measurements, and how special care in the design of the tasks affects the quality of the outcome

What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • The fact that certain HITs were done in poor quality may not be due to malicious incentives. Turkers tend to optimize for their work (which is justifiable given their low wages) and the HITs may not be well-designed to allow turkers to have a clear understanding on what they have to do
    • One way requesters can ensure better solutions is to engage with Turkers on forums such as Turker Nation, so as to get a better feedback on the design of their tasks.
    • The way to improve the quality of the tasks is to include concretely verifiable questions that signal users that their answers will be scrutinized. By doing so, turkers will know what they are expected of the task and do the tasks more seriously.

Worker perspective: Turkopticon

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

Workers need:

  • to efficiently search for a requester, navigate and browse through the available tasks easily
  • a trustwothiness guarantee for requesters. Good work needs not to be rejected and workers need be paid on time.
  • a better user interface
  • a prediction of the completion time of the posted tasks


2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requesters need:

  • to be able to scale up easily
  • a user friendly interface and workflow system to post tasks
  • a good reputation system for workers
  • ensure quality

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

List out the observations you made while doing your fieldwork. Links to examples (posts / threads) would be extremely helpful.

Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

A set of bullet points summarizing the needs of workers.

  • Example: Workers need to be respected by their employers. Evidence: Sanjay said in the worker panel that he wrote an angry email to a requester who mass-rejected his work. Interpretation: this wasn't actually about the money; it was about the disregard for Sanjay's work ethic.

Requester Needs

A set of bullet points summarizing the needs of requesters.

  • Example: requesters need to trust the results they get from workers. Evidence: In this thread on Reddit (linked), a requester is struggling to know which results to use and which ones to reject or re-post for more data. Interpretation: it's actually quite difficult for requesters to know whether 1) a worker tried hard but the question was unclear or very difficult or an edge case, or 2) a worker wasn't really putting in a best effort.