WinterMilestone 2 Enigma

From crowdresearch
Revision as of 12:45, 24 January 2016 by Dineshdhakal (Talk | contribs) (Worker Needs)

Jump to: navigation, search

Attend a Panel to Hear from Workers and Requesters


The interaction with the panel was very fruitful. The following are some of the observations we could make


  • The reasons for workers to use MTurk are quite varied.
  • Some workers like the flexibility that it offers. They seemed to like the idea of being in control of their working hours
  • Another major factor is money.
  • Some workers are employed elsewhere and use MTurk as an alternative source of income
  • The working hours also depend on the availability of good work.
  • Unpredictable work days are common. "One day you end up making considerable amount of money and the very next day you'll be struggling to make a dollar"
  • Workers feel there is a lack of respect for them and their work
  • Some workers pointed out that there was no definitive way of knowing when a good work shows up unless you keep monitoring continuously
  • There is an active community of workers that communicate through a number of channels like TurkNation etc.
  • They share with each other promising HITs and other interesting work, suggestions, tips and tricks and often help each other out
  • There is a considerable barrier for new workers who don't already have good ratings as requesters might be hesitant to let new workers work on their HITs
  • Unclear or ambiguous instructions from the requesters is a challenge for workers
  • Rejection of work without explanation or any kind of feedback is disheartening.
  • Different workers have different ways of picking what to work on. Eg: Well rated tasks, Reward vs Time comparison etc.

  • The major concern for requesters is the quality of work they get
  • Requesters are also worried about workers dropping out of tasks
  • Sometimes they face difficulties in finding qualified workers who would deliver good quality results
  • Finding replacements when certain workers drop out at critical stages is a challenge for many requesters
  • Workers are mostly unknown and sandboxed in a sense; no proper channel for instant communication with the workers
  • Dealing with poor quality of work compels the requesters to increase the qualification criteria for their future HITs

Reading Others' Insights

We attended the requester and worker panel to gain a better understanding of what actually goes through the entire process.

Worker perspective: Being a Turker

  • In spite of a presence of a large number of workers only about 20% of them do the 80% of the work
  • The major drive for the workers is money
  • For a worker the wages are highly unpredictable. Some days are highly unproductive while other might be rushed. Although they set targets for themselves, they cannot be certain about achieving them
  • There is a small set of workers who work with a perspective to learn or to gain some skill sets
  • There is a good deal of pre-processing involved for the workers in terms of task selection
  • The requester demands are sometimes unclear. As a result the worker does not get clear instructions. A lot of time is spent trying to understand and get clarity on tasks.
  • If the worker finishes up early or late as expected by requester, the requester might doubt the quality of work.
  • The work available on the platform might not be consistent. The worker may have to work at odd hours if he is really keen on finishing up the work.
  • Sometimes the user does not get any feedback and just a rejection hence all the effort put by the worker gets wasted. Not getting paid for their work is the biggest fear for the majority of the workers
  • Given a choice, a lot of workers would rather have a steady salary than deal with the uncertainties of the crowdsourcing industry
  • Workers seek transparency, fairness and respect for their work along with a fair wage
  • Turkers can discuss and review requesters and share their experiences with each other

Observations that can be drawn about requesters

  • Requesters have a goal to get good quality work results for their tasks
  • On AMT, requesters are the dominant side given the power imbalance and asymmetrical distribution of information in their favor. Eg: Ability of requester to block workers while the vice-versa is not possible
  • Some workers go the extra mile to clarify instructions or answer workers' queries about the task they have put up.
  • Some requesters communicate with workers on various forums on the internet

Worker perspective: Turkopticon

Observations about Workers

  • No proper channels for workers to address grievances
  • Skewed system where requesters get full access to the workers' submission irrespective of whether the requesters accepts or rejects the submission
  • Astounding impact of negative rating from a requester; lower ratings make it difficult to find good quality of work
  • The process of decision making w.r.t. the submissions is very opaque and workers seldom have any idea why their submissions were rejected
  • It provides a platform to review and rate the requesters in some form
  • Other workers' reviews about a requester is a significant variable in the decision making process on which tasks to work on
  • Amazon is oblivious or ignorant of workers' problems and does not have a reliable resolution mechanism for the same

Observations about requesters=

  • Quality of work is the major focus
  • Requesters that post huge batches of tasks find it difficult to deal with queries, complaints and suggestions from workers
  • They do not seem to have the complete visibility of the process from the workers' perspective
  • Requesters don't seem to realize that there is an actual "fear" of not getting paid among the workers

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

Observations about workers

  • The experiment reveals that a small section of workers who are looking for "A quick buck" are more ignorant and end up submitting bad quality results
  • The more detailed a task is, better the quality of results
  • Workers looking to take the system for a ride avoid tasks that ask for specific information
  • Gaming the system is more probable in a task with ambiguous instructions and poor design where task submissions don't have any verifiable outputs

Observations about requesters

  • Requesters have to rely on trivial tricks like finding tasks completed in shorter durations to take rejection decision in the absence of a verifiable output
  • Requesters can avoid bad quality results by avoiding poorly designed tasks

In my opinion, the following curve perfectly sums up the observations. Okay, it might be a bit of a stretch. :)

Dineshd Quality curve.png

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

List out the observations you made while doing your fieldwork. Links to examples (posts / threads) would be extremely helpful.

Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

A set of bullet points summarizing the needs of workers.

  • Workers need to paid a fair wage. A number of workers pointed out that even similar tasks that require similar skills vary greatly in terms of how much they pay. Requesters look at crowdsourcing as a way to get cheap labour.
  • Workers need the freedom to work on their own time. Eg:- Laura has to take care of her children and do the household chores. Flexibility of work hours seems to be something liked by a majority of the workers.
  • Workers need to lose fear of not getting paid. Eg:- Rochelle is hesitant about working with new requesters and has to confirm that there is a genuine requester. This could be a loss of good opportunity, but workers like to play it safe given that there are scams going around.
  • Workers need to be respected for their work. A mere worker ID does not define a worker. HITs are Human Intelligence Tasks. "Human"

Requester Needs

A set of bullet points summarizing the needs of requesters.

  • Example: requesters need to trust the results they get from workers. Evidence: In this thread on Reddit (linked), a requester is struggling to know which results to use and which ones to reject or re-post for more data. Interpretation: it's actually quite difficult for requesters to know whether 1) a worker tried hard but the question was unclear or very difficult or an edge case, or 2) a worker wasn't really putting in a best effort.

Milestone Contributors

Team Enigma

  • Archana Kumar - @dhankie
  • Dinesh Dhakal - @dineshd