WinterMilestone 2 Crayons

From crowdresearch
Revision as of 08:05, 24 January 2016 by Ameenmkhan (Talk | contribs) (Worker perspective: Turkopticon)

Jump to: navigation, search

Attend a Panel to Hear from Workers and Requesters


Report on some of the observations you gathered during the panel.

Reading Others' Insights

Worker perspective: Being a Turker

1. What observations about Workers can you draw from the readings?

  1. There are about 500,000 workers on Amazon Mechanical Turk(AMT). Although a study shows that only around 10% workers are active.
  2. Majority (80 percent) of the tasks are carried out by 20 percent most active workers.
  3. Workers are primarily from US(56%) and India(36%).
Why do turkers turk?
  1. Monetary gains are the primary reason why turkers turk.
  2. Some workers may also prefer to do work which they find interesting or which increases their knowledge or skills, however these comprise just a small section of the reasons chart dominated by money.
How much do they earn?
  1. Earning depend highly on the number of hours workers contribute and whether AMT is their prime source of income, their source of survival, or they just work part-time or just for fun.
  2. Skill set of the workers also affect the earnings.
  3. Even the earning of the most experienced workers just touches the mark set by minimum wages.
  4. Workers set targets for themselves.
Relation with Requesters
  1. A major section of the Turker nation forum is dedicated to Workers-Requesters relations.
  2. Workers post reviews about their experiences with requesters. Reviews may consist of how well the requester communicated the task, whether the workers were paid well and on time etc.
  3. Workers do some initial search about the requester before taking his/her job.
  1. Majority of the turkers oppose the idea to regularize the AMT platform.
  2. They believe that they are in the best position to influence and manage the market.
  3. They also share a strong opinion against the attention given to the crowdsource industry by journalists and academicians as they fear this would demotivate requesters and amount of work would reduce.
The Invisible Work
  1. Workers spent a good amount of time in doing the work which is hiden from the outside world.
  2. This includes finding the most suitable HITs in terms of payment, their knowledge and other hardware or software restrictions, searching about reputation of the requesters, learn new skills, manage AMT work etc.
Major concerns
  1. The main concerns for the workers are employers who don't pay, unfair rejections, identifying scams, the cost of poorly designed tasks.

2. What observations about Requesters can you draw from the readings?

Getting the job done
  1. The main focus of the Requesters is to get the job done as quickly as possible within the budget.
  2. Sometimes they don't even shy from being unfair to the workers.
  3. They rate workers on the basis of the work. They can even block the workers. Although the same option is not provided to the workers.
  1. Some requesters follow forums like Turker Nation to communicate with workers during the work.
  2. They remain online during the task to care of any problems that the workers might face.

Worker perspective: Turkopticon

Turkopticon is developed in response to invisibility of worker in AMT design. Turkopticon is a system that was created to allow workers to publicize and evaluate their relationships with employers since workers need to know who are the bad ones and who are the good ones. That is, in essence, Turkopticon is a platform to rate employers.

1. What observations about Workers can you draw from the readings?

  • Low income due to losses because of arbitrary rejection of work they did.
  • Low income for workers because the do not get fair compensation for their work.
  • Workers are paid late because of considerable delay in acknowledgement of the task payments from requester.
  • Workers do the work even without assurances of getting payment in return.
  • Rejection of work leads to lowering of approval ratings.
  • Low ratings of workers in AMT results in inaccessibility for higher rating work.
  • Amazon response to workers complaints are not up to mark.
  • No feedback mechanism for worker to rate requester/employer, no rating mechanism for workers to rate employers
  • The communication between the worker and a requester is inefficient.

2. What observations about Requesters can you draw from the readings?

  • Requester/employer do not have to provide valid reason for rejection of work.
  • Although payment rejected by the employer, they(employer) still own the work which was provided to them.
  • Requester/employers are given a window of 26 days to evaluate workers task.
  • Difficult for requesters/employers to rate task which are subjective in nature.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

User studies are an important part of the design process. It can help to improve the design by providing relevant inputs and feedback. Mechanical Turk is a platform which provides a low cost alternative to requesters to collect data from users with different backgrounds.

As a part of this study, two experiments were conducted:

Experiment 1 : Users were asked to evaluate Wikipedia article on a scale of 7 with optional text feedback. The feedback aimed at finding the veracity of ratings provided by users.

Experiment 2 : This experiment was on the same lines with one addition. A verifiable questionnaire was included to reduce the number of malicious users.

1. What observations about Workers can you draw from the readings?

  1. Malicious workers may try to enter fake input without going through the data just to complete the tasks in short duration of time.
  2. Such practices increase with the tasks where the output is not verifiable.

2. What observations about Requesters can you draw from the readings?

  1. Without verifiable output, requesters cannot check the authenticity of the task.
  2. Workers have to reply on factors like task completion duration, plagiarism etc to separate valid and invalid inputs.
  3. Workers may add verifiable questionnaire to discourage malicious users.

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

A Plea to Amazon: Fix Mechanical Turk


Issue 1: Posting tasks and creation of workflows
  1. Might need to hire a full-time developer to deal with the complexities of the system.
  2. Learning to break tasks into a workflow.
  3. Stratify workers, according to quality.
  4. Have to build their own interfaces, workflow systems and quality assurance systems from scratch.
  1. Command line tools to post tasks considered user-friendly by the platform.
  2. No easy implementation of workflows.
  3. Most have crowd-sourced workflows instead of one-pass tasks.
  4. Only a few "big requesters" and very many "small requesters".
  5. Difficult for small guys to grow.
Issue 2: Bad reputation system for workers
  1. Increased gaming of system by the workers.
  2. When no differentiation can be made in the quality of workers, requesters tend to think that every worker is bad.
  3. New requesters then get only low quality workers, get disappointed with the quality of the results and they leave the market.
  1. "Number of completed HITs" and "approval rate" are easy to game by spammer workers.
  2. Good workers leaving the market.
  3. New requesters leaving the market
Issue 2: Quality Assurance
  1. Get multiple, redundant answers for the same question.
  2. Qualification tests, testing if they the workers were competent enough to participate.
  3. To make sure the instructions were followed, users were asked to submit the answers to already completed tasks.
  1. Uncertainty about the validity of the submitted answers.
  2. Clarified in the instructions that we will pay only for submissions that agree with the responses submitted by other workers.
  3. Increased the costs.
  4. Slows down the process.


Issue 1: A Trustworthiness Guarantee for Requesters
  1. Scam requesters post HITs, behave badly.
  2. Requesters can reject good work and not pay for the work they get to keep. Requesters do not have to pay on time.
  3. The new requester treated with caution until he becomes trustworthy.
  4. Good workers do a very small number of tasks to see if the new requester is trustworthy.
  1. Cause good workers to avoid any newcomer.
  2. Turker Nation and TurkOpticon, make it possible to know about the about the badly behaving requesters.
  3. New requesters are satisfied if they post only small batches of work.
  4. New requesters posting large batches are often disappointed as the large subset of the work is done by the spammers.
  5. Subjective reputation is not enough
Issue 2: A Trustworthiness Guarantee for Requesters
  1. Only 2 ways of sorting, the most recent HITs, or the HITgroups with the most HITs.
  2. Workers use priority queues to pick the tasks to work on.
  1. Highly restricted by the interface; cannot search for a requester, unless the requester put their name in the keywords; no way to find the tasks of interest.
  2. Workers use priority queues to pick the tasks to work on.
  3. It is effectively impossible to predict the completion time of the posted tasks (the mean completion time is expected to increase continuously as we observe the market for longer periods of time).

Synthesize the Needs You Found

List out your most salient and interesting needs of workers, and of requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.


  • Example: Workers need to be respected by their employers. Evidence: Sanjay said in the worker panel that he wrote an angry email to a requester who mass-rejected his work. Interpretation: this wasn't actually about the money; it was about the disregard for Sanjay's work ethic.
  • Good workers need a method to signal to the buyer their higher quality. "Number of HITs worked" and the "acceptance percentage" are simply not sufficient signalling mechanisms.
  • Trustworthiness guarantee of the requesters.


  • Example: requesters need to trust the results they get from workers. Evidence: In this thread on Reddit (linked), a requester is struggling to know which results to use and which ones to reject or re-post for more data. Interpretation: it's actually quite difficult for requesters to know whether 1) a worker tried hard but the question was unclear or very difficult or an edge case, or 2) a worker wasn't really putting in a best effort.
  • Better interface to post the tasks.
  • Requesters need a better reputation profile for the workers.