WinterMilestone 2 Crayons

From crowdresearch
Jump to: navigation, search


Attend a Panel to Hear from Workers and Requesters


The hangout session was kind of interaction between workers and requester.The set was like a online round table conference. All were introduced one by one .Both workers and requester were found enthusiastic enough to showcase their experience as a worker and requester.

The people who were on hangouts were of different field and work. Some were professors at elegant universities working on their research and some were their to make their work easy by easily accessing Mturk to get the crowd for there work from the requester side(Christ,Xaio,Peter). As workers(Laura,Kristene) there were people like who were already employed but use mTurk as an alternative source of income, and people who are totally dependent on mTurk for their whole income.

Some of the folk told about the kind of work they are dealing with.There is work is done through Tagging Images, Collecting Directive Data , working on Survey Data and with Research Perspective too.People shared their experience of working in mTurk.

They talked about problems they face while working on mTurk like-'

  1. High drop out from time to time ,its real problem both sides
  2. Large variation in wages from day to day.One may have planned to earn enough to do sort of earning but end up in earning just pennies.
  3. You need to complete the hit even when you have other important work to do, here you cant delay you only have to hit and win or leave and loose.

(eh its time for kid care,its time for teaching,its time for lunch etc)

  1. You have to sacrifice a lot to go with the hit.

They talked about how they find work:-

  1. Some go by network, by rating , by organization they have already worked for, pay, time and also by good organizations. The people that are quite cooperative and like to work in an environment where workers also helps each other, if someone is stuck. Workers generally admire and encourage to work with ethical behavior and where there easy reach out to the requesters.
  2. Choose only from rated things and email if you don't get the instructions
  3. Estimate how long its going it take from your prior experience.
  4. Look up your already work done tables , and go for what suits them best.

They also shared the time of day work on mTurk-:

  1. It depends on person to person . Hits are 24*7 on and so workers can any time that comforts them. Working can also be decided by the type of work you want or the work that interest you or may be there is very high paying hit , so you may decide to leave everything to go work or may be you take a leave , its all kind of decision one have to make .
  2. Its never the time or situation when there is nothing posted on ,its only when and what you are hitting.
  3. Large individual variation based on task or typing speed and that your hourly is something that can be tracked but it may be hard to make generalization.


mTurk is a great source help when we are doing some research. You need to go and find lots of crowd when its available online with a little pay and worth work.Quickly test our hypothesis and design experiment in a day and sense whether it makes sense or whether it work by deploying it live. Its online community and crowd sourcing and it found to have high turn over rate is very high and also a good source of income. Inspite all help there are lots of problem that workers and requesters are facing .There are problems like

Worker side

  1. High drop out from time to time
  2. There lot of variation in wages.
  3. Lack of transparency between workers and requesters.
  4. No platform so that workers and requesters can interact directly on mTurk.
  5. Unexpected rejection without proper explanation.
  6. Poor presentation of instructions for the workers.

Requesters side

  1. There are dealing with lots of poor quality work .
  2. Poor interaction with workers.
  3. There is also transparency problem to deal with ,some folks are suspected to be cheating.



  1. You need to properly understand the instruction before hitting because it might effect your reputation.
  2. Also look for the qualification for the work to be done.
  3. Workers need to estimate the difficulty of the hit from the feedback of others and then estimate the possibility if yours doing it.
  4. Workers need to know there potential and go on hit according to save there time or make income some other it.
  5. Worker need to maintain a sort of database of their prior work.For example-You may also maintain a spread sheet documenting all you work, time taken, pay , got rejected etc. So that in future you are prevented from doing such sort of work and that could avoid rejection.
  6. There is need of a more transparent platform for interaction to requesters.


  1. Requesters need to make some sort of threshold for acceptance of work and with proper explanation for the rejection of the work so that worker feel satisfied.
  2. There is a need of a proper platform to interact with workers so that they can direct them while workers are working on problem for a better quality of works.

Reading Others' Insights

Worker perspective: Being a Turker

1. What observations about Workers can you draw from the readings?

  1. There are about 500,000 workers on Amazon Mechanical Turk(AMT). Although a study shows that only around 10% workers are active.
  2. Majority (80 percent) of the tasks are carried out by 20 percent most active workers.
  3. Workers are primarily from US(56%) and India(36%).
Why do turkers turk?
  1. Monetary gains are the primary reason why turkers turk.
  2. Some workers may also prefer to do work which they find interesting or which increases their knowledge or skills, however these comprise just a small section of the reasons chart dominated by money.
How much do they earn?
  1. Earning depend highly on the number of hours workers contribute and whether AMT is their prime source of income, their source of survival, or they just work part-time or just for fun.
  2. Skill set of the workers also affect the earnings.
  3. Even the earning of the most experienced workers just touches the mark set by minimum wages.
  4. Workers set targets for themselves.
Relation with Requesters
  1. A major section of the Turker nation forum is dedicated to Workers-Requesters relations.
  2. Workers post reviews about their experiences with requesters. Reviews may consist of how well the requester communicated the task, whether the workers were paid well and on time etc.
  3. Workers do some initial search about the requester before taking his/her job.
  1. Majority of the turkers oppose the idea to regularize the AMT platform.
  2. They believe that they are in the best position to influence and manage the market.
  3. They also share a strong opinion against the attention given to the crowdsource industry by journalists and academicians as they fear this would demotivate requesters and amount of work would reduce.
The Invisible Work
  1. Workers spent a good amount of time in doing the work which is hiden from the outside world.
  2. This includes finding the most suitable HITs in terms of payment, their knowledge and other hardware or software restrictions, searching about reputation of the requesters, learn new skills, manage AMT work etc.
Major concerns
  1. The main concerns for the workers are employers who don't pay, unfair rejections, identifying scams, the cost of poorly designed tasks.

2. What observations about Requesters can you draw from the readings?

Getting the job done
  1. The main focus of the Requesters is to get the job done as quickly as possible within the budget.
  2. Sometimes they don't even shy from being unfair to the workers.
  3. They rate workers on the basis of the work. They can even block the workers. Although the same option is not provided to the workers.
  1. Some requesters follow forums like Turker Nation to communicate with workers during the work.
  2. They remain online during the task to care of any problems that the workers might face.

Worker perspective: Turkopticon

Turkopticon is developed in response to invisibility of worker in AMT design. Turkopticon is a system that was created to allow workers to publicize and evaluate their relationships with employers since workers need to know who are the bad ones and who are the good ones. That is, in essence, Turkopticon is a platform to rate employers.

  1. Low income due to losses because of arbitrary rejection of work they did.
  2. Low income for workers because the do not get fair compensation for their work.
  3. Workers are paid late because of considerable delay in acknowledgement of the task payments from requester.
  4. Workers do the work even without assurances of getting payment in return.
  5. Rejection of work leads to lowering of approval ratings.
  6. Low ratings of workers in AMT results in inaccessibility for higher rating work.
  7. Amazon response to workers complaints are not up to mark.
  8. No feedback mechanism for worker to rate requester/employer, no rating mechanism for workers to rate employers
  9. The communication between the worker and a requester is inefficient.
  1. Requester/employer do not have to provide valid reason for rejection of work.
  2. Although payment rejected by the employer, they(employer) still own the work which was provided to them.
  3. Requester/employers are given a window of 26 days to evaluate workers task.
  4. Difficult for requesters/employers to rate task which are subjective in nature.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

User studies are an important part of the design process. It can help to improve the design by providing relevant inputs and feedback. Mechanical Turk is a platform which provides a low cost alternative to requesters to collect data from users with different backgrounds.

As a part of this study, two experiments were conducted:

Experiment 1 : Users were asked to evaluate Wikipedia article on a scale of 7 with optional text feedback. The feedback aimed at finding the veracity of ratings provided by users.

Experiment 2 : This experiment was on the same lines with one addition. A verifiable questionnaire was included to reduce the number of malicious users.


  1. Malicious workers may try to enter fake input without going through the data just to complete the tasks in short duration of time.
  2. Such practices increase with the tasks where the output is not verifiable.


  1. Without verifiable output, requesters cannot check the authenticity of the task.
  2. Workers have to reply on factors like task completion duration, plagiarism etc to separate valid and invalid inputs.
  3. Workers may add verifiable questionnaire to discourage malicious users.

Requester perspective: The Need for Standardization in Crowdsourcing


  • Workers tend to work on similar tasks
  • Tasks of some complexity are comprised of tasks anbalogous to building blocks
  • Workers don't get payed for HITs they completed if the work is not to the requesters liking.
  • Workers are free to choose any tasks varying in of level of difficulty and skills required to complete each task.


  • Requester request similar tasks.
  • Wide range on the rewards for similar task.
  • Current crowd-labor market which is unorganized due to a lack of standardization or proper handling of negative externalities.
  • Requesters are sometime scammers.
  • Requesters cannot rely on the quality of the task.

Both Perspectives: A Plea to Amazon: Fix Mechanical Turk


Issue 1: Posting tasks and creation of workflows
  1. Might need to hire a full-time developer to deal with the complexities of the system.
  2. Learning to break tasks into a workflow.
  3. Stratify workers, according to quality.
  4. Have to build their own interfaces, workflow systems and quality assurance systems from scratch.
  1. Command line tools to post tasks considered user-friendly by the platform.
  2. No easy implementation of workflows.
  3. Most have crowd-sourced workflows instead of one-pass tasks.
  4. Only a few "big requesters" and very many "small requesters".
  5. Difficult for small guys to grow.
Issue 2: Bad reputation system for workers
  1. Increased gaming of system by the workers.
  2. Requesters tend to think that every worker is bad.
  3. New requesters then get only low quality workers, get disappointed with the quality of the results and they leave the market.
  1. "Number of completed HITs" and "approval rate" are easy to game by spammer workers.
  2. Good workers leaving the market.
  3. New requesters leaving the market
Issue 2: Quality Assurance
  1. Get multiple, redundant answers for the same question.
  2. Qualification tests, testing if they the workers were competent enough to participate.
  3. To make sure the instructions were followed, users were asked to submit the answers to already completed tasks.
  1. Uncertainty about the validity of the submitted answers.
  2. Clarified in the instructions that we will pay only for submissions that agree with the responses submitted by other workers.
  3. Increased the costs.
  4. Slows down the process.


Issue 1: A Trustworthiness Guarantee for Requesters
  1. Scam requesters post HITs, behave badly.
  2. Requesters can reject good work and not pay for the work they get to keep. Requesters do not have to pay on time.
  3. The new requester treated with caution until he becomes trustworthy.
  4. Good workers do a very small number of tasks to see if the new requester is trustworthy.
  1. Cause good workers to avoid any newcomer.
  2. Turker Nation and TurkOpticon, make it possible to know about the about the badly behaving requesters.
  3. New requesters are satisfied if they post only small batches of work.
  4. New requesters posting large batches are often disappointed as the large subset of the work is done by the spammers.
  5. Subjective reputation is not enough
Issue 2: Restrictive user interface
  1. Only 2 ways of sorting, the most recent HITs, or the HITgroups with the most HITs.
  2. Workers use priority queues to pick the tasks to work on.
  1. Highly restricted by the interface; cannot search for a requester, unless the requester put their name in the keywords; no way to find the tasks of interest.
  2. Workers use priority queues to pick the tasks to work on.
  3. It is effectively impossible to predict the completion time of the posted tasks (the mean completion time is expected to increase continuously as we observe the market for longer periods of time).

Synthesize the Needs You Found


Good workers need a method to signal to the buyer their higher quality


A plea to Amazon, states very clearly, how ineffective signalling mechanisms - "Number of HITs worked" and the "acceptance percentage" - are.


These, in no way tell the requester about the skills or effectiveness of the worker. This point is very similar to the "better reputation" point in the requester section. Hence, there is a need for a better worker profile on both - worker and requester - sides.

Workers need to have a trustworthiness guarantee for the requesters

  1. The subjective reputation of the requesters is not enough. (A plea to Amazon)
  2. Largest section of Turker Nation forum is dedicated to Turker-Requester relationship, where Turkers provide their reviews about requesters and do some initial search about requesters before taking up their tasks. (Being a Turker)

Good workers are always vary of the new requesters in the present state of things. The worker should see a set of objective characteristics of the requester, and decide whether to pick a specific HIT or not. Characteristics like - speed of payment, rejection rate, appeal rate, showing the total volume of posted work - are much better modes of instilling a sense of trust among the workers. This need reduces the search costs to get a worker who will complete the work and to get the requester who does not behave unreasonably.

Workers need to have a better user interface


Currently, the interfaces they work on are very restricted. They can't search the tasks which will be best fit to their skilsets. Thus, the way they pick tasks to work on, is based on the priority - most recent HITs, or the HITgroups with the most HITs. This makes it impossible to predict the completion time of a task as, theoretically speaking, it is infinity.


Under the current conditions, requester can't even predict the average waiting time, let alone, the task completion time. The theoretical average is infinite.

Workers need to communicate with the requesters


jenny492 posted in Turker Nation, "I just started working on his hits this week. I've probably done several hundred of his .15 hits. They all approved right away, and he got back to me quickly when I had a question. Thumbs up from me " Buffy posted in Turker Nation," Evidently he learned from earlier experience. I have been doing work for him since last month. He is just super nice, normally online while the work is going on and answers any questions right away. The HITs are now paying $0.15 each. My favorite requester."


Requesters who communicate well are appreciated by workers.

Workers need to be heard against unfair blockings by requesters


Requesters can bar a Turker from working for them. Turkers do not have a reciprocal system action to blocking and it is complicated for them to prove their innocence. (Being a Turker)

Workers need to have fair rejections with properly explained reasons


"Got a mass rejection from some hits I did for them! Talked to other turkers that I know in real life and the same thing happened to them. There rejection comments are also really demeaning. Definitely avoid!" (Posted in Turker Nation by neilrsj)


Workers need to keep the market free from interference by society and govt.


"I don't like it. Another idiot professor who thinks he knows what's best for the private market. This will only mean the government getting involved and regulating the requester's which in turn will end up in less pay for us. Someone please tell this idiot professor to stay in the classroom. " (Posted in Turker nation by general65).


Better interface needed to post the tasks


In A plea to Amazon, the author talks at length about the difficulties requesters face in posting the tasks. How every worker has to build a quality assurance system, learn to break tasks into a workflow, etc. This barrier has created a sort of segmentation in the types of requesters on the platform - a few "big requesters" and a long tail of "small requesters".


The obvious reason being, big requesters have the resources to produce complex crowd-sourced workflows and the small ones just post a 1-pass tasks so they don't actually need such complex workflows. Present interface doesn't let the new/small requesters to grow.

Requesters need a better reputation profile for the workers


In A plea to Amazon, the author showed that "number of completed HITs" and "approval rate" are easy to game and these metrics are the only ones through which a requester can differentiate between a good and a bad worker.


Without a reputation system the requester thinks that all workers are bad and sets wages according to the average workers. This in turn disappoints the good workers who then leave the platform. Same with the new requesters, who, when receive bad quality results, they too leave the platform. There is a need for public qualification tests, keeping track of working history, rating of workers and make these things accessible from API to make automatic decisions.

Requesters need to trust the users' submissions


In the Experiment 1, the submissions were found to be less accurate when compared with the expert opinions, indicating fake data. However, in the experiment 2 when some verifiable questions were also added, results improved. (Crowdsourcing User Studies With Mechanical Turk)

Requester need to evaluate the tasks quickly


Workers’ responses to the question of a “Bill of Rights” revealed a range of concerns, one of them being the demand faster payment to the workers (Amazon allows employers 30-days to evaluate and pay for work)

Milestone Contributors

  • Ameen Mohammed Khan @ameenkhan07
  • Farheen Nilofer @farheen
  • Sachin Sharma @beingcooper
  • Sarah Masud @sarah
  • Shivam Rana @tminima