WinterMilestone 2 Enigma
- 1 Attend a Panel to Hear from Workers and Requesters
- 2 Reading Others' Insights
- 2.1 Worker perspective: Being a Turker
- 2.2 Worker perspective: Turkopticon
- 2.3 Requester perspective: Crowdsourcing User Studies with Mechanical Turk
- 3 Synthesize the Needs You Found
- 4 Milestone Contributors
Attend a Panel to Hear from Workers and Requesters
The interaction with the panel was very fruitful. The following are some of the observations we could make
- The reasons for workers to use MTurk are quite varied.
- Some workers like the flexibility that it offers. They seemed to like the idea of being in control of their working hours
- Another major factor is money.
- Some workers are employed elsewhere and use MTurk as an alternative source of income
- The working hours also depend on the availability of good work.
- Unpredictable work days are common. "One day you end up making considerable amount of money and the very next day you'll be struggling to make a dollar"
- Workers feel there is a lack of respect for them and their work
- Some workers pointed out that there was no definitive way of knowing when a good work shows up unless you keep monitoring continuously
- There is an active community of workers that communicate through a number of channels like TurkNation etc.
- They share with each other promising HITs and other interesting work, suggestions, tips and tricks and often help each other out
- There is a considerable barrier for new workers who don't already have good ratings as requesters might be hesitant to let new workers work on their HITs
- Unclear or ambiguous instructions from the requesters is a challenge for workers
- Rejection of work without explanation or any kind of feedback is disheartening.
- Different workers have different ways of picking what to work on. Eg: Well rated tasks, Reward vs Time comparison etc.
- The major concern for requesters is the quality of work they get
- Requesters are also worried about workers dropping out of tasks
- Sometimes they face difficulties in finding qualified workers who would deliver good quality results
- Finding replacements when certain workers drop out at critical stages is a challenge for many requesters
- Workers are mostly unknown and sandboxed in a sense; no proper channel for instant communication with the workers
- Dealing with poor quality of work compels the requesters to increase the qualification criteria for their future HITs
Reading Others' Insights
We attended the requester and worker panel to gain a better understanding of what actually goes through the entire process.
Worker perspective: Being a Turker
- In spite of a presence of a large number of workers only about 20% of them do the 80% of the work
- The major drive for the workers is money
- For a worker the wages are highly unpredictable. Some days are highly unproductive while other might be rushed. Although they set targets for themselves, they cannot be certain about achieving them
- There is a small set of workers who work with a perspective to learn or to gain some skill sets
- There is a good deal of pre-processing involved for the workers in terms of task selection
- The requester demands are sometimes unclear. As a result the worker does not get clear instructions. A lot of time is spent trying to understand and get clarity on tasks.
- If the worker finishes up early or late as expected by requester, the requester might doubt the quality of work.
- The work available on the platform might not be consistent. The worker may have to work at odd hours if he is really keen on finishing up the work.
- Sometimes the user does not get any feedback and just a rejection hence all the effort put by the worker gets wasted. Not getting paid for their work is the biggest fear for the majority of the workers
- Given a choice, a lot of workers would rather have a steady salary than deal with the uncertainties of the crowdsourcing industry
- Workers seek transparency, fairness and respect for their work along with a fair wage
- Turkers can discuss and review requesters and share their experiences with each other
Observations that can be drawn about requesters
- Requesters have a goal to get good quality work results for their tasks
- On AMT, requesters are the dominant side given the power imbalance and asymmetrical distribution of information in their favor. Eg: Ability of requester to block workers while the vice-versa is not possible
- Some workers go the extra mile to clarify instructions or answer workers' queries about the task they have put up.
- Some requesters communicate with workers on various forums on the internet
Worker perspective: Turkopticon
Observations about Workers
- No proper channels for workers to address grievances
- Skewed system where requesters get full access to the workers' submission irrespective of whether the requesters accepts or rejects the submission
- Astounding impact of negative rating from a requester; lower ratings make it difficult to find good quality of work
- The process of decision making w.r.t. the submissions is very opaque and workers seldom have any idea why their submissions were rejected
- It provides a platform to review and rate the requesters in some form
- Other workers' reviews about a requester is a significant variable in the decision making process on which tasks to work on
- Amazon is oblivious or ignorant of workers' problems and does not have a reliable resolution mechanism for the same
Observations about requesters=
- Quality of work is the major focus
- Requesters that post huge batches of tasks find it difficult to deal with queries, complaints and suggestions from workers
- They do not seem to have the complete visibility of the process from the workers' perspective
- Requesters don't seem to realize that there is an actual "fear" of not getting paid among the workers
Requester perspective: Crowdsourcing User Studies with Mechanical Turk
Observations about workers
- The experiment reveals that a small section of workers who are looking for "A quick buck" are more ignorant and end up submitting bad quality results
- The more detailed a task is, better the quality of results
- Workers looking to take the system for a ride avoid tasks that ask for specific information
- Gaming the system is more probable in a task with ambiguous instructions and poor design where task submissions don't have any verifiable outputs
Observations about requesters
- Requesters have to rely on trivial tricks like finding tasks completed in shorter durations to take rejection decision in the absence of a verifiable output
- Requesters can avoid bad quality results by avoiding poorly designed tasks
In my opinion, the following curve perfectly sums up the observations. Okay, it might be a bit of a stretch. :)
Synthesize the Needs You Found
List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.
A set of bullet points summarizing the needs of workers.
- Workers need to paid a fair wage. A number of workers pointed out that even similar tasks that require similar skills vary greatly in terms of how much they pay. Rochelle mentioned how a fair pay per task might help rather than trying to establish a minimum wage per hour. Requesters look at crowdsourcing as a way to get cheap labour.
- Workers need the freedom to work on their own time. Eg:- Laura has to take care of her children and do the household chores. Flexibility of work hours seems to be something liked by a majority of the workers.
- Workers need to lose fear of not getting paid. Eg:- Rochelle is hesitant about working with new requesters and has to confirm that there is a genuine requester. This could be a loss of good opportunity, but workers like to play it safe given that there are scams going around.
- Workers need to be respected for their work. There should be a mutual trust between the workers and the requesters. A mere worker ID does not define a worker. HITs are Human Intelligence Tasks. Human.
- Workers need a better way to report and resolve grievances. Eg:- Chrissie expresses her dissatisfaction by saying that the workers' perspective is irrelevant. In times of difference in opinion, there has to be a scalable mechanism to resolve these differences. Amazon seems to simply ignore this as highlighted by our workers panels and also on TurkerNation.
- Workers need more information about the tasks requested, ratings, statistics and choices natively on the platform. Eg:- Laura uses scripts to find relevant work while doing household chores. A lot of other users use scripts for monitoring, data gathering and analyzing the HITs they've worked on. There is a disparity in the amount of data available to requesters and to workers.
A set of bullet points summarizing the needs of requesters.
- Requesters need good quality of results. Eg:- Peter started with enforcing quality checks in the beginning, but later put up quality restrictions and started automatic acceptance of tasks. Either way it is of upmost importance that the quality of results is maintained.
- Requesters need reliable workers. Eg:- Shawn had brought up how workers dropping out of a task makes it difficult for them to find replacements in critical situations. This is even more difficult when tasks require certain specific qualifications.
- Requesters need a way to get feedback from workers and improve the tasks on the go. Eg:- Chris posts a batch of 10 tasks to get a feedback on what can be improved. Sometimes even the requesters are not certain what might work and what might not. It is important to get the other side's perspective.
- Requesters need a way to better interact with workers. Eg:- Shawn goes on to the TurkerNation to interact with the workers, get to know them better and try to help them out and listen to them.
- Requesters do not want to reject work unfairly. Eg:- As Christ, Peter and Shawn have all pointed out that they try their own ways and workarounds to get feedback from workers, to help them with any queries, clearly they want the workers to work better and be more efficient. Workers being happy and well paid is mutually beneficial.
- Archana Kumar - @dhankie
- Dinesh Dhakal - @dineshd