Winter Milestone 2 presrini

From crowdresearch
Jump to: navigation, search

Observation from others' insights

Reading material covered --

Worker perspective: Being a Turker

Martin D, Hanrahan B V, O'Neill J, et al. Being a turker. Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 2014: 224-235.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

Kittur A, Chi E H, Suh B. Crowdsourcing user studies with Mechanical Turk. Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 2008: 453-456.

Requester perspective: The Need for Standardization in Crowdsourcing

The Need for Standardization in Crowdsourcing

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

A Plea to Amazon: Fix Mechanical Turk


1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

• Martin and colleagues (2014) identify are two kinds of workers: novice and expert.

• Money is a significant factor that is considered by workers before they choose a HIT.

• New workers tend to accept low paid, easy work from credible requesters to increase their HIT count. One of the goals of new workers is to increase number of completed and accepted HITs to obtain a good approval rating (desired value is >=90%). A good approval rating provides access to better paid and more professional, interesting work.

• The only factor used to differentiate good from bad workers is the approval rating system. There is no clear distinction between how the workers are paid based on their performance.

• More experienced workers might sacrifice pay to some extent if the work fits their preferences (easy to complete, interesting etc.)

• Workers perform a lot of work not visible to the requesters. Invisible work is unpaid work. Some examples of invisible activities include – searching for a preferred HIT, conversing with fellow workers on gathering knowledge about optimizing time spent working on HITs, sharing opinions on good/bad requesters and techniques to find finding HITs and so on.

• Workers prefer turking as – a means to make extra money, a job that does not involve transportation costs or judgments by requesters on their personality, how they look or are dressed.

• Workers try to motivate themselves with a set target of earnings based on their ability to complete tasks and life circumstances.

• Workers are offended if their work is rejected with improper communication of why their work was rejected. Workers also pay attention to the tone of comments posted by the requesters on rejecting their work.

• Workers consider the lack of being visible in the labor market as a benefit – they prefer the anonymity, flexibility to work on their preferences on requester, the type of HITs, its pay, and alignment with their interests.

• Workers are open to communicating and cooperating with requesters in making the HITs design better. An emotional and moral involvement can develop over time, resulting in the establishment of a relationship between workers and requesters.

• Workers are ethical with respect to categorizing a requester as good or bad. Good requesters are those that provide fair pay, have increased responsiveness on accepting/rejecting their work, is sensible while rejecting their work, and explains clearly when a work is rejected. Bad requesters are those that perform mass rejection and providing demeaning comments to the workers. Workers tend to complete multiple HITs for requesters they consider or have received feedback from fellow workers as “good requester”.

• Workers prefer self-governing the marketplace. They are active in performing a collective and cooperative action by choosing what jobs to do and what pay levels to accept. They do not want someone else to make decisions on their behalf.

• Workers tend to share their experience with fellow workers, thus establishing a collective community – especially with respect to – their extrapolated opinion of and experience with requesters, the responsiveness of the requesters to their completed effort, and their judgment on whether the pay is fair or not. Workers also share their problems (both with life circumstance and AMT experience) and look to fellow workers for support or advice.

• Workers are not uniformly good at all types of tasks.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

• Requesters tend to suspect work that is completed multiple times or within a very short duration.

• Requesters consider detecting malicious worker behavior as time consuming even though a reward is not provided to the worker.

• Requesters tend to avoid workers providing invalid responses by creating HITs that are as effortful as completing the actual task.

• Requesters add questions that require workers to pay attention to the tasks and provide responses that can be verified against a set of already known answers. This is believed to increased worker performance and quickly identify malicious worker.

• Requesters view workers as novices for the tasks they generate. Workers are identified as being a part of a widespread user population from a diverse geographical diversity.

• Requesters tend to accept work based on “majority rule” – which refers to HITs that are only accepted by the Requester if the worker has responded with the same answer as ‘the majority’ of those doing the same HIT. This is a common way to achieve good quality in tasks.

• Workers sharing bad reviews about requesters with fellow workers without consideration can negatively impact the workers community – this act can put off workers from working for decent requesters.

• The requesters who have been in the system for a long time learn from their mistakes and fix the design problems.

• Requesters often have the following problems requiring time and effort – learning to understand using the API provided by AMT, estimating the time taken for a HIT to be completed, identifying if a worker qualifies to complete the HIT, ensuring the HIT is broken up into meaningful tasks that are easily understood by workers, and ensuring the quality of workers’ work.

• There is no time limit within which requesters have to pay the workers for their work.

• Requesters have the authority to reject a worker’s work. Too many blocks from a requester can remove a worker from the AMT system. However, workers cannot block requesters in a similar manner other than avoiding or publicizing other fellow workers not to complete HITs posted by the requester.

• New requesters are evaluated and treated with caution by the experienced, good workers. Most often, good workers complete some HITs posted by new requesters and observe how the requester behaves.

Synthesized Needs

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

• Workers need to see the overall rate at which a requester had rejected workers in the past. In addition, workers need to see the rate at which a requester had rejected workers in the past for the specific type of task.

• Workers need to see the progress of their work being checked – from the time they submitted the work to the time the work is accepted and paid.

• Workers need to be evaluated based on their quality of work for that task that they submitted.

• Workers need to be paid a percentage of the set payment based on their quality of performance. (Workers needn’t have to be evaluated based only on their approval ratings since the quality of work can vary based on the type of HIT. A honest worker shouldn’t be penalized for a task he/she is not familiar with or lacks knowledge or understanding for the HIT. Further, such a honest worker’s work should not be rejected without pay since it can lead to a breakdown of relationship between the requester and the worker. Workers need to be evaluated and paid based on their performance specific to the HIT in addition to their overall performance for all the HITs they had completed in the past).

• Workers need to optimize their invisible activities such as searching for trusted requester, identifying tasks of interest etc.

• Workers need to categorize requesters based on quality of HITs posted, payment, and so on.

• Requesters need to identify different types of workers based on the quality of work specific to the task and overall performance for the time the workers have been in the marketplace.

• Requesters need to be given a chance to correct themselves over time (Kittur and colleagues (2008) were able to learn from their mistake and modify the HIT design and obtain better results – possibly an implication that requesters do not necessarily need to communicate with workers to improve HIT design. Conversely, Martin and colleagues (2014) discuss how requesters communicate with workers on forums to modify the HIT designs. Further, Ipeirotis (2010) discusses how new requesters can look out for help if they can afford it).

• Requesters need to determine an overall payment for a HIT and pay workers based on the percentage quality of performance.

• Requesters need to build trust with workers over an experimentation or probation time period.

• Requesters need to post additional verifiable tasks that help ascertain quality of work.