Milestone 2 pixelpals

From crowdresearch
Jump to: navigation, search

SUBMISSION OF MILESTONE 2

Attend a Panel to Hear from Workers and Requesters

Workers Side

  • Workers extensively use various forums and to discuss any and all aspects related to issues they face. They communicate about their experience and help each other resolve everyday issues.
  • Everybody has disappointing days, even the most experienced workers.
  • User – testing videos are easier and less competitive.
  • Sending Mail to requesters upon rejection is now become a tradition.
  • There exists a very strong-knit co-worker community in which they collaborate and communicate with them. Eg Facebook group for Turk-Workers is a very popular.
  • Motivation goes beyond money in various cases and that becomes the sticky factor that allows users to stay connected with the crowd sourcing platforms even in disappointing times.

Requester Side

  • Pricing a HIT is generally based on the number of hours they expect the workers to take to do the task. Since 8-10 dollars per hour is the prevailing price the pricing the task by the requesters itself can give a big hint to the workers about the number of hours you expect.
  • Putting in a open ended question to differentiate b/w genuine workers and spammers is a more effective and commonly used way as compared to putting in “Gold Standard Questions”
  • Putting in a script to find out time spend by workers on individual pages in task is an interesting way to find outliers to your task.
  • M-Turk restriction to Not allow workers to “Download” tools to complete the task is a very big disadvantage to the requesters
  • Bonus/incentives that are not relived before hand to workers can be a really quick and interesting way to remove workers who do just for money.

Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Workers turk on AMT primarily for money.
  • They use forums like Turk Nation to share their problems with people looking for advice and support.
  • They prefer platforms like AMT because there are no set hours, no transportation money is required and they get the flexibility of working from home.
  • Workers judge requesters not only on the basis of the money they pay but also on the responsiveness of their evaluation and the rating they give to the turkers.
  • They don’t want to be unfairly treated by the requesters. They want requesters to being available for communication and treat them fairly with respect.
  • Turkers want a system to in which they could reciprocate their feelings to the requesters in terms of ratings or some similar way.
  • They search intensively on forums like Turk Nation for information so that they could make informed choices about AMT.
  • New Turkers are likely to accept low paid easy work to increase their HIT count.
  • In addition to the actual work done by the workers on the HITS, an incredible amount of time is spend by them to find HITs, gain information, improve their skills and knowledge. This work remains unpaid and hidden.
  • They don’t like interference by journalists and academics as they believe regulations could ruin AMT for turkers.


2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • There is very less communication between requesters and workers.
  • They are very critical about the work of the turkers and may label them as bots or spammers if they don’t like their work.
  • When requesters fail to acknowledge the hard work, effort and the involvement put by the workers, it is generally the case the latter has to suffer.
  • Some requesters appreciate feedback from workers.

Worker perspective: Turkopticon

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Workers are treated as infrastructure.
  • No way for the workers to question the employers on rejection of a task.
  • Workers in lower cost countries gets more benefit than US citizens.
  • Interface of mturk do not suffice the needs of workers.
  • Prioritization of employers over workers.
  • A need for communication bridge between Employers and workers.
  • Calls and needs of workers should be listened/considered.
  • Workers cannot question the employers regarding their rejection on basis of skill set.
  • Workers had conflicting opinions about unions, forums to publicly air their views.
  • Rejection of worker can lead a requestet to put down his reputation in crowdsourcing market.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Design task in such wasy that it becomes easy for workers to do.
  • Requester use alogorithm to check how well the tasks have been completed.
  • Requesters have acces to APIs to keep track on workers.
  • A proper series of steps/intructions are defined by employers starting from sign up to desired deliverables.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

Brief overview: Two experiments were conducted to test the utility of Mechanical Turk as a user study platform.

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Workers response to jobs is extremely fast. But in some instances workers don’t complete their tasks fairly and complete it just for the sake of it. For example, writing a review.
  • They don’t mind engaging in the same task again and again.
  • If there is no scrutiny or check, workers may produce invalid answers in order to save their time.


2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Requesters very the HITs on the basis of time workers take to complete a task.
  • For tasks which involve user studies, requesters should include explicitly verifiable questions as part of the task.
  • Requesters who design task in such a manner that completing it accurately and sincerely requires less effort are less prone to malicious turkers as compared to others who don’t.

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Workers have no particular learning criteria or specific tasks that they continuously follow once in the labour market, unlike traditional markets. There is flexibility in taking up jobs.
  • Workers suffer from lack of pricing standardization. Similar jobs often have very different remunerations and in the confusion of a large open market and several jobs, workers with less experience lose out.
  • Workers have no guidance when they start out. There is no standard method to achieve better results. They have to hit and try a lot before they can apply their skills effectively.
  • In most platforms, there is little screening of a worker’s skill level before he is allowed to accept a task. The insufficient workers are filtered out by the requester through feedback rather than upfront. However, some platforms like uTest and MicroTask provide standardised simple tasks whose skill requirements can be checked beforehand to an extent.
  • For each job, workers have a different requester and need to adapt to different specifications and quality requirements for the deliverables as per the requester.
  • Standardization helps workers gain familiarity with a type of work and reuse their knowledge. The pricing of jobs becomes more reliable, they get correct value for work done.

2) What observations bout requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Requesters also have a lot of flexibility in the kind of jobs they offer. But as a downside, they need to create tasks for each job that would use the crowd labour market in the best way possible.
  • Requesters have to put up specific and detailed instructions for each task since the workers accepting the job may not be familiar with the particular type of job. This is unlike a traditional market where workers learn the tasks needed in their particular field and gain familiarity.
  • Number of spammers is growing with the lack of checks and a satisfactory reputation management system.
  • Even honest requesters do not have a standard pricing scheme to put up jobs for the market.
  • Requester is burdened with checking low-quality submissions due to lack of pre-task filtering of workers according to skill.
  • Standardization helps in reducing burden of creating new tasks for each job and setting prices.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • A need to reduce wage per HITS, traction cost etc.
  • A way to rate requesters so the basis of rejection,completion of tasks and designing of the tasks.
  • US citizen should be given sufficient wage as compared to other workers.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • A need to have good interface to post the tasks so that workers easily find their task.
  • Prioritize employers over workers.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

The following are the observations we made while doing your fieldwork:

  • Minimum wages for HITs.As Requestors from India where $1 equals 62 rupees should not pay high amount for every completed tasks.
  • Workers in US should not have to compete with workers who have 5x buying power for every dollar.This calls for minimum wage per hit.
  • Action from the host when HITs violate terms and conditions.
  • From Requester Side -Minimum number of workers required to engage for tasks should be flexible, that is, there should not be threshold for minimum number of of workers. As number of workers increase , the cost to post/upload the task increases.
  • Clearly defined length of task for every HIT.
  • Transparent reviewing process for completed tasks.
  • Review should take place quickly.
  • Skill set required for a worker should be flexible.
  • Mock test for workers should take place to decide whether workers are able to do the task or not.
  • UX of various mturk should be improved, that is, worker should easily find their desired jobs.
  • From Requester and Worker side - Amazon mturk should be open for other countries as well , not just for US Citizens.
  • Turkers should be able to satisfy their needs outside the mturk world.

Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

A set of bullet points summarizing the needs of workers.

  • Minimum wage per hits:

Evidence: Kevin steve from US claimed that he did not like to those who had 10x times buying power, that is where workers from lower costs countries. Interpretation: For a task which paid $10, do for US citizen it is $10 but on other hand it is equivalent to 1000 cents.So accoiding to him, low wage should be assigned to every tasks so that every worker gets equal advantage.

  • Flexible skill set:

Evidence- I was denied to do really simple task( Google search ) just because i did not fit into the skill set required. Interpretation: Workers skill set should not be decided by data he puts in while registering but instead his skill set should be decided on th basis of past records/completed tasks.

Solution: Conduct Mock Test for tasks for the workers to judge whether the particular worker is elligible to do the task or not.

  • User experience should be improved:

Evidence- It took me hour to search for desired task i wanted to do, just because there was option filter search.

Interpretation: workers want easy looking websites so that they dont have to struggle to register for task.

Solution: Emphasis of UI/UX design of mturk should be done.

Requester Needs

A set of bullet points summarizing the needs of requesters.

  • Minimum of number of required workers to engage for tasks:

Evidence: For survey task,Johnson needed only 18 workers but on microworker did not allow it as the threshold for minimum number of worker was 30.So when he entered 18 workers , total cost to upload was $2.08 but this got rejected by mturk but as he entered 30 workers, cost raised to $9.67 and this got accepted.

Interpretation: Minimum number of workers should be flexible.

  • Minimum wage for HITS:

Evidence- Harshdeep from India registered in microworker as requester.He uploaded the task and minimum wage given to worker per hit was $5 which is eqivalent to 300 rupees.So he wanted minimum wage per tasks given to worker should be less for lower cost countries.

Interpretation: Minimum wage should be flexible according to economy of the country.

Solution: Threshold of minimum wage should be reduced suffice all kinds of requesters.