Milestone 2 pixelpals

From crowdresearch
Revision as of 00:09, 12 March 2015 by Harkiratsinghlamba (Talk | contribs) (Attend a Panel to Hear from Workers and Requesters)

Jump to: navigation, search

Template for your submission for Milestone 2. Do not edit this directly - instead, make a new page at Milestone 2 YourTeamName or whatever your team name is, and copy this template over. You can view the source of this page by clicking the Edit button at the top-right of this page, or by clicking here.

Attend a Panel to Hear from Workers and Requesters

Workers Side

  • Workers extensively use various forums and to discuss any and all aspects related to issues they face. They communicate about their experience and help each other resolve everyday issues.
  • Everybody has disappointing days, even the most experienced workers.
  • User – testing videos are easier and less competitive.
  • Sending Mail to requesters upon rejection is now become a tradition.
  • There exists a very strong-knit co-worker community in which they collaborate and communicate with them. Eg Facebook group for Turk-Workers is a very popular.
  • Motivation goes beyond money in various cases and that becomes the sticky factor that allows users to stay connected with the crowd sourcing platforms even in disappointing times.

Requester Side

  • Pricing a HIT is generally based on the number of hours they expect the workers to take to do the task. Since 8-10 dollars per hour is the prevailing price the pricing the task by the requesters itself can give a big hint to the workers about the number of hours you expect.
  • Putting in a open ended question to differentiate b/w genuine workers and spammers is a more effective and commonly used way as compared to putting in “Gold Standard Questions”
  • Putting in a script to find out time spend by workers on individual pages in task is an interesting way to find outliers to your task.
  • M-Turk restriction to Not allow workers to “Download” tools to complete the task is a very big disadvantage to the requesters
  • Bonus/incentives that are not relived before hand to workers can be a really quick and interesting way to remove workers who do just for money.

Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit. Workers turk on AMT primarily for money. They use forums like Turk Nation to share their problems with people looking for advice and support. They prefer platforms like AMT because there are no set hours, no transportation money is required and they get the flexibility of working from home. Workers judge requesters not only on the basis of the money they pay but also on the responsiveness of their evaluation and the rating they give to the turkers. They don’t want to be unfairly treated by the requesters. They want requesters to being available for communication and treat them fairly with respect. Turkers want a system to in which they could reciprocate their feelings to the requesters in terms of ratings or some similar way.They search intensively on forums like Turk Nation for information so that they could make informed choices about AMT. New Turkers are likely to accept low paid easy work to increase their HIT count. In addition to the actual work done by the workers on the HITS, an incredible amount of time is spend by them to find HITs, gain information, improve their skills and knowledge. This work remains unpaid and hidden.They don’t like interference by journalists and academics as they believe regulations could ruin AMT for turkers.


2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit. There is very less communication between requesters and workers. They are very critical about the work of the turkers and may label them as bots or spammers if they don’t like their work. When requesters fail to acknowledge the hard work, effort and the involvement put by the workers, it is generally the case the latter has to suffer. Some requesters appreciate feedback from workers.

Worker perspective: Turkopticon

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

Brief overview: Two experiments were conducted to test the utility of Mechanical Turk as a user study platform.

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

Workers response to jobs is extremely fast. But in some instances workers don’t complete their tasks fairly and complete it just for the sake of it. For example, writing a review. They don’t mind engaging in the same task again and again. If there is no scrutiny or check, workers may produce invalid answers in order to save their time.


2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requesters very the HITs on the basis of time workers take to complete a task. For tasks which involve user studies, requesters should include explicitly verifiable questions as part of the task. Requesters who design task in such a manner that completing it accurately and sincerely requires less effort are less prone to malicious turkers as compared to others who don’t.

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

We exlored mturk Grind.The following are the observations we made while doing your fieldwork:

  • Minimum wages for HITs.As Requestors from India where $1 equals 62 rupees should not pay high amount for every completed tasks.
  • Workers in US should not have to compete with workers who have 5x buying power for every dollar.This calls for minimum wage per hit.
  • Action from the host when HITs violate terms and conditions.
  • From Requester Side -Minimum number of workers required to engage for tasks should be flexible, that is, there should not be threshold for minimum number of of workers. As number of workers increase , the cost to post/upload the task increases.
  • Clearly defined length of task for every HIT.
  • Transparent reviewing process for completed tasks.
  • Review should take place quickly.
  • Skill set required for a worker should be flexible.
  • Mock test for workers should take place to decide whether workers are able to do the task or not.
  • UX of various mturk should be improved, that is, worker should easily find their desired jobs.
  • From Requester and Worker side - Amazon mturk should be open for other countries as well , not just for US Citizens.

Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

A set of bullet points summarizing the needs of workers.

  • Example: Workers need to be respected by their employers. Evidence: Sanjay said in the worker panel that he wrote an angry email to a requester who mass-rejected his work. Interpretation: this wasn't actually about the money; it was about the disregard for Sanjay's work ethic.

Requester Needs

A set of bullet points summarizing the needs of requesters.

  • Example: requesters need to trust the results they get from workers. Evidence: In this thread on Reddit (linked), a requester is struggling to know which results to use and which ones to reject or re-post for more data. Interpretation: it's actually quite difficult for requesters to know whether 1) a worker tried hard but the question was unclear or very difficult or an edge case, or 2) a worker wasn't really putting in a best effort.