Milestone 2 pentagram

From crowdresearch
Revision as of 10:30, 11 March 2015 by Karthiksenthil (Talk | contribs)

Jump to: navigation, search

This is the submission page for Milestone 2 by Team pentagram.

Attend a Panel to Hear from Workers and Requesters

Morning panel


Some observations/ideas noted down in the morning panel session :-

Worker perspectives


  • some sites to find list of HITs- easy way to find tasks undertaken by workers
  • the joy of giving back to community is a motivating point for turkers
  • HITExploiter - a script to automate and rate tasks using Turkopticon
  • Motivation for workers in MTurk
    • money
    • wanting to help people
    • social concepts
  • the worker community is very friendly and helpful ; more or less like a Facebook group discussing about anything under the sun
  • worker <-> requester interaction
    • shoot emails directly (instructions not clear)
    • invite requesters to forums
    • verify credibility of requesters
  • many a times pro workers help newbie requesters on how to use GUI,API and all
  • the biggest hurdles for newbie workers
    • finding a matching job - very hard
    • poor UI in Amazon Mturk
    • requires scripting knowledge to make decent money
    • in case of non-US countries ; more difficult to find work
    • frustration due to poorly paying jobs
    • some very important suggestions --> ask questions
  • the dropoff rate on mTurk --> severe, because of no feedback and rejections


Requester perspectives


  • Some common thoughts
    • prevailing wage 8$-10$ per hour - why ? (demographically and ethically minimum wage)
    • main problem -> was task taken seriously ?
    • how to detect cheating -> using open-ended questions
    • time vs. money
      • balancing act
      • give incentives/bonus
    • requesters dont have time to follow forums to know about their tasks
    • requesters prefer personal email
    • no assurity about completion of tasks
    • very complicated to get a task done by a specific worker
  • threshold for rejecting HITs adopted by many requesters
    • use the open ended questions - if responses are direct from Wikipedia or gibberish implies bad work
    • use timers
    • feel that workers didn't read instructions completely
  • problem in India and outside US
    • proxy accounts
    • Indians using USA-based MTurk account to turk and make money
    • account selling - very common pre-2012
  • Some mistakes by requesters-
    • give very less time (no tutorial from Amazon side)


Evening Panel


Some observations/ideas noted down in the evening panel session :-

Worker perspectives


Requester perspectives


Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Worker perspective: Turkopticon

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

List out the observations you made while doing your fieldwork. Links to examples (posts / threads) would be extremely helpful.

Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

A set of bullet points summarizing the needs of workers.

  • Example: Workers need to be respected by their employers. Evidence: Sanjay said in the worker panel that he wrote an angry email to a requester who mass-rejected his work. Interpretation: this wasn't actually about the money; it was about the disregard for Sanjay's work ethic.

Requester Needs

A set of bullet points summarizing the needs of requesters.

  • Example: requesters need to trust the results they get from workers. Evidence: In this thread on Reddit (linked), a requester is struggling to know which results to use and which ones to reject or re-post for more data. Interpretation: it's actually quite difficult for requesters to know whether 1) a worker tried hard but the question was unclear or very difficult or an edge case, or 2) a worker wasn't really putting in a best effort.