WinterMilestone 2 nalinc

From crowdresearch
Revision as of 11:20, 24 January 2016 by Nalinchhibber (Talk | contribs)

Jump to: navigation, search

This page is intended to serve for Winter Milestone 2 submission by user @nalinc[1].

Do not edit this directly

Attend a Panel to Hear from Workers and Requesters

Report on some of the observations you gathered during the panel.

Chris: Shawn: Peter:

Laura: Rochelle: Chrissie:


Reading Others' Insights

Worker perspective: Being a Turker

This paper Looks behind the walls of abstraction that enable human computation in one specific system, Amazon Mechanical Turk (AMT). AMT can be described in many ways. Describing it as a:

  • microlabor marketplace draws attention to pricing mechanisms, how workers choose tasks, and how transactions are managed.
  • crowdsourcing platform draws attention to the dynamics of mass collaboration among workers, the aggregation of inputs, and the evaluation of the crowdsourced outputs.
  • source of human computation resources, however, is consistent with how both the computing research community and Amazon’s own marketing frames the system.

What observations about Workers can you draw from the readings?

1) Motivation for turking

  • Turking is a "safty net" for majority of worker. They consider it as a decent option while they dont have any other means to make the ends meet.
  • Regulars or set hours are not required, money or set hours are not required, money does not have to be spent on transportation costs,and judgements are restricted to work you submit rather than the way you appear or present yourself.

2) Money turkers make from turking

  • Median annual income is less than $10000, with highest earing as $15k per year.
  • $8 considered a good pay while minimum wage in USA is $(7.25). Hourly wage can be as low as $1
  • Novices can take up low paying jobs because they need to increase their HIT count and Approval rating. This kind of invisible work is necessary for the goal of better paying work.
  • Experienced Turkers dont take jobs which go below ~$7/hour

3) Relation with requesters

  • Turkers interaction with requesters is limited to the process of choosing and submitting HITs, receiving approval (or not), and getting paid (or not). Turkers extrapolate opinions on Requesters based on sparse information.
  • Turker Nation is a big forum where Turkers can communicate within themselves and with requesters.
  • When a requester communicates with turkers while processing HIT, i.e. answering their questions, the reputation of the Requester increases. The promptness of requester is a huge concern for turkers. They seem to appriciate a requester who answers to their questions and pays on time.
  • A requester is judged on the basis of pay, rejections and responsiveness. Lack of information seems to increase the adversarial tension between Turkers and their requesters
  • Turkers are willing to discuss how to improve HIT design with requesters.

3) Workplace Ethics

  • There has been a lot of activity where Turkers seek to make sense of why things have gone wrong and self-critical is part of this. They do look for causes that are technical or with HIT design, and turkers are tolerent of genuine mistakes, perticularly when requester seeks to sort them out.
  • Some of the turkers suggest that they should not unfairly judge the requesters as it will put off other Turkers from working for decent requesters.

4) Turkers view of market

  • Turkers believe that involvment of legislation raises cost for employers which threatens market vaibility and/or workers take a hit in their wage levels.
  • Turkers orient to "positive invisibility" - freedom from surveillence, control and intervention in their personal affairs.

5) Work to make Turking work

  • Find HITs make good choices. Learn about requesters. Gathering information about turking an HITs.
  • Learn and improve skills and knowledge, and manage their AMT work.
  • This work is hidden and unpaid, but necessary to get the best return on turking.

Worker perspective: Turkopticon

This paper by Lilly et al presents the role Turkopticon plays in crowdsourcing market and how it allows workers to rate requesters and see their ratings inline on AMT. The paper tries to address some serious concerns regarding a typical worker on AMT. Some of these concerns include worker invisibility, below minimum wage payments, lack of concerned with the human costs of human computation, little opportunities for workers to build solidarity, offering them little chance of creating sufficiently coordinated actions to exert pressure on requesters.

This papers makes several contributions in the following ways:

  • offers a case study designing an intervention into a highly distributed microlabor system
  • shows an example of systems design incorporating tools feminist analysis and reflexivity
  • presents lessons learned from intervening in existing, large-scale sociotechnical systems.
  • focused to bring worker-employer relations visible and provoke ethical and political debate.

What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Workers on AMT are framed as computational services. Crowds of workers are brought together as a form of infrastructure, thereby rendering employees into reliable sources of computation.
  • Amazon legally defines these workers as 'contractors' subject to laws designed for freelancers and consultants; this attempts to strip workers of minimum wage requirements in their countries.
  • Since AMT’s participation agreement grants requesters full intellectual property rights over submissions regardless of rejection, workers have no legal recourse against employers who reject work and then go on to use it.
  • workers are the digital force, distributed across the world and organized through task markets, APIs, and network connections
  • Workers dissatisfied with a requester’s work rejection can contact the requester through AMT’s web interface, but Amazon does not require requesters to respond and many do not.
  • Not all workers engage in crowdsourcing for an income to pay for household expenses. Few workers report working for fun or to pass the time while bored (sometimes even at another job)

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

What observations about Requesters can you draw from the readings?

  • AMT platform is designed to provide little information on Requesters. This gives rise to suspicion, as well as the highly motivated search for better information, e.g. through forums, blogs etc.
  • Requesters have better information on the Turkers than vice versa, as well as greater powers of redress.
  • “Blocking” is when Requesters officially bar a Turker from working for them, which is a legitimate tool to have at their disposal if used fairly. However, if Amazon becomes aware of any Turkers getting too many blocks they may well suspend their account.
  • A good requester is one who communicates well and thinks about what is best for turkers as much as himself.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

List out the observations you made while doing your fieldwork. Links to examples (posts / threads) would be extremely helpful.

  • Dissatisfied workers’ within AMT had little option other than to leave the system altogether. Because AMT treats workers interchangeably and because workers are so numerous (tens of thousands by the most conservative estimates), AMT can sustain the loss of workers who do not accept the system’s terms.
  • Consequently, workers need to reply of varous unofficial scripts to make their life easier. Following are some of the commonly used scripts/tools:
- Turkopticon - allows workers to rate requesters and see their ratings inline on AMT.
- HITDB - keeps track of HITs workers have done.
- MTurk Status Page Chart turns your status page into a graph and colour-coded table.
- Crowd Workers keeps track of all HITs you've done and lets you search the results.
- TurkMaster allows you to get alerts through your Dashboard page when your favourite Requesters post HITs.
- HIT Scraper searches mTurk for specific keywords so you know when HITs that match them are posted.

Research Engineering (Test Flight)

Reported 1 issue(#641, now closed) and made two pull-requests #643 and #642 (one of which is merged).

Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

A set of bullet points summarizing the needs of workers.

  • Workers want their efforts and time be rightly paid.
  • Need a reliable mechanism to trust the requesters they work with. Often this leads them use some external scripts/tools or plugins to ensure requesters ratings.
  • Workers also try to work with those requesters who they have experience working with in past.

Requester Needs

A set of bullet points summarizing the needs of requesters.

  • Requesters need a robust mechanism to trust their workers and the quality of work they submit. Most of the present platforms try to address this issue by reputation scores which are highly inflated anyways.
  • An easy-to-use interface to share their mental-model with workers to ensure better work quality.
  • Need to establish a trustworthy relationship with 'skilled' workers and they actually want the same workers to work on their HITs who have better track-record of quality work in past.