WinterMilestone 2 DavidThompson

From crowdresearch
Jump to: navigation, search

Attend a Panel to Hear from Workers and Requesters

Deliverable - Report on some of the observations you gathered during the panel.

Workers expressed concern over unpredictably - unpredictability around when high quality work would be available. Workers also expressed concern around the variance in their earnings. Comments were made throughout the panel to this effect.

I observed that Workers used a variety of tools: to learn more about requesters, to learn more about 'good' opportunities, and to communicate with other Turkers. Workers shared their use of forums, social media platforms (a snapchat group was mentioned), browser extensions, and scripts.

Workers described the importance of their worker rating, something that was used to get and keep access to more lucrative HITs. This informed their perspective on the negotiation of rejection very differently than requesters.

Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

All quotes are directly from this paper.

  • That while there is a large population of Turkers, most of the work is done by a small population ("... 80% of the tasks are carried out by the 20% most active"). That population are using sites like Turker Nation to better enable their ability to earn money as they "orientate to AMT as a labour marker".
  • Workers on Turker Nation are not doing this work for fun. For many the work represents a safety net
  • Workers exhibit an emotional connection to the work "...even though the major part of the relationship is constituted simply in the process of choosing and submitting HITs, receiving approval (or not), and getting paid (or not)."
  • Worker expectations for online work mirror those of offline work - clarity, fairness, respect, and value. "As with traditional labour markets, the relationship between worker and 'work provider' is key."
  • Workers exhibit everyday moral standards in the AMT workplace. Turkers look to make sense of rejection not for learning sake (with respect to the domain), but with respect to the process.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Within the existing system design, requesters are differentially enabled as compared to Workers. This is quite clear from the imbalance of power and the information asymmetry.
  • Fair requestor behaviour is expected.

Worker perspective: Turkopticon

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

All quotes are directly from this paper.

  • Workers are participating in a system for which power is asymmetrically distributed. For instance, Amazon Mechanical Turk's (AMT's) participation agreement grants requesters full intellectual property rights over submissions, regardless of rejection.
  • The very system of AMT inculcates a marketplace for invisible or 'hidden labour'
  • Worker resources for dispute resolution do not scale
  • The feedback mechanism that workers experience is 'too tightly wound' (my own interpretation - reading this section of the paper made me think of a mousetrap - the slightest 'nudge' could trigger sharp and damaging feedback). Specifically: "When an employer rejects an employer's work, whether because it did not meet their needs or simply so they employer did not have to pay, the worker's approval rating goes down."
  • Through use of Turkopticon workers experience access to more information about which to make decisions.
  • Because this mode of work is emergent there are no established models and systems. It is important that a system that respects this is itself open to 'ongoing maintenance and repair'.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • The system is constructed in a way that requesters do not fully appreciate, or perhaps care enough, about the working reality of Turkers and, as such, are not suitably incentivized to want to work differently. The AMT model is such high volume that "You cannot spend time exchanging email" regarding a dispute.
  • Requestor incentives have been separated quite completely from those of Workers through the AMT system design. Power is firmly in Requestor control: "Amazon seems to pay more credence to the requesters, simply ignoring the fact that without workers, nothing would be done!"

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

All quotes are directly from this paper.

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • When tasks were poorly designed (or open to being 'gamed'), a small number of workers generated many invalid responses.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Requestors had to take the time to design the task such that completing it appropriately or in good faith requires as much effort or less, as doing so maliciously.
  • Requestors had to own the careful design of the tasks to avoid the small minority who might look to game the system.

Requester perspective: The Need for Standardization in Crowdsourcing

All quotes are directly from this paper.

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Workers currently using AMT would benefit from the three modes standardization described. The standardization of tasks would remove some of the friction from the existing marketplace, thus making it more flexible - and the associated pay more equitable.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • It is suggested that requestors would benefit from the standardization by being able to more fairly peg prices of tasks to historical values. It is implied that Requestors do this on an ad hoc basis currently (determining what the market will bear/tolerate).

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

All quotes are directly from this paper.

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • That there is a large enough pool of workers available to endure the existing AMT design, or with access to work arounds, that continued participation is less costly than not.
  • That Workers fully understand and appreciate the downsides of the current system.
  • Workers desire a better user interface that Amazon has, historically, been uninterested in providing.
  • Workers are adept at finding hacks and systems to apply as bridges to this marketplace.
  • Workers would benefit from a reputation system that was somewhat divorced from payment and also future opportunities.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Requesters desire a better user interface
  • Requesters need a reputation system that does not unfairly penalize 'good faith' attempts.
  • Requesters would benefit from input and participative involvement of workers in the design, building, and execution of hits.

Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

A set of bullet points summarizing the needs of workers.

  • Workers need tools and/or system features to help them manage the unpredictability of when good quality, high paying, work becomes available. Evidence: This theme manifested itself during the panel session from all of the Turkers who described how this was a source of income for them. For a minority of Turkers, AMT represents a financial safety net. Currently, those Turkers are using external tools to aggregate information to help them make choices about how to spend their time, and on what work (see analysis above from 'Being a Turker', and 'Turkopticon').
  • Workers need to experience a system that is more similar to the traditional labour model, while preserving the flexibility of contributing to microtasks. Within large firms there is a tacit expectation that as an employee your work will be presented with clarity, and you will be treated with fairness, respect, and value for your contribution. This is clearly of no less importance for the community of Turkers who view their contribution as a job. Evidence: A number of comments in 'Being a Turker' describe the emotional connection of Turkers to the work, and that workers exhibit everyday moral standards in the AMT workplace; the majority of people looking to use this as a financial 'safety net' are treating this as a job, and they take pride in their contribution. This manifests itself in the negotiation of rejection, specifically, from 'Being a Turker': "Turkers look to make sense of rejection not for learning sake (with respect to the domain), but with respect to the process."
  • Workers need to be treated as equal participants in these marketplaces. Evidence: At present, through system design, Worker contributions are invisible, with power wresting solely in the hands of the requesters (this was heard during the panel discussion, and throughout the papers read, specifically 'Turkopticon'). This fundamental dichotomy is the biggest source of inefficiency, manifesting as mistrust, in the system and as such, both workers and requesters, suffer.

Requester Needs

A set of bullet points summarizing the needs of requesters.

  • Requesters need to be treated as equal participants in these marketplaces. Evidence: At present, through system design (terms & conditions, IP rights, issue resolution etc.), power wrests solely in the hands of the requesters (this was heard during the panel discussion, and throughout the papers read, specifically 'Turkopticon'). This fundamental dichotomy is the biggest source of inefficiency, manifesting as mistrust, in the system and as such, both workers and requesters, suffer.
  • Requesters need a feedback process that scales, and that is more forgiving of good faith work attempts. Evidence: The cost of largescale feedback is prohibitive, and as such isn't given as needed (see comments above re. Turkopticon). This deepens mistrust with the workers, many of whom make good faith attempts to do 'their job' as they felt they should.

Milestone Contributors

Slack usernames of all who helped create this wiki page submission: @dcthompson