WinterMilestone 2 Jin

From crowdresearch
Revision as of 19:38, 24 January 2016 by Faithjin (Talk | contribs) (Milestone2)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Attend a Panel to Hear from Workers and Requesters

Deliverable

Report on some of the observations you gathered during the panel.

On the topic of daily life:

  • As a worker: some workers have a more rigid schedule and work during certain hours of the day whereas some workers sort of play it by ear and take it as it comes. But all workers (or at least the ones in the panel) use some type of outside information source (like Turk Nation) to help them.
  • As a requester: what might make sense to you might not make sense to a worker. Sometimes requesters need to deploy a smaller batch just to ensure the results they get are worthwhile and understandable.

On the topic of ratings:

  • As a worker: workers care deeply about ratings. If ratings go below a certain threshold it prevents them from getting certain hits.
  • As a requester: it’s important to manage reputation as a requester as well. Requesters sometimes utilize forums (such as Turk Nation) as well to keep an open dialogue with workers. Some requesters automatically accept all work done which leads to problems such as people doing terrible work.

On the topic of what you can’t do:

  • As a requester: MTurk has prebuilt templates that you can utilize but it doesn’t cover all use cases. You cannot confirm if people are sincere and answering question carefully in the case of surveys.

On the topic of tracking wages:

  • As a worker: there is no easy way to track wages. It may take a lot of experience with MTurk to predict which hits are worthwhile and how long things take. Time estimates indicated on the hit can be very off. There is a dramatic fluctuation in wages. Many workers create a system for themselves (using tools such as excel) to track wages.

On the topic of rejection:

  • As a worker: even if you try to do the work carefully, sometimes you are rejected. If you get rejected from too many hits it drops your reputation.
  • As a requester: some opt to automatically accept all hits but some have varying degrees of thresholds in terms of rejecting.

Reading Others' Insights

Worker perspective: Being a Turker

What observations about workers can you draw from the readings?

  • while there are many workers, most of the HITs are done by a relatively small portion of those people
  • a large portion of Turkers do it for the money but the amount of money they make falls below minimum wage, the importance of how much money they make is a direct reflection of their life circumstances (is it a primary or supplementary source of income)
  • workers have reputation ratings while requesters do not
  • workers have different ideas on what makes something worthwhile

What observations about requesters can you draw from the readings?

  • requesters engage directly with workers on forums
  • being a good requester (in the perspective of a worker) are fair pay and approval of hits
  • being a bad requester (in the perspective of a worker) are mass rejections
  • hasty judgement of a requester can really damage a worker’s access to better work

Worker perspective: Turkopticon

What observations about workers can you draw from the readings?

  • workers don’t alway use MTurk to make money to pay for life expenses (like rent)
  • workers are defined as freelancers or consultants which makes minimum wage laws irrelevant
  • workers have created a system that reveals their relationship with requesters and their thoughts concerning them (rating system)

What observations about requesters can you draw from the readings?

  • requesters often don’t see workers but rather web forms and APIs
  • requesters can reject work and still use it due to intellectual property rights
  • requesters are not obligated/required to respond to workers

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

What observations about requesters can you draw from the readings?

  • requesters can use MTurk as a platform to gain usability feedback from a large pool of people
  • when there is no definite answer it’s difficult to determine which users are not giving genuine answers
  • although requesters are able to reject (not pay for) responses, it still takes resources
  • in order to harness the power of crowdsourcing, its important to thoughtfully design tasks

Requester perspective: The Need for Standardization in Crowdsourcing

What observations about workers can you draw from the readings?

  • workers need to adapt to interfaces of each individual employer
  • workers do not know the quality standards to each employer
  • by having standardizations, you don’t need to think about reputation of requesters

What observations about requesters can you draw from the readings?

  • with standardization, requesters can ensure work will be done to a particular standard

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

What observations about workers can you draw from the readings?

  • the one sided rating system on Amazon.com ensures trustworthiness from one side
  • difficult to search for specific tasks

What observations about requesters can you draw from the readings?

  • there are many complexities that can hinder a requester from fully utilizing MTurk
  • building your own interface can be difficult
  • there is no clear way to differentiate good and bad workers

Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

A set of bullet points summarizing the needs of workers.

Needs for workers:

  • Workers need to feel confident that they will be fairly paid. Evidence: On the worker/requester panel several people mentioned the need to utilize external sources such as TurkNation. When posting about “good requesters”, getting paid quickly is one of the criteria many look at. Interpretation: although most feel money is an important factor, how heavily it’s weighed is a direct reflection of the workers life circumstances. Because there is such a wide cast of reason people use MTurk, some people think they are being paid fairly while some would disagree. Finding common ground in which most parties are satisfied will be difficult.
  • Workers need a method of communicating to requesters. Evidence: In the worker/requester panel, Rochelle mentions emailing the requester and the broad array of responses. Requesters are not obligated to reply and often have negative reactions towards receiving emails. Chris also mentioned having to hire someone to be able to go through all the emails. Interpretation: the need for communication often occurs due to poorly written tasks, lack of prompt paying, rejecting HITs. If HITs were better designed and better written, the need to have to communicate may possibly decrease.

Requester Needs

A set of bullet points summarizing the needs of requesters.

Needs for requesters:

  • Requesters need to be able to create customized HITs interfaces effectively. Evidence: Peter states in the worker/requester panel that he doesn’t use any of the built in templates because they do not fit his needs. Interpretation: when workers need to adapt to different cumbersome interfaces it drops their effectiveness and quality. Tools such as Soylent create a flexible easy to use tool for word processing related tasks. Users only need to get used to one interface to do many tasks.
  • Requesters need to be able to trust that the results they get are genuine (even if it’s a test where there is no right or wrong answer). Evidence: In the article Crowdsourcing User Studies with Mechanical Turk it states that when there is no right or wrong, users then to try and game the system. One way you can tell is by the speed in which they go through the task. Interpretation: Workers need to know that they are being held to a particular standard but need to be paid fairly for the extra time it will take.