WinterMilestone 2 biubiubiu

From crowdresearch
Revision as of 19:07, 23 January 2016 by Xichen (Talk | contribs) (Both perspectives: A Plea to Amazon: Fix Mechanical Turk)

Jump to: navigation, search

Attend a Panel to Hear from Workers and Requesters

Deliverable

Report on some of the observations you gathered during the panel.

Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Worker perspective: Turkopticon

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

- Experienced and good workers treat new requesters cautiously. They simply don't devote much time and effort for a new requester until that requester's information like payment speed, rejections, possibility of coming back to the marketplace to post tasks in the future.

    - afraid of getting rejected for no reason or waiting too long for payment, good workers just do a little bit of unknown requesters' tasks.
    - if the requester just have one-pass tasks and they are relatively difficult, good workers will not spend time to learn how to do well.

- Workers tend to seek help from some well-known forums.

- If workers get rejected,

   - they want to keep their work to themselves;
   - they need a way to appeal and can win if their work is properly done.

- Certain worker-facing applications can improve efficiency.

- Workers want to find wanted tasks AE(asy)AP:

   -they want to find certain requesters' HITs by searching names of requesters;
   -they want to complete HITs that they're interested in and good at.
   - now on MTurk, workers use priority queues to choose HITs.
   - separate HITs by type. also mentioned in requesters part

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

- New-coming requesters' tasks will be done mostly by careless workers, also known as "spammers", which discourage requesters' interest in the marketplace.

- It is impossible to predict the completion time of the posted tasks due to user-unfriendly interfaces on MTurk.

- Requesters want to post tasks easily:

   - Small requesters want to cut overhead and other costs when using crowdsourcing marketplaces. But now they, and every other requester, need to spend extra effort and time building accessible systems on their own.

- Requesters need more details besides simpy "Approval Rate" and "Number of Completed HITs"

   - they want to be ensured that they can get results of good quality.
   - they need to know if a worker is qualified for their tasks.
   - working history of a worker can reveal something important.
   - rate and pay a worker according to submitted work.

- Conscientious requesters say,honest workers' reputation should not be affected if their work gets rejected becaues quality cannot meet requesters' standard.

- Task and Rating Categorization.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

List out the observations you made while doing your fieldwork. Links to examples (posts / threads) would be extremely helpful.

Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

A set of bullet points summarizing the needs of workers.

  • Example: Workers need to be respected by their employers. Evidence: Sanjay said in the worker panel that he wrote an angry email to a requester who mass-rejected his work. Interpretation: this wasn't actually about the money; it was about the disregard for Sanjay's work ethic.

Requester Needs

A set of bullet points summarizing the needs of requesters.

  • Example: requesters need to trust the results they get from workers. Evidence: In this thread on Reddit (linked), a requester is struggling to know which results to use and which ones to reject or re-post for more data. Interpretation: it's actually quite difficult for requesters to know whether 1) a worker tried hard but the question was unclear or very difficult or an edge case, or 2) a worker wasn't really putting in a best effort.