Difference between revisions of "Milestone 2 AltaMira"

From crowdresearch
Jump to: navigation, search
(Attend a Panel to Hear from Workers and Requesters)
(Attend a Panel to Hear from Workers and Requesters)
Line 34: Line 34:
  
 
- Finding the right person for a task is not congruent with how MTurk's system. The system does not have a good way to deliver the right workers.
 
- Finding the right person for a task is not congruent with how MTurk's system. The system does not have a good way to deliver the right workers.
'''
+
 
Both:'''
+
'''Both:'''
  
 
- Regardless of the system, oDesk, MTurk all have very high user ratings, oDesk which counts rejection rates less severely still has an avg user rating of 4.5/5
 
- Regardless of the system, oDesk, MTurk all have very high user ratings, oDesk which counts rejection rates less severely still has an avg user rating of 4.5/5

Revision as of 19:28, 11 March 2015

Alta Mira Milestone 2 Submission:

Attend a Panel to Hear from Workers and Requesters

We attended Panel 2 at 6pm PST. here are some of our observations:

Workers:

- A lot of the work (apparently up to 50%) comes from academic sources.

- The group of workers we were talking to were involved in academic studies and chose tasks in the higher bracket (not the lower $0.01 HITs)

- Workers work varies, many HITs come between 7am and 4pm, some workers prefer to work during their off time, some work during regular hours.

- Workers do not treat MTurk or freelancing only as a source of income. In a 1:1 converstaion, I was told you need to have strong financial skills to make freelancing a full time job. If you have a mortgage, this is probably not the best route.

- Workers hours are highy variable, sometimes they make a lot of money, sometimes they don't. They have to balance it out over weeks.

- Skills are not well measured on MTurk and they donot like the qualification system.

- Rejection percentages (whether they are employed by requestors or not) are a cause of concern/grief for workers.

Requesters:

- Requesters generally don't reject work due to a fear of backlash on forums, the high level of impact each rejection has on a worker. They are more likely to accept even bad work to avoid what happens if they reject.

- In the words of 2 long term requesters, rejection system in MTurk especially is broken, it is not a system that requesters use and it does not quantify what they expect from workers.

- Approval rate is one of the most commonly used qualifications for a HIT.

- Like the idea of micro tasks and using it to create a gold standard.

- Don't prefer Master's qualifications becuase it costs more, is hard to get and doesn't always offer better results.

- Finding the right person for a task is not congruent with how MTurk's system. The system does not have a good way to deliver the right workers.

Both:

- Regardless of the system, oDesk, MTurk all have very high user ratings, oDesk which counts rejection rates less severely still has an avg user rating of 4.5/5

Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Worker perspective: Turkopticon

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

List out the observations you made while doing your fieldwork. Links to examples (posts / threads) would be extremely helpful.

Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

A set of bullet points summarizing the needs of workers.

  • Example: Workers need to be respected by their employers. Evidence: Sanjay said in the worker panel that he wrote an angry email to a requester who mass-rejected his work. Interpretation: this wasn't actually about the money; it was about the disregard for Sanjay's work ethic.

Requester Needs

A set of bullet points summarizing the needs of requesters.

  • Example: requesters need to trust the results they get from workers. Evidence: In this thread on Reddit (linked), a requester is struggling to know which results to use and which ones to reject or re-post for more data. Interpretation: it's actually quite difficult for requesters to know whether 1) a worker tried hard but the question was unclear or very difficult or an edge case, or 2) a worker wasn't really putting in a best effort.