Difference between revisions of "Winter Milestone 2 AtinMittra"

From crowdresearch
Jump to: navigation, search
(Deliverable)
(Requesters)
Line 4: Line 4:
  
  
== '''Requesters''' ==
+
Requesters
  
  
'''Chris:'''
+
'''Chris'''
 +
 
 
- Is able to conduct test hypothesis on MTurk within one day
 
- Is able to conduct test hypothesis on MTurk within one day
 +
 
- Acknowledges the importance of well designed HIT request
 
- Acknowledges the importance of well designed HIT request
 +
 
- Will release "small batch" of request instructions to gain feedback on their wording
 
- Will release "small batch" of request instructions to gain feedback on their wording
 +
 
- Can be difficult to work with non-native English speakers as their e-mails and feedback may be difficult to understand
 
- Can be difficult to work with non-native English speakers as their e-mails and feedback may be difficult to understand
 +
 
- Talks about inhumane symbolism of identifying workers with serial numbers instead of names
 
- Talks about inhumane symbolism of identifying workers with serial numbers instead of names
 +
 
- Will run threshold - if worker gets multiple choice answer right only 25% of the time, he knows they are randomly guessing. "It gets tricky when they're doing better than chance but worse than doing tasks conscientiously."
 
- Will run threshold - if worker gets multiple choice answer right only 25% of the time, he knows they are randomly guessing. "It gets tricky when they're doing better than chance but worse than doing tasks conscientiously."
  
Peter:
+
 
 +
'''Peter'''
 +
 
 
- Puts a lot of effort into designing HIT instruction because he can't pay a lot of money
 
- Puts a lot of effort into designing HIT instruction because he can't pay a lot of money
- Instead of rejecting work of rouge workers, he instituted qualifications. He mentioned rejecting work will effect long-term engagement of even productive workers.
 
  
Xiao:
+
- Instead of rejecting work of rouge workers, he instituted qualifications. He mentioned rejecting work will effect long-term engagement of even
 +
productive workers.
 +
 
 +
 
 +
'''Xiao'''
 +
 
 
- Something to grapple with is attention span of workers
 
- Something to grapple with is attention span of workers
 +
 
- Goes to TurkerNation to find good workers
 
- Goes to TurkerNation to find good workers
  

Revision as of 16:21, 24 January 2016

Attend a Panel to Hear from Workers and Requesters

Deliverable

Requesters


Chris

- Is able to conduct test hypothesis on MTurk within one day

- Acknowledges the importance of well designed HIT request

- Will release "small batch" of request instructions to gain feedback on their wording

- Can be difficult to work with non-native English speakers as their e-mails and feedback may be difficult to understand

- Talks about inhumane symbolism of identifying workers with serial numbers instead of names

- Will run threshold - if worker gets multiple choice answer right only 25% of the time, he knows they are randomly guessing. "It gets tricky when they're doing better than chance but worse than doing tasks conscientiously."


Peter

- Puts a lot of effort into designing HIT instruction because he can't pay a lot of money

- Instead of rejecting work of rouge workers, he instituted qualifications. He mentioned rejecting work will effect long-term engagement of even productive workers.


Xiao

- Something to grapple with is attention span of workers

- Goes to TurkerNation to find good workers

Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Worker perspective: Turkopticon

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.


Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

A set of bullet points summarizing the needs of workers.

  • Example: Workers need to be respected by their employers. Evidence: Sanjay said in the worker panel that he wrote an angry email to a requester who mass-rejected his work. Interpretation: this wasn't actually about the money; it was about the disregard for Sanjay's work ethic.

Requester Needs

A set of bullet points summarizing the needs of requesters.

  • Example: requesters need to trust the results they get from workers. Evidence: In this thread on Reddit (linked), a requester is struggling to know which results to use and which ones to reject or re-post for more data. Interpretation: it's actually quite difficult for requesters to know whether 1) a worker tried hard but the question was unclear or very difficult or an edge case, or 2) a worker wasn't really putting in a best effort.