Winter Milestone 2 AtinMittra

From crowdresearch
Jump to: navigation, search

The following document contains the observations from ethnographic research and readings on the needs of users on Mturk. For questions or comments Slack @atinmittra.

Attend a Panel to Hear from Workers and Requesters

Requesters

Chris (Assistant Professor at Upenn. Conducts research on Natural Processing Language. Posts many Requests on MTurk for Academic purposes)

- Is able to conduct test hypothesis on MTurk within one day

- Acknowledges the importance of well designed HIT request

- Will release "small batch" of request instructions to gain feedback on their wording

- Can be difficult to work with non-native English speakers as their e-mails and feedback may be difficult to understand

- Talks about inhumane symbolism of identifying workers with serial numbers instead of names

- Will run threshold - if worker gets multiple choice answer right only 25% of the time, he knows they are randomly guessing. "It gets tricky when they're doing better than chance but worse than doing tasks conscientiously."


Peter (Requester primarily posting book labeling HITS)

- Puts a lot of effort into designing HIT instruction because he can't pay a lot of money

- Instead of rejecting work of rouge workers, he instituted qualifications. He mentioned rejecting work will effect long-term engagement of even productive workers.


Xiao (Assistant Professor at University of Arkansas School of Business. Primarily uses MTurk for survey research)

- Something to grapple with is attention span of workers

- Goes to TurkerNation to find good workers


Workers

Laura (Started using MTurk as income when son was born. Disability prevents her to pursue alternative career opportunities)

- Takes care of kids at home while working on MTurk

- Uses combination of script, TurkerNation and thread to find suitable tasks

- Most important thing is approval rating. Never risks going below 99%. Will try a new requester.

- Constantly aware of that threat of rejection - "for a new worker, [rejection] is death."

- Takes screenshot of confirmation of work completion email. If issue arises with Requester, uses screenshots as evidence to prove completion. She acknowledges human error may be cause of quarrel, so she remains polite.


Rochelle (Worker since 2008. Advocate for new workers)

- Doesn't schedule breaks during work

- "You're kind of always on the edge of your seat... Because there are workers around the world, this is a 24 hour cycle...7 days a week."

- Needs high approval ratings. Is more hesitant with new requesters. Will send email to make sure there is a real responsive human on the other side.

- Tries 1 or 2 tasks to see if batch is worth doing - cross references her performance with estimated time on HIT. Checks forums to further cross reference Requester and task batch.


@SpamGirl (Crowdworker and Worker advocate)

- Tries 2 minutes of HIT to check it out, if she is trying HIT it is because other people have said it was lucrative for them. Acknowledges wide variation in performance and efficacy due to differences in worker skill level.

- Checks community boards frequently when assessing to fill a potential HIT

Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

- "Number of active Turkers is between 15,059 and 42,912; and that 80% of the tasks are carried out by the 20% most active (3,011–8,582) Turkers."

- "Turkers (56%) are U.S. based, but there is a growing number of Indian Turkers (36%) and other nationalities. Nearly one-third of respondents had a median annual income of <$10,000."

- Common issues for Turkers include "employers who don’t pay; identifying scams; the cost (to workers) of poorly designed tasks."

- Turkers use 'Requesters hall of fame/shame' forum to report and warn community on Requesters

- There is a forum for 'community interests' and 'prayers and good vibes'

- Generally, wages made from HITS are not sufficient enough for full-time employment

- Workers communicate good HIT design to Requesters

- There is a subcommunity of Workers who explain how to cheat tasks to other

- Generally, the community of Workers encourages each Worker to be cordial with Requesters in event of quarrel


2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

- Requesters will use "Majority Rules" metric to assess quality of HITS

- Can block Workers (who Amazon may suspend from MTurk) but can't be blocked by Workers


Worker perspective: Turkopticon

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

- Timeliness of payment is a major concern

- Willing to work with same Requesters on an ongoing basis if a good business relationship has been formed

- Not a unified body of people, just some similar requests


2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

- Will take negative reviews personally and seek retribution

- Exhibit malicious behavior towards workers


Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

- A small number of Workers choose to cheat Requesters and the system, corrupt the reputation of the workforce.

- Cheating Workers don't even try to complete a HIT when they see a qualifying test


2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

- Requesters use qualifying questions or tests to filter cheating Workers

- May disqualify Workers from future tasks based on performance on past tasks


Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

- Crowdsourcing task platforms enable greater opportunity for workers in impoverished countries to access greater pay for their work

- Standardized HIT design can be tremendously beneficial for overall Worker comprehension

- Workers, especially unskilled ones need handholding and a formalized process in order to be productive


2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

- Providing standardized wage can incentivize Requesters to increase wages for tasks they need to get done quickly

- Giving an accurate estimation for how long a task will take is an important component needed for a satisfied Requester


Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

- Small batches are their favorite - larger batches, especially from unknown Requesters scare the herd

- Willing to leave MTurk for platforms with better design and reputation engines


2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

- New Requesters are not properly onboarded onto MTurk to understand consequences of certain actions and the stigma of their role as a new requester

- "Masters" in the MTurk ecosystem who can pay and reject as they see fit, maniacally laughing at the shattering of dreams and lowering of worker ratings

- Inefficiencies of platform including no estimated completion time, are seen by the Requester as unreliable and lazy workers

Synthesize the Needs You Found

Worker Needs

  • Workers need a way to rate, report, and block Requesters. Evidence: The potential for Requesters to block Workers was discussed, where Amazon would eventually suspend a Worker's account if blocked by too many Requesters was discussed in the paper 'Being a Turker'. Also, Workers could be disqualified by Requesters for future tasks based on their previous performance as discussed in 'Crowdsourcing User Case Studies'. Interpretation: The power dynamic is unbalanced and leans in the favor of Requesters who are the privileged class on Mturk thus wielding superiority on the Working class. Third party sites like Turker Nation cannot act as the community policing agent forever. Without a systematic approach to assessing the reputation of Requesters (within MTurk), the platform will continue to be corrupt.


  • Workers need a way to ensure the Requester will pay in a fair and timely manner. Evidence: Timeliness of payment was one of the major issues discussed in the "Turker Bill of Rights" in 'Turkopticon'. Laura mentioned in the panel that getting the rejection is a constant threat and fear of hers and one that has influenced her behavior working with new Requesters. Interpretation: Because there is no real in-process regulations regarding fair payment, Requesters are allowed to do whatever they want. There needs to be a clearinghouse type of system that locks in a contract between two parties when the Worker takes on a HIT. Taking the power of payment out of the Requesters hand makes the process less corruptible.


Requester Needs

  • Requesters need a way to filter out ineffective workers from their HITs. Evidence: In the panel interview, Peter mentioned that he posts qualifying questions in order to make sure those workers who respond to his HITS are effective workers - he also mentioned that rejecting people will hurt his reputation with the community. Interpretation: There is a general practice of filtering but no standardized method. Furthermore, the Requester acknowledges the power of the community and their unwillingness to work with Requesters who reject many workers.


  • Requesters need a standardized method of requesting, estimating, and communicating on Mturk. Evidence: Both Chris and Peter mentioned during the panel interview that they check multiple times to ensure their HIT instructions are clear. Also, they mentioned that communication between the two parties is essential for an effective partnership. The paper 'Being a Turker' discussed how workers will communicate specifically about good HIT design in order to complete tasks more effectively. Interpretation: There is a fundamental breakdown in communication between the two parties in which the requester spends too much or too little time designing their HIT, sends it out and is unsure whether their task will be filled - or filled correctly for that matter. And on the other side, the Worker suffers from having to interpret new forms of HIT thus confusing and/or limiting the HITs they are willing to take on. This lack of standardization narrows and corrupts the number of tasks in the ecosystem.