Talk:WinterMilestone 2 UX-Nick

From crowdresearch
Jump to: navigation, search


Attend a Panel to Hear from Workers and Requesters

Winter Worker-Requester Panel Notes

Observations -

The jobs resume qualifies her for are not physically possible for her. Felt satisfaction working fast food, “good at it” - fast paced mindset suits her well but she can’t do it anymore - has a joint disorder. As a result - started “Turking” - $10 a day at first, then she had a $50 day and a $100 day and was amazed.

“What’s a typical day feel like”

Gets up, helps kids, runs scripts to find jobs and goes through the thread of “TurkerNation” - people actually help and teach each other how to make it work - encourage ethical behavior between people. “I don’t really schedule breaks, it depends on what’s available - if my favorite HIT is up i’ll just do that - it’s unpredictable because you don’t know when requesters will post HITs” Trying to achieve a goal despite large variation in HIT listings, etc.

“What works well for task design”

How do we clearly convey what we want done The kinds of tasks where someone is trying something for the first time, versus the types of tasks that occur all the time (high volume.) “I never risk going below a 99% approval rating” “We never rejected a HIT, but then we realized we had a few workers in there who were submitting crap results” Hired an undergrad who had a job of monitoring behavior on mTurk

“How do you use mTurk well”

Grappling w attention span - how do you make sure people are carefully and honestly answering questions. “I know TurkerNation and Christine and her people, the problem of honest and good workers has largely disappeared”


We don’t use any of the built in templates. People accepting “HIT”s and then not realizing how much time it will actually take. Idea for people to come back and report how long something actually took.

How variable are earnings?

Earnings are all over the place Difficult to budget for if you need money at a certain time

What does rejection feel like?

“Sometimes I accept and reject proportionately to get rid of people who do poorly on tasks, it’s a tricky subject” “For a new worker, it’s death” “I will screenshot contact information, phone numbers and emails” If there’s a rejection, I go to the requester immediately

If you could use 5 words to describe your experience w workers/requesters, what would they be?

“Incredible enablers of scientific research”


“anonymous frustrating complicated tricky “absolutely unpredictable, great, terrible” “frustrating, lonely, isolated, empowering” “diverse passionate futuristic dynamic human”

Reading Others' Insights

Worker perspective: Being a Turker

Being a Turker

(Martin, Hanrahan, O’Neill)

Turkers report that their main motivation to Turk is earning money, but this is hard to believe as most of the time they are doing the work for a few cents. Requesters have a lot more info about Turkers than Turkers have about requesters, and there are allegations of “deception” and “lack of privacy” on the part of the Turkers against the requesters. Plugin to AMT allows Turkers to rate requesters Turker Nation allows for the Turker community to congregate and share ideas. Over half a million people find work at pay rates they are “happy with” but that are well below minimum wage. While in India 15k is a good salary, a great deal of Turkers are using the site as a means to slightly bolster living hand to mouth.

Turkers rate requesters on pay, responsiveness, rejections. Turkers do not have a reciprocal system to block and it’s complicated for them to prove innocence. Pay on AMT is considered “fair” when hourly rate aligns somewhat closely to minimum wage in the U.S. High approval rating is crucial to wider selection and better paying HITs

Turkers worry that interference from academics will lead to requesters fleeing the marketplace, not wanting to pay an instituted “hourly wage”. They are afraid of regulation leading to them not having work. There are both ethical and practical implications to this study. Turkers have been misunderstood /denigrated and better engagement w them can help design better HITs and technologies for their relationships w requesters. Concluding thoughts : Turkers want the best info on how to select jobs, interact with requesters, act collectively.

Worker perspective: Turkopticon

“You cannot spend time exchanging email. The time you spent looking at the email costs more than what you paid them.

This has to function on autopilot as an algorithmic system…and integrated with your business processes.”

Survey research on Turk worker motivations, for example reports that though a significant minority of workers rely on their income from the platform to pay for household expenses.

Because Amazon collects money for task volume, Amazon has little reason to prioritize worker needs in a market with a labor surplus.

Because AMT treats workers interchangeably and because workers are so numerous (tens of thousands by the most conservative estimates), AMT can sustain the loss of workers who do not accept the system’s terms.

Although quantification has myriad problems as a description of lived practice, Turkopticon employs tactical quantification to enable worker interaction and employer accountability while integrating into the rhythms of AMT.

Tactical quantification is a use of numbers not because they are more accurate, rational, or optimizable [10], but because they are partial, fast, and cheap – a way of making do in highly constrained circumstances.

Workers’ responses revealed vastly disparate visions and self understandings when it came to issues of minimum wage, relations with requesters, and desire for additional forms of support.

Ideally, however, we hoped that Amazon would change its systems design to include worker safeguards. This has not happened. Instead, Turkopticon has become a piece of software that workers rely on funded through subsidies from academic research – an unsustainable foundation for such a critical tool

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

It’s often challenging to acquire user input in both a timely and cost-effective manner, especially in a micro-task market like mTurk. First, an important driver of the success of the system appears to be the low participation costs for accepting and completing simple, short tasks Diverse user base of mTurk a benefit and a drawback. Experiment conducted having Turkers rate the quality of wikipedia articles. Correlation between Wiki admin ratings and Turker wiki ratings was very weak. It appeared that rather than widespread gaming, a small group of users were trying to game the system multiple times. New rating system was done in experiment 2 , with article keywords and some questions about the images, sections, etc. in the article asked for the experiment. Differences in the failure of experiment one and success of experiment 2 came down to: First, it is extremely important to have explicitly verifiable questions as part of the task. Second, it is advantageous to design the task such that completing it accurately and in good faith requires as much or less effort than non-obvious random or malicious completion.

Requester perspective: The Need for Standardization in Crowdsourcing

Despite the excitement and apparent industry maturation, there has been relatively little innovation—at least at the micro-work level—in the technology of how workers are allocated tasks, how reputation is managed and how tasks are presented etc. As innovative as MTurk is, it is basically unchanged since its launch. Advantages to standardizing basic tasks - reusability, trading commodities, true market pricing. Standardizing workflows will allow for optimization of Turkers and requesters in the mTurk market.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

A Plea to Amazon: Fix Mechanical Turk

Requesters need - a better interface to post tasks and a better appraisal system for Turkers. Suggestions - rating system, keep track of working history, public qualification tests, separate HITs and ratings by type - make all this accessible by API. Compares mTurk to slaves and masters - that the requesters are the “masters” that can choose their slave! Good suggestions - suggest an appeal rate for requesters, rejection rates and how many appeals are won - total volume of posted work. Disallow ability to reject work that is not spam - and make all this accessible from an API. browsing system by task type Adopt a recommender system to propose HITs to workers

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

List out the observations you made while doing your fieldwork. Links to examples (posts / threads) would be extremely helpful.

Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

•Better time estimates from requesters for how long a task would take. •Increased requester accountability for rejected work. •Better task organization - most requested tasks versus more fringe-type tasks • More wage standardization and cost of living adjustments •Better ways of findings HITs

Requester Needs

•More accountability for workers who try to game the system versus honest workers •Better methods for rating workers who do quality work •Easier methods to find the correct workers for the job.