WinterMilestone 2 kbourque

From crowdresearch
Jump to: navigation, search


Attend a Panel to Hear from Workers and Requesters

Deliverable

Report on some of the observations you gathered during the panel.


WORKERS

There are a lot of “back door” solutions to earn more money, ones that aren’t really apparent to newcomers. Workers help each other out, if you’re in the community. The workers on this panel seem to choose this work due to its flexibility. However, they also must be flexible themselves as a result: not having a regular 9 to 5 must mean that sometimes you have to work on off-hours.

Worker approval is really important for workers. In terms of choosing work, better ratings are more attractive. Time estimates are off, “you can’t rely on it”. But it depends on the individual! Mechanical Turk is very unreliable in terms of pay.

It seems that people are looking for better communication.

REQUESTERS Workers are pretty responsive with feedback through e-mail, but there could be improvements made. There is some incentive to not rate workers badly (library book example).

BOTH

People use e-mail quite often for communication between workers and requesters.

Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

Pay is not the sole motivator for workers on mechanical turk, but it is the primary one. Workers on sites like Turker Nation are usually from the US. The type of the task affects the worker’s decision on whether to accept the task. Fair pay is a large concern for workers. Many turkeys have boundaries that prevent them from making a decent wage without Mechanical Turk, whether it be inability to work a full time job or the fact that their current job does not pay enough. Workers seek more communication with requesters; they find that good communicators make for good requesters. The mobility of Amazon Turk is attractive for workers, and may be a worthwhile trade off to some. Workers tend to be divided into two groups: new ones who focus on getting as many HITs as possible to increase their approval rate (which is why they do not make as much), and experienced ones who have already built up the approval rating and as a result are more conservative with the HITs they choose. Turkers fear a different system due to the possibility of failure, even though AMT is highly flawed.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requesters often give unfair wages. Requesters can have unrealistic expectations for workers. Requesters that are prompt and that are willing to communicate are most popular with workers. Many requesters have not properly experienced the workers’ side of things.

Worker perspective: Turkopticon

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

Workers take value in community, but they also have different expectations and backgrounds which makes it difficult to standardize the worker experience. Indian Turkers tend to be well-payed and highly educated in comparison to US Turkers.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

It takes considerable effort in to construct tasks in the first place, which may make requesters (especially large requesters) less willing to comply with workers’ numerous requests.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

Workers are not all the same. you can’t really tell how workers will be in general, because they come from diverse backgrounds. you cannot assume many workers will be experts at the tasks you give them. there are a lot of issues with malicious workers, which makes the experience worse for everyone.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

This makes it difficult for requesters to be able to create tasks that produce optimal results. However, requesters must format their tasks in a way that gives them more accurate results anyway. work that may require excessive thought may not be best for accuracy.

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

Workers are not all trained to be the same level, and they should be in order to produce standardization. Workers are not always trustworthy. Knowing about a requester detracts from the quality of work. Workers often do not give honest feedback.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requesters should work to create tasks that are in a standardized format. Unsuccessful tasks are usually ones that are not designed “narrowly” enough. Requesters do not have as accurate pay rates as they should. Requesters often do not give honest feedback.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

Workers do not go through proper qualification tests. Experienced workers are hesitant to work on new requesters’ work due to unreliability. Experienced workers rely heavily on reliable and accurate ratings. New workers may not know about this.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Most requesters are not exactly technically savvy. Requesters tend to rate more negatively than positively. Requesters can also be fraudulent; they have the option to take work for free essentially.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

List out the observations you made while doing your fieldwork. Links to examples (posts / threads) would be extremely helpful.

Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

A set of bullet points summarizing the needs of workers.

  • Example: Workers need to be respected by their employers. Evidence: Sanjay said in the worker panel that he wrote an angry email to a requester who mass-rejected his work. Interpretation: this wasn't actually about the money; it was about the disregard for Sanjay's work ethic.

1. Workers need to be able to effectively communicate with requesters. Evidence: on Turker Nation, workers seemed to positively review requesters that responded to their concerns quickly and thoroughly. Interpretation: Workers highly valued responsive requesters.

2. Workers need to work within their own specialties. Evidence: The diversity of workers, ranging from the US to India and from low-income to high-income folks all come from different backgrounds, yet they have access to the same types of work. Interpretation: Systems like AMT try to fit all workers inside a certain box, which leads to unrealistic expectations and disappointing results.

Requester Needs

A set of bullet points summarizing the needs of requesters.

  • Example: requesters need to trust the results they get from workers. Evidence: In this thread on Reddit (linked), a requester is struggling to know which results to use and which ones to reject or re-post for more data. Interpretation: it's actually quite difficult for requesters to know whether 1) a worker tried hard but the question was unclear or very difficult or an edge case, or 2) a worker wasn't really putting in a best effort.

1. Requesters need to easily create their tasks on a standardized system. Evidence: “Every employer has to implement from scratch the “best practices” for each type of work. For example, there are multiple UI’s for labeling images…” Interpretation: If a requester is not experienced, they have the potential of creating a badly designed HIT. This leads to negative results on both the requesters’ and workers’ sides.

2. Requesters need to provide examples of standardized results. Evidence: From Turker Nation, iambob’s post: “I have done a number of his hits. Then one day I get this email: … If you spent half the amount of time looking for a job as you spent scamming your way through…” Interpretation: Worker did not meet requester’s expectations, but requester seems to not have properly communicated them and the requester’s expectations have not aligned to similar jobs that iambob has completed before.


Contributors

@kbourque