Milestone 2 TeamInnovation2
[Team Innovation 2 is experiencing technical difficulties unrelated to this wiki! This is a work in progress.]
- 1 Attend a Panel to Hear from Workers and Requesters
- 2 Reading Others' Insights
- 3 Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc
- 4 Synthesize the Needs You Found
Attend a Panel to Hear from Workers and Requesters
Observations From the Panels
- Work hours on MTurk are very variable between workers.
- Work availability on MTurk is also very variable.
- Some MTurk workers fit Turking into their existing schedules, and other workers make their schedules around MTurk. The ones who fit Turking into their schedules may make less money since they are not available as often to discover work as it is posted.
- Client/requester work hours and worker work hours don't necessarily match up.
- On both oDesk and MTurk, those requesting or posting work can invite specific individuals to do their work, but this seems easier on oDesk than MTurk.
- On oDesk, workers need to convert their experience into skills that display well on the web site.
- Beginning workers on oDesk may find it difficult to prove their skills to prospective employers.
- Workers on oDesk who are invited to do work need to filter their invitations to decide which to accept.
- Some workers on oDesk provide prospective clients with sample work when applying for a job.
- On oDesk, beginning workers may have trouble getting work, because many jobs require previous experience within the oDesk system.
- The amount of money workers make crowdsourcing is highly variable with the time of the year, the day of the week, etc. It is often influenced heavily by the academic schedule, as a lot of work is related to academics.
- MTurk workers trying to find work they're interested use tags to search. Good tags help workers find good tasks.
- Rejecting work on MTurk is very harmful to workers because it can severely limit their access to work. This causes some requesters to avoid rejecting work. It also means workers are very concerned with their approval ratings, and workers spend time and effort trying to boost their ratings by contacting clients or requesters.
- Requesters who give rejections on MTurk receive a lot of feedback from dissatisfied workers, which takes a lot of time to deal with.
- Workers' approval rating does not apply to specific tasks. Requesters can't tell, for instance, if a worker has a 99 percent approval rating in transcription tasks only, or if they have a 99 percent rating in categorization and a 0 percent rating in transcription.
- There are many ways a requester can do attention and honesty checks. Many requesters include at least one of these checks in each HIT (for instance, asking a question with a previously verified answer).
- Requesters don't always know who their target audience or worker is.
- Researchers don't always know for sure that answers are truly coming from their target demographic.
- Techniques for screening people for their demographic are nonstandardized across requesters.
- Personalized tasks are difficult and complex to run on MTurk.
- Requesters on MTurk use a wide variety of techniques to design their HITs effectively. Some of them use worker feedback on pilot tasks to guide the creation of the actual task.
- Requesters prefer to find the right people and have them do a task well rather than having a task done by people and then having to determine whether the work was done properly.
- It takes valuable time to create a test task, but it can also help requesters learn about their own tasks.
- Sometimes putting together a non-crowdsourced solution takes as much time and effort as designing a way to crowdsource a given piece of work.
- MTurk requesters get higher quality results by breaking large tasks down into smaller tasks, but oDesk clients are often looking for a smaller number of workers to perform work that makes sense if done by one person (like writing a single paper).
- Requesters on MTurk are discouraged from rejecting work, thus high approval ratings are not necessarily indicative of high work quality.
- There are many different types of qualifications and ratings requesters use to filter who can work on their tasks. Some filter the whole population, and some allow only hand-picked workers access to work. Some do help requesters get higher quality work, and others don't. Different requesters have different experiences with using qualifications to seek higher-quality work.
- The Masters qualification on MTurk does not eliminate spam results, even though it costs more to use it.
- One efficient way for MTurk requesters to get high quality results is to form a customized pool of qualified workers.
- Requesters find it difficult to reach a mutual understanding of tasks with workers, sometimes because the requester doesn't fully understand the task yet, and sometimes because it is difficult to communicate what the task is and how to do it.
- Requesters find it difficult to find skilled, honest workers.
- Some requesters are not skilled in rating workers accurately, which skews ratings.
- Requesters often get work from workers they do not want work from (for instance, a researcher looking for female respondents will get responses from males).
- When a batch of HITs doesn't get completed, a requester might e-mail people or ask on one of the MTurk communities about pricing, then reprice and relaunch the task. Alternately, if the instructions are unclear they might need to be clarified. Sometimes there's a bug in MTurk that keeps workers from seeing a task.
Reading Others' Insights
Worker perspective: Being a Turker
- Workers on MTurk are primarily motivated by earning money.
- There are benefits of Turking aside from monetary gain, but workers consider these to be side benefits.
- Workers do different amounts of work and have different earnings goals on MTurk, which are related to their other life circumstances. Some people do it for spare change, and some people do it for full-time income.
Worker perspective: Turkopticon
- The Turkopticon rating system, which is separate from MTurk, is a staple tool for many MTurk workers. Workers use it to rate and review requesters and to read other workers' reviews and ratings.
- There is no requester rating system on MTurk itself.
Requester perspective: Crowdsourcing User Studies with Mechanical Turk
- Work requested on MTurk is highly subject to malicious user behavior.
- Work sourced through MTurk is not likely to be of the same quality as work performed by experts.
- When requesters design tasks well, this strongly influences the quality of the work performed.
Requester perspective: The Need for Standardization in Crowdsourcing
- Many crowdsourcing tasks require workers to adhere to a strict set of instructions.
- There are scammers (and spammers) in both the worker population and the requester population.
- It is difficult to get consistently good results with crowdsourcing. People who crowdsource work often have to develop their own original methods to get good results from workers.
- There are types of crowdsourced work that appear over and over again on the market.
Both perspectives: A Plea to Amazon: Fix Mechanical Turk
- The MTurk API is complex and challenging to use for both requesters and workers.
- Amazon is not very responsive to the needs of the people using MTurk, and does not often make changes to the system.
- Both workers and requesters are dissatisfied with the rating system on MTurk. It does not accurately represent workers, and it does not provide ratings of requesters at all.
- MTurk workers often share good work with each other. For instance, in the daily HIT threads on Turker Nation: http://turkernation.com/forumdisplay.php?157-Daily-Threads
- Some requesters recruit workers from the Turker communities off the MTurk site. For instance, there is a post on Turker Nation inviting forum members to participate in a HIT specifically for Turker Nation members: http://turkernation.com/showthread.php?23735-Well-paid-survey-HIT-for-TurkerNation-members-only&p=245585&viewfull=1#post245585
- MTurk workers turn to each other for help with troubleshooting virtually every aspect of Turking, on topics like finding good work, technical difficulties with the MTurk site, and how to do certain types of work. For instance:
- Workers ask whether problems they are having are common ones: http://turkernation.com/showthread.php?23753-Wasted-time-doing-an-un-submittable-hit-and-wondering-if-it-s-a-common-issue
- Workers ask each other about work availability: http://turkernation.com/showthread.php?20581-Is-there-more-work-at-different-times-of-the-year
- Workers ask about whether the site is accessible from different devices: http://turkernation.com/showthread.php?21874-Does-anybody-MTurk-with-smart-devices-or-tablets-only
Synthesize the Needs You Found
List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.
- Workers need to thoroughly understand the web site they use to deliver work. Evidence: harrisnyc posted a thread on Turker Nation asking about what "HITs available" actually means: http://turkernation.com/showthread.php?23485-Silly-question-about-quot-HITs-Available-quot Interpretation: There's a lot of unexplained terminology on MTurk, and people want to know what these things mean in case they're important to the work experience.
- Workers need to know how much they will make working. Evidence: hidingmyidentity asked on Turker Nation about how much money it's possible to make on MTurk: http://turkernation.com/showthread.php?21815-Can-I-get-an-estimate-of-how-possible-it-is-to-make-money-off-of-mturk Interpretation: This question is actually about finding out whether working on MTurk will be worth the time invested. People don't want to do a job without knowing they'll get paid enough for it to be worth it.
- Workers need to reliably know when they will be able to make money. Evidence: Stripeypants started a thread on Turker Nation asking whether there is more work available at different times of year. http://turkernation.com/showthread.php?20581-Is-there-more-work-at-different-times-of-the-year Interpretation: Basically, this question is about anticipating when work will be available. Whether it's related to time of year or time of day, people need to know when they are going to have to show up to get the money they want to earn.
- Workers need to learn how to become good workers. Evidence: The Turker communities outside of MTurk itself are busy places, full of questions on many topics. Interpretation: People have a lot of questions about how to Turk effectively, or how to Turk at all, and because there is a lack of support on MTurk itself, they have created their own communities where people can share experiences and knowledge.
- Requesters need to find workers who have the skills to deliver high quality work. Evidence: In the second panel discussion, Dahn talked about how MTurk's rejection-based approval rating system gives an inaccurate picture of work quality. Several requesters on the panel also talked about how they have many different methods for filtering workers and for doing attention and honesty checks. Edwin mentioned that when he wants really high quality work, he uses a hand-picked pool of workers that he knows from past experience. Interpretation: All of these efforts are geared toward finding workers with the right skills to perform good work.
- Requesters need to effectively communicate their needs to workers. Evidence: In the second panel discussion, Niloufar mentioned that one of the most difficult things about being a requester on MTurk is coming to a mutual understanding with workers of what a task means. Edwin mentioned that he sometimes puts up pilot tasks and uses worker feedback from those tasks to design the real task he wants done. Interpretation: Both of these requesters are dealing with the complexities of communicating what they want to workers, so that they can get the results they want.