Milestone 1 TeamInnovation2

From crowdresearch
Jump to: navigation, search

Our team consists of three seasoned MTurk workers, which is the perspective reflected in the content below.

Experience the life of a Worker on Mechanical Turk

Pros of using MTurk as a worker:

  • It's reasonably simple to sign up for. There's no interview process, unlike with a traditional job, and you can start looking for work as soon as your account is open.
  • Work can be done from anywhere, as long as there's an Internet connection and a way to access the web site.
  • Workers can truly work whenever they need to work, and don't have to ask anyone's permission to take a day off or change my schedule.
  • If a task is too stressful or unpleasant for some other reason, workers can choose not to do it.
  • There's a variety of different types of work available, which keeps things interesting.
  • There's a lot of work available, so there's always something to do.
  • Workers can withdraw the money earned at any time, and it appears in our linked bank accounts or in Amazon Payments quickly and reliably.

Cons of using MTurk as a worker:

  • Often, the pay is abysmal - for many tasks, pay isn't even equivalent to half the local minimum wage.
  • Not only are pay rates low, the pay is often disproportionate to the effort and time required to complete a task.
  • Searching for tasks that are suited to a given worker is inefficient and takes up valuable time that might be better spent working on tasks.
  • There's no built-in feedback system to filter requesters, so workers have to use external resources to check whether a requester is reliable or reasonable.
  • Although theoretically there's a variety of work types available, in practice, it can be difficult, convoluted, or even impossible to get access to desirable work.
  • The web site interface is clunky and non-intuitive.
  • Methods of getting qualifications aren't always obvious, so some work is inaccessible even to workers who might be able to do it well.
  • There's a lack of stability, in terms of specific tasks. There's no guarantee that a given requester will be on MTurk for the long term, so workers may need to have fingers in a lot of pots in order to feel like there will always be work that's worth doing.
  • You don't get much for being a dedicated or skilled worker. You do get ratings (like the approval rating), but there's no reliable correlation between pay rates on tasks and good ratings.
  • There's not really a sense of cohesion among workers, except where they gather together off the MTurk site, and some workers may miss the camaraderie of being part of a work group that can directly share experiences and feedback.
  • Workers may not feel much of a sense of warmth or loyalty toward MTurk, as it seems very impersonal, and oriented more toward making other people money than toward being a user-friendly experience for the workers, who are a necessary part of the system.
  • There seems to be no quality control for tasks that are posted, so some of them are really badly designed or broken, and it's up to workers to inform requesters, who aren't always responsive for one reason or another.

Experience the life of a Requester on Mechanical Turk

Pros of using MTurk as a requester:

  • It's pretty quick to get a task running and open for workers.
  • As sad as it is for workers, because the bar for MTurk wages is set really low, work can be done for a very small investment.

Cons of using MTurk as a requester:

  • The interface for requesters is just as clunky as the interface for workers, so it's a pain to use and requesters may want to spend as little time as possible using it. If a requester doesn't have a strong motivation to use it, they might give up before ever posting work.
  • The process of creating tasks isn't intuitive, and uses foreign terminology that may only be familiar to established workers. One of our team still had to reference the help files several times while figuring it out. Even the help interface seems like it was designed with someone other than a beginning requester in mind, though, and the aforementioned team member had to hunt around for information that probably should have been accessible from within the HIT creation interface itself.
  • If there are hard and fast guidelines that should be followed as a requester, they aren't presented in an obvious and user-friendly way. Even the best practices file was hidden away in the walls of text that make up the requester interface.
  • The editing part of the HIT creation interface isn't simple to use, e.g., one of our team members wanted to add questions to a survey and didn't see a simple way to do it, and cutting and pasting in the non-HTML view of the editing screen didn't seem to work. She resorted to editing the HTML source for the survey HIT by hand, but not everyone will have that option. (Or are MTurk requesters all supposed to be coders?)
  • For one team member, the process of funding the first HIT was frustrating and unclear. For one thing, it isn't obvious that requesters are required to buy prepaid HITs — it seemed like calling them prepaid implied that there is another option, but if there is one, it wasn't clearly visible. The notification that the team member's first attempt at funding had failed didn't appear in the MTurk interface, but in e-mail, so it wasn't noticed right away. Information about the different funding methods was not readily available, so trial and error had to be used to find out what would work, and how, and even when the process of funding succeeded, some of the details were unclear.
  • When creating a task, the qualification settings are automatically set to allow only workers with the Masters qualification to do HITs, and this isn't obvious at all, so if a requester isn't paying attention, they may end up unnecessarily restricting the pool of workers who have access to tasks.
  • There's no way to use the MTurk system itself to hand-pick and invite workers who are qualified to do my work.
  • There's no built-in system for communicating with workers within the MTurk interface.
  • The purposes of different parts of the system (qualifications, batches, rejections) aren't immediately obvious to a beginning requester, and it's easy to make mistakes that will annoy or alienate workers.

Here's an example CSV file for a HIT from one of our team members: Media:Batch_1845062_batch_results.csv The HIT was a survey regarding fiction reading preferences.

Explore alternative crowd-labor markets

oDesk is the alternate market chosen for comparison with MTurk.

Differences between oDesk and MTurk:

  • oDesk immediately seemed more appealing than MTurk, at a cursory glance for both those looking for work and those looking for workers. The site is designed much more thoughtfully, and it's more welcoming, which immediately gives a more positive impression.
  • oDesk allows workers to create a profile to showcase their talents to people who are hiring, which is a pretty major advantage over MTurk for both workers and employers. However, one team member didn't like that jobs couldn't (apparently) be browsed without putting time into creating a profile.
  • oDesk requires a lot more information of workers before they can start working.
  • oDesk also requires that profiles be approved by its staff before they can go live, so access to work is more restricted than access to work on MTurk.
  • oDesk offers different membership plans (and indeed is selling the idea of "membership" where MTurk is not), one free and one paid, with different feature sets, while on MTurk, everyone has access to pretty much the same features.
  • If there's a way to browse oDesk by work category, or to browse a list of all the available work, it isn't obvious. One team member had to use the search box to look for specific criteria, which was less convenient than being able to browse all available tasks on MTurk.
  • In the job listings, it was again obvious that the interface is much more user-friendly than MTurk's. There's more information on each job. Jobs can be filtered by many more criteria. Workers can save job searches. Overall, the job search interface is a huge improvement over MTurk's.
  • It appears that employers can set more (and more useful) parameters on their job postings. For instance, they can require that workers have a certain level of fluency in a language or that they have certain skills.
  • You have to apply for jobs on oDesk, even if you're theoretically qualified for them, instead of just accepting work and doing it right away as is done on MTurk, so it seems as if oDesk is less of a work platform than MTurk, and more of a facilitator that matches employers to workers, who then use their own resources to perform and assess work.
  • There's more information available on employers right on oDesk, so there's no need for an independent, off-site rating system for employers.
  • oDesk appears to limit the amount of work that you can apply for (and maybe this is related to the membership level), whereas MTurk will allow you to take on as much work as you think you can do.
  • There's actually somewhat less information on oDesk about the jobs themselves, as it's up to employers to describe the job, while on MTurk you can preview the HIT or accept it to see exactly what you're getting into. You can also return HITs on MTurk without interacting with the employer and without any serious penalty, whereas on oDesk it seems likely that once you have a job, there's an expectation that you will complete it.
  • The types of work available on each site are different. MTurk is designed for short tasks to be completed by many people, and oDesk is designed for larger, more complex projects to be completed by fewer people.

Similarities between oDesk and MTurk:

  • Registering was easy on oDesk, but it was also fairly easy on MTurk.
  • On both sites, workers can search for work in their areas of interest and with their preferred scope and level of pay.

In sum, it seems like MTurk, despite its flaws, is better for getting work quickly, and for fitting work into small amounts of time, while oDesk has a format that feels more traditional (and slower), where you're encouraged to show off your skills, build a reputation, and commit to larger projects.




  • Work can be done anywhere, at any time of day, even while doing other things, like traveling or watching TV.
  • Accessible on relatively low-tech mobile phones, which allows people without computers and with limited financial resources to do crowdsourced work.
  • Connects worker quality ratings with pay.
  • Delivers 89 percent or higher accuracy for employers.
  • Because workers always do the same type of work (on the current platform), they have the chance to improve their skills as they do more work.

Things that could be improved:

  • The type of work available through the platform is limited, so it's not useful to people who want to do other types of work or who aren't capable of doing the type of work offered.
  • This platform could potentially be less appealing to workers who are not at the bottom of the economic pyramid, who have more resources and may therefore have greater access to higher-paying work.



  • Accessible on relatively low-tech mobile phones.
  • Work can be done anywhere, any time.
  • Potentially uses a leaderboard system to provide incentive for workers to do well.
  • Gives workers feedback and encouragement to keep them working.
  • Rewards workers for referring other workers.
  • Designed to provide supplementary income that workers would not otherwise earn.

Things that could be improved:

  • Language and font issues currently limit the efficiency of the system.
  • Focuses on a specific type of work, so workers who don't want to do that type of work or can't do it will not find this platform useful.
  • Delivers a relatively low accuracy, compared to the market standard for this type of work.
  • Data entry is slower than it is on a computer.
  • Not designed to provide full-time work.

Flash Teams


  • Links together modular tasks in a workflow to achieve large, complex goals.
  • Uses experts as workers, which is likely to result in high-quality work.
  • System is elastic, and allows for hiring more workers as needed.
  • Uses a web application to direct the team working on a given goal or project.

Things that could be improved:

  • Since a certain level of expertise is required, work seems likely to be of limited availability, and unavailable to people without the proper expertise.
  • Since work is based on large projects and goals, there may be strict deadlines, and it may not be as easy for workers to do tasks in their leisure time.
  • Uses another web site, oDesk, for recruitment, rather than using its own recruitment system, so if oDesk has issues, then Flash Teams also has issues.