Milestone 1 TeamInnovation2

From crowdresearch
Revision as of 19:07, 4 March 2015 by Crystalcalhoun (Talk | contribs) (Experience the life of a Requester on Mechanical Turk)

Jump to: navigation, search

Our team consists of three seasoned MTurk workers, which is the perspective reflected in the content below.

Experience the life of a Worker on Mechanical Turk

Perspective 1

I've used MTurk for a few years to help me pay bills and household expenses, and that's the perspective from which I approach this.

Things I like about using MTurk as a worker:

  • It's reasonably simple to sign up for. There's no interview process, unlike with a traditional job, and you can start looking for work as soon as your account is open.
  • I can work from anywhere, as long as I have an Internet connection and a way to access the web site.
  • I can truly work whenever I need to work, and I don't have to ask anyone's permission to take a day off or change my schedule.
  • If a task is too stressful or I don't like it for some other reason, I can choose not to do it.
  • There's a variety of different types of work available to me, which keeps things interesting.
  • There's a lot of work available, so there's always something to do.
  • I can withdraw the money I earn whenever I want, and it appears in my bank account quickly and reliably.

Things I don't like about using MTurk as a worker:

  • Often, the pay is abysmal - for many tasks, pay isn't even equivalent to half the minimum wage where I live.
  • Not only are pay rates low, the pay is often disproportionate to the effort and time required to complete a task.
  • Searching for tasks that are suited to me is inefficient and takes up valuable time I would rather be using to work on tasks.
  • There's no built-in feedback system to filter requesters, so that I have to use external resources to check whether a requester is reliable or reasonable.
  • Although theoretically there's a variety of work types available, in practice, it can be difficult, convoluted, or even impossible to get access to the tasks I want to work on.
  • The web site interface is clunky and non-intuitive.
  • Getting qualifications isn't obvious
  • Lack of stability. Even if I find a requester that I really like, there's no guarantee that requester will be on MTurk for the long term, so I really need to have my fingers in a lot of pots in order to feel like I'll always have work that's worth doing.
  • You don't get much for being a dedicated or skilled worker. You do get ratings (like the approval rating), but there's no reliable correlation between pay rates on tasks and good ratings.
  • There's not really a sense of cohesion among workers, except where they gather together off the MTurk site, and I kind of miss the camaraderie of being part of a work group that can directly share experiences and feedback.
  • I don't feel much of a sense of warmth or loyalty toward MTurk, as it seems very impersonal, and oriented more toward making other people money than toward being a user-friendly experience for the workers, who are a necessary part of the system.
  • There seems to be no quality control for tasks that are posted, so some of them are really badly designed or broken, and it's up to workers to inform requesters, who aren't always responsive for one reason or another.

Experience the life of a Requester on Mechanical Turk

Perspective 1

I've never used MTurk as a requester before this week, so my experience with that side of things is limited.

Things I like about using MTurk as a requester:

  • It's pretty quick to get a task running and open for workers.
  • As sad as it is for workers, because the bar for MTurk wages is set really low, I can get work done for a very small investment.

Things I don't like about using MTurk as a requester:

  • The interface for requesters is just as clunky as the interface for workers, so it's a pain to use and I want to spend as little time as possible using it. If I didn't have a strong motivation to use it, I might give up before ever posting work.
  • The process of creating tasks isn't intuitive, and uses terminology I'm only familiar with because of being a worker already. I still had to reference the help files several times while figuring it out. Even the help interface seems like it was designed with someone other than a beginning requester in mind, though, and I had to hunt around for information that I feel should have been accessible from within the HIT creation interface itself.
  • If there are hard and fast guidelines that I should be following as a requester, I didn't see them. Even the best practices file was kind of hidden away in the walls of text that make up the requester interface.
  • The editing part of the HIT creation interface isn't simple to use, e.g., I wanted to add questions to my survey and didn't see a simple way to do it, and cutting and pasting in the non-HTML view of the editing screen didn't seem to work. I resorted to editing the HTML source for my survey HIT by hand, but not everyone will have that option. (Or are MTurk requesters all supposed to be coders?)
  • The process of funding my first HIT was frustrating and unclear. For one thing, it isn't obvious that requesters are required to buy prepaid HITs — to me, it seems like calling them prepaid implies that there is another option, but if there is one, I didn't see it. The notification that my first attempt at funding had failed didn't appear in the MTurk interface, but in my e-mail, so I didn't notice it right away. Information about the different funding methods was not available (that I could see) before I tried them, so I had to use trial and error to find out what would work, and how. Even now I'm not 100 percent sure Amazon isn't going to try to give me twice the amount of prepaid funding that I asked for.
  • When creating a task, the qualification settings are automatically set to allow only workers with the Masters qualification to do HITs, and this isn't obvious at all, so if I'm not paying attention, I'll end up unnecessarily restricting the pool of workers who have access to my tasks.
  • I can't use the MTurk system itself to hand-pick and invite workers who are qualified to do my work.
  • There's no built-in system for communicating with workers within the MTurk interface.
  • The purposes of different parts of the system (qualifications, batches, rejections) aren't immediately obvious to a beginning requester, and it's easy to make mistakes that will annoy or alienate workers.

Explore alternative crowd-labor markets

Perspective 1

I chose to check out oDesk as an alternate market.

Differences between oDesk and MTurk:

  • oDesk immediately seemed more appealing than MTurk, at a cursory glance for both those looking for work and those looking for workers. The site is designed much more thoughtfully, and it's more welcoming, which immediately gave me a more positive impression of it.
  • oDesk allows workers to create a profile to showcase their talents to people who are hiring, which is a pretty major advantage over MTurk for both workers and employers. However, I didn't really like that I couldn't (apparently) browse jobs without creating a profile.
  • oDesk requires a lot more information of workers before they can start working.
  • oDesk also requires that profiles be approved by its staff before they can go live, so access to work is more restricted than access to work on MTurk.
  • oDesk offers different membership plans (and indeed is selling the idea of "membership" where MTurk is not), one free and one paid, with different feature sets, while on MTurk, everyone has access to pretty much the same features.
  • If there's a way to browse oDesk by work category, or to browse a list of all the available work, I didn't see it. I had to use the search box to look for specific criteria, which I actually like less than on MTurk.
  • Once I got into the job listings, it was again obvious that the interface is much more user-friendly than MTurk's. There's more information on each job. I can filter jobs by many more criteria. I can save searches. Overall, the job search interface is a huge improvement over MTurk's.
  • It appears that employers can set more (and more useful) parameters on their job postings. For instance, they can require that workers have a certain level of fluency in a language or that they have certain skills.
  • You have to apply for jobs on oDesk, even if you're theoretically qualified for them, instead of just accepting work and doing it right away as is done on MTurk, so it seems as if oDesk is less of a work platform than MTurk, and more of a facilitator that matches employers to workers, who then use their own resources to perform and assess work.
  • There's more information available on employers right on oDesk, so there's no need for an independent, off-site rating system for employers.
  • oDesk appears to limit the amount of work that you can apply for (and maybe this is related to the membership level), whereas MTurk will allow you to take on as much work as you think you can do.
  • There's actually somewhat less information on oDesk about the jobs themselves, as it's up to employers to describe the job, while on MTurk you can preview the HIT or accept it to see exactly what you're getting into. You can also return HITs on MTurk without interacting with the employer and without any serious penalty, whereas on oDesk it seems likely that once you have a job, there's an expectation that you will complete it.
  • The types of work available on each site are different. MTurk is designed for short tasks to be completed by many people, and oDesk is designed for larger, more complex projects to be completed by fewer people.

Similarities between oDesk and MTurk:

  • Registering was easy on oDesk, but it was also fairly easy on MTurk.
  • On both sites, workers can search for work in their areas of interest and with their preferred scope and level of pay.

In sum, it seems like MTurk, despite its flaws, is better for getting work quickly, and for fitting work into small amounts of time, while oDesk has a format that feels more traditional (and slower), where you're encouraged to show off your skills, build a reputation, and commit to larger projects.



  • What do you like about the system / what are its strengths?
  • What do you think can be improved about the system?


  • What do you like about the system / what are its strengths?
  • What do you think can be improved about the system?

Flash Teams

  • What do you like about the system / what are its strengths?
  • What do you think can be improved about the system?