WinterMilestone 1 Dubs

From crowdresearch
Jump to: navigation, search

Click on the EDIT Button on the top-right of this page to edit.

Experience the life of a Worker on Mechanical Turk

Reflect on your experience as a worker on Mechanical Turk. What did you like? What did you dislike? If you're from outside the USA and unable to access MTurk, try try the worker sandbox, or other sites like CrowdFlower or Microworkers or Clickworker.

Experience the life of a Requester on Mechanical Turk


First of all, this is definitely a hard problem to solve. There are infinite possibilities of works that can be requested and it’s complicated to create a system that can not only serve all the possibilities, but also maintain a good traction with all of them. Usually, there is a trade-off with specificity and traction. How to make a good service for a diverse group of requesters. Here, we will try to consider some opinions we had about the requesting systems, with some comments that can lead to future implementations/suggestions.

For that part, we made a simple survey asking for the workers their opinion about what are the main problems about MTurk and

Good qualities/Positive aspects

  • Straight-forward publication. Except from the very beginning (login, etc), there are not a lot of doubts about how to use the system. It’s a very direct process, but a limited one.
  • There is a certificate system. Although there are a lot of ways to improve it, it’s interesting to avoid scammers/bad workers/bad intentions in the community.
  • Pre-organized models. There are some models to help you get ready to create something, so you don’t necessarily need to start from the scratch.
  • HTML edition. There are possibilities to edit the HTML and there is some bootstrap/css on it already.
  • It generates a .csv file that can later be used for other programs/analysis.
  • Cheaper than hiring a group of people to do the same. It’s pretty cheap to the requester to get work done and it’s a pretty random group of people, which is good for statistics.

Bad qualities/Negative aspects

  • Limited edition options. It’s possible to edit the HTML, but the way the structure is constructed makes it really hard to truly make a task different from the sample ones provided. An API or a system that could connect different systems/apis/etc with them in an easy/intuitive way would be a huge step forward to guarantee that all kind of tasks could be done with the application.
  • Old design. We believe that especially when we’re dealing with money, the design takes an important place: it inspires reliability. The way MTurk is currently designed doesn’t really bring any kind of confidence for someone that wants to invest money.
  • It’s restricted for people outside the US. We are Brazilians and, although we could use it since one of our members is in the US right now, it’s a really bad problem. Currency can be a big deal for a lot of workers. For instance, the Brazilian currency is around 1 to 4 right now. It means that a Brazilian worker would really benefit for a payment made in dollars from the same system and would be great for requesters (not too good for american workers though).
  • The data visualization is poor. Even though the results page is interesting, it could have a way better system to visualize data. D3.js could be really useful to truly bring meaning to the data acquired. Certainly, it would depend on the acquired data, but it would be a good possibility for a lot of parameters or sample batches.
  • Payment systems could be easier to navigate. For instance, the MTurk fees are not explained, so it’s not possible to know the exact total amount of money needed before finishing the batch. It would be really useful to know it before. Also, sometimes you don’t have enough money to make a batch public on the MTurk account and it takes too long to put money again. Maybe, we could have an easier way to refill the account.
  • There are no tag system or any way to provide a specific information/ranking about your performance for specific tasks. Sometimes, someone is really good at filling surveys, but not that good at processing signals. It would be interesting to see that function.

Explore alternative crowd-labor markets


The idea of behind TaskRabbit, according to themselves is "outsourcing household errands and skilled tasks to trusted people in your community.". Basically, it is an online and mobile marketplace that allows each neighborhood to outsource small tasks.

The biggest factor differentiating MTurk from TaskRabbit is the key role played by computer automation on the former. While Mechanical Turk heavily relies on artificial intelligence to deliver digital tasks such as image recognition and OCR, TaskRabbit has been facing trouble getting traction from human-only based tasks. While the background idea of outsourcing mechanical flows of your day-to-day routine is basically the same in both of them, MTurk has the intricate advantage of not only being a provider of digital services but also the infrastructure provided by its main company, Amazon. All in all, for what we're able to sense from both products as of today, it seems that getting people to outsource their mundane activities rather than their digital ones seems to be a huge yet not impossible challenge faced by any initiative aiming at getting people to work together using collective intelligence.



  • What do you like about the system / what are its strengths?
  • What do you think can be improved about the system?


  • What do you like about the system / what are its strengths?
  • What do you think can be improved about the system?

Flash Teams

  • What do you like about the system / what are its strengths?
  • What do you think can be improved about the system?

Milestone Contributors

Our contributors:

  • Flavio Scorpione - @scorpione
  • Gabriel Bayomi - @gbayomi
  • Henrique Orefice - @horefice
  • Lucas Bamidele - @lucasbamidele
  • Teogenes Moura - @teomoura