WinterMilestone 1 Dubs

From crowdresearch
Jump to: navigation, search

We are the Brazilian team: Dubs! This is our first milestone!

Experience the life of a Worker on Mechanical Turk


The most important thing here is to understand that the worker is a fundamental problem in the system. It is even hard to make positive aspects about it because the payment are very low and it's not easy to find work.

As a requester, we decided to make a task to ask workers the main problem with the platform and payment was the most frequent issue followed by finding work.

On the other hand, people from other countries might benefit from the dollar currency of the system. Although it would still be a low payment, it would certainly be a way better deal to them than to americans/strong currency countries. Unfortunately, the platform is not available around the world.

Positive aspects

  • UI is very simple and the most important tools are found easily.
  • You can find some decent tasks (extra money).
  • (If the platform was available around the world, it would be an interesting way to make money).

Negative aspects

  • The payment is really low. There is no way to count on that as the primary source of money on the US at least. There are a lot of better options to earn money out there.
  • Tasks need to be done precisely (concentration and speed needed). If you can't do it correctly, you just won't get the money.
  • It's not easy to find work and the certificate system sometimes just make it harder instead of really helping people to get qualified work.
  • Workers are very be vulnerable to untrustworthy requesters. There is no feedback for the worker and a lot of scam activities can happen.
  • UI is simple, but it is not welcoming and it's also very outdated. It gives the impression that the platform was abandoned, or is harder to use than it actually is.

Experience the life of a Requester on Mechanical Turk


First of all, this is definitely a hard problem to solve. There are infinite possibilities of works that can be requested and it’s complicated to create a system that can not only serve all the possibilities, but also maintain a good traction with all of them. Usually, there is a trade-off with specificity and traction. How to make a good service for a diverse group of requesters. Here, we will try to consider some opinions we had about the requesting systems, with some comments that can lead to future implementations/suggestions.

For that part, we made a simple survey asking for the workers their opinion about what are the main problems about MTurk and

Positive aspects

  • Straight-forward publication. Except from the very beginning (login, etc), there are not a lot of doubts about how to use the system. It’s a very direct process, but a limited one.
  • There is a certificate system. Although there are a lot of ways to improve it, it’s interesting to avoid scammers/bad workers/bad intentions in the community.
  • Pre-organized models. There are some models to help you get ready to create something, so you don’t necessarily need to start from the scratch.
  • HTML edition. There are possibilities to edit the HTML and there is some bootstrap/css on it already.
  • It generates a .csv file that can later be used for other programs/analysis.
  • Cheaper than hiring a group of people to do the same. It’s pretty cheap to the requester to get work done and it’s a pretty random group of people, which is good for statistics.

Negative aspects

  • Limited edition options. It’s possible to edit the HTML, but the way the structure is constructed makes it really hard to truly make a task different from the sample ones provided. An API or a system that could connect different systems/apis/etc with them in an easy/intuitive way would be a huge step forward to guarantee that all kind of tasks could be done with the application.
  • Old design. We believe that especially when we’re dealing with money, the design takes an important place: it inspires reliability. The way MTurk is currently designed doesn’t really bring any kind of confidence for someone that wants to invest money.
  • It’s restricted for people outside the US. We are Brazilians and, although we could use it since one of our members is in the US right now, it’s a really bad problem. Currency can be a big deal for a lot of workers. For instance, the Brazilian currency is around 1 to 4 right now. It means that a Brazilian worker would really benefit for a payment made in dollars from the same system and would be great for requesters (not too good for american workers though).
  • The data visualization is poor. Even though the results page is interesting, it could have a way better system to visualize data. D3.js could be really useful to truly bring meaning to the data acquired. Certainly, it would depend on the acquired data, but it would be a good possibility for a lot of parameters or sample batches.
  • Payment systems could be easier to navigate. For instance, the MTurk fees are not explained, so it’s not possible to know the exact total amount of money needed before finishing the batch. It would be really useful to know it before. Also, sometimes you don’t have enough money to make a batch public on the MTurk account and it takes too long to put money again. Maybe, we could have an easier way to refill the account.
  • There are no tag system or any way to provide a specific information/ranking about your performance for specific tasks. Sometimes, someone is really good at filling surveys, but not that good at processing signals. It would be interesting to see that function.

Here is our csv file.

Explore alternative crowd-labor markets


The idea of behind TaskRabbit, according to themselves is "outsourcing household errands and skilled tasks to trusted people in your community.". Basically, it is an online and mobile marketplace that allows each neighborhood to outsource small tasks.

The biggest factor differentiating MTurk from TaskRabbit is the key role played by computer automation on the former. While Mechanical Turk heavily relies on artificial intelligence to deliver digital tasks such as image recognition and OCR, TaskRabbit has been facing trouble getting traction from human-only based tasks. While the background idea of outsourcing mechanical flows of your day-to-day routine is basically the same in both of them, MTurk has the intricate advantage of not only being a provider of digital services but also the infrastructure provided by its main company, Amazon. All in all, for what we're able to sense from both products as of today, it seems that getting people to outsource their mundane activities rather than their digital ones seems to be a huge yet not impossible challenge faced by any initiative aiming at getting people to work together using collective intelligence.



  • What do you like about the system / what are its strengths?

The platform has very good reach, easy to use in many platforms, finely adapted to the target audience and their available resources. The app itself has stuck to the basics, which is what makes it possible to reach such a great number of workers.

  • What do you think can be improved about the system?

The system does not give any insight to the user about what he is doing, which does not clarify to the user (worker) what is the purpose of the work being done (for example, in the situation described in the paper, it could have some way to explain to the user why he/she is typing out several aparently random words).


  • What do you like about the system / what are its strengths?

The way that iterative task creating yields way more relevant results is quite impressive. It is very important the way Boomerang creates a meaningful (not inflated) rating system for workers and requesters, making it so more requests get more compatible and more apropriate workers on them. So far, the platform looks and interacts with the user in a much friendlier way than many other platforms of the same genre.

  • What do you think can be improved about the system?

Could it be possible to make similar requesters to those the worker has rated highly, not only those that the worker has already rated appear towards the top of the task feed? That would make it easier for the worker to find new requesters that provide tasks similar than the ones the worker has liked (possibly with tags of some sort). Could it be that research projects that need a wide variety of people in its audience, get requested by a requester that has already rated several workers, and by doing so, biased its results by favoring a selected group of high rated workers, that could have been highly rated because of a specific profile? Would it be apropriate to have a way of bypassing the selective task feed ranking, to get a more random group of people for research requests? Maybe allowing users to bypass Boomerang. In boomerang, there could be some way of giving gidance to the users as to what is a highly ratable user by giving them criteria to be guided by. Also, why not create a way to evaluate if the user is rating carelessly (only rating highly, or poorly), by making the system recognize that the user is giving outlying evaluations. So if requester X has been given 99% good evalutions on a particular request, and worker Y gives a negative evaluation, could the system see that as an outlier and act accordingly?

Flash Teams

  • What do you like about the system / what are its strengths?

The way that Flash Teams aims to enable real-world, complex tasks, to tackle a part of the market that has not yet been addressed fully by the crowdsourcing communities. It is great that this way of working can get a wider audience of people with ideas and maybe not so many resources to get works done quickly and effectivelly. A great strength of Flash Teams is its easily editable blocks and the way they interact to create unique and adaptable work forces.

  • What do you think can be improved about the system?

There is no way, in the current model of the platform, for the expert workers to get time for more in-depth analysis of the job they are given. This way a piece of work might not get the time it needs and deserves to reach its full potential. Some kinds of projects can fail to have a needed diagnostic by an expert, due to the current model of work. By having modular teams and fixed project methodologies, it could make the projects, and consequently the results, a bit stiff and shallow. When assemblying Flash Teams and creating the workflow on Foundry, unexperienced users could make decisions that can reduce the quality of an end product, by thinking about reducing workforce to reduce costs and situations alike.

Milestone Contributors

Our team:

  • Flavio Scorpione - @scorpione
  • Gabriel Bayomi - @gbayomi
  • Henrique Orefice - @horefice
  • Lucas Bamidele - @lucasbamidele
  • Teogenes Moura - @teomoura