WinterMilestone 2 Dubs

From crowdresearch
Jump to: navigation, search

Our submission for Winter Milestone 2. We are the Brazilian team: Dubs!

Attend a Panel to Hear from Workers and Requesters


It was a very interesting pane. It was a great opportunity to understand better the issues discussed in the previous milestone and, also, understand deeply how they affect the life of both requesters and workers. However, three very interesting topics were discussed and we decided to talk about them since we believe they have a different perspective from before.

Variable Earning

Although the low payment is a known issue, it was kind of a surprise the very high variation from different daily payments. It shows that it's truly hard to rely on MTurk for a regular annual payment for example. The variation indicates that there are structural problems with the offer/demand of work and it seems to have a deeper impact on workers than on requesters.

Rejection Perspective

At the beginning of the Panel I had a different impression about rejection and its impacts. The main problem is the way the whole system is structured. It was certainly not meant to be the main income process of the workers. However, as we could see, most of the workers use the system as the primary source of income. It means that the rejected work has a huge impact on the human behind the work requested. Not only that, workers have to guarantee that they will be able to maintain a high percentage of approved work to be able to work again in MTurk. Therefore, rejecting a work is a more complicated situation than it appears to be at the first place.

Requesters estimate of time

Another different perspective we could get from the Panel is related to the estimation of time from the requesters. It has a huge impact on the workers' decision to accept the job, but it's a very complicated future for the requester. It's very easy to understimate or overstimate the capacity of the workers and in both situations it has a considerable impact on both workers and requesters and certainly affect the relationship between them.

Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit. 2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

It was one of the most interesting articles from the suggested readings. It shows a very direct perspective, especially from the workers' point of view. The most important thing to consider about the whole system of MTurk and, probably, about the whole crowd research experience is: workers use the system as a primary source of income. The statistics reinforce this fact, according to the article: " 80% of the tasks are carried out by the 20% most active (3,011–8,582) Turkers". It shows that the vast majority of the active workers probably spend a significant amount of hours in MTurk and use it as an important souce of money. This fact is also reinforced by the different Turkers forums.

MTurk goals for Requesters/Workers

It brings us to complicated problems related to workers and to requesters.

The main problem is that there is a gap between goals of Requesters and Workers and this issue needs to be addressed by both parts. The main objective for workers is to use MTurk as the primary source or at least important source of income. It means they will spend a considerable part of their time using the system. The advantages are:

You can decide your own hours. You can decide your time investment. You can decide the works you will accept or not.

Now, we have the advantages and goals of the workers: a good job opportunity that can provide a good amount of daily income.

We should analyze the same situation for the requesters. What brings most of the requesters to MTurk is a human-as-service product. It means that it allows requesters to have work that wouldn't be easily done by computers to be done by humans. It's the concept of humans-as-service. At the same time, the cost efficiency is a very important issue: they need the task to be cheap enough in order to be a good investment of money. Therefore, we can also design the advantages and goals: good work quality for a task (that we assume that wouldn't be possible with a computer) for a low cost.

If we analyze demand/offer, it should be simple. There are a good amount of requesters willing to pay workers to do it and there are a lot of workers willing to invest some time to make some money by completing the tasks. However, it's not necessarily a direct relationship. The system itself, which is one of the main points of discussion for this entire research process, is the main provider of the relationship between requesters/workers. It has the power to change the balance between both of them, modifying the work quality, the workers' income and all the system infrastructure.

Although there are problems on both sides, we could currently conclude that MTurk is very concerning for the workers right now. The design certainly doesn't help their main goal: primary source of income. This might suggest that it's better for requesters than for workers, however this is not necessarily true. Although there are certainly some advantages in comparison to the workers' side, the whole problem directly decreases the work quality for the requesters. If the workers doesn't have the incentives and conditions to provide their best, it affects the whole environment. The health of the system depends on the fact that both workers and requesters can provide their best to guarantee that all the resources are correctly used.

Worker perspective: Turkopticon

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

Turkopticon proposes a way to guarantee that the workers are will not remain invisible. What does that mean? It means that the system provides a meaningful way of evaluating requesters. It's a powerful tool that workers can have to avoid scamming and unfair rejection of work. It gives a balance for the working environment, seen that requesters already have a lot of knowledge about the workers, but the opposite is certainly not true. It works for the benefit and health of the whole system.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Although Turkopticon consists of an evaluating method for requesters, it's certainly a benefit for them as well. Building a healthy relationship between requesters and workers is certainly the best way to guarantee a better quality of and, also, providing the path to more advanced and complicated tasks.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

All workers wants to make the most money in the least amount of time, and sometimes the quality of a given task could be compromised by that. It might be because of bad faith(such as not doing a work properly just to clear it quickly) or a poorly written request(in which an evaluation is completely subjective, for example), but either way, that is bad for both requester and worker. The paper show us experiments that proves that there are ways to avoid that. The experiment results suggests that there are ways to improve workers efficiency, by making the task simple enough to be done by one person without major issues and guiding the worker through the objectives expected. That makes it harder for bad intentioned workers to submit incomplete or poorly executed tasks and provide guidance to workers who could potentially provide good results.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

The experiment results suggests that a well thought request, one which provides guidance and makes “gaming” less desirable, can achieve better results. A requester must be able to express clearly what he expects from a task. They should avoid: Instructions that leads to misinterpretation, tasks that can be done subjectively(when the expected result should not depend on a single person’s judgment), and tasks that can be “gamed” easily.

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Having in mind the issues brought to discussion by the author, it is quite clear for the reader that the Mturk marketplace while performing its main function (to connect workers and requesters) properly fails to fullfill it's potential entirely. That happens for reasons that affects both workers and requesters, as it is clear that they both have issues when it comes to working with Mturk and that allow competitors to catch up or provide better solutions than that of Amazon's service. For requesters, the issues are mainly focused on the worker's reputation system. Trust is the most important asset on a marketplace between the participants, and certainly is the one factor that can determine the rise or fall of any marketplace-based application, and it is no different in the MTurk case. Several problems are stated when it comes to arguing on how to build a better reputation system for Mturk, as not allowing requesters to see the history of activities from a certain worker or allowing rating the workers. As for the other side, the workers also need changes so that they can have a better experience while using MTurk, such as the very fundamental guarantee that valid work won't be discarded as not good and that the person hiring them will actually pay in a reasonable timeframe. So, it is quite clear that while MTurk is still one of the big players in the market it will have to update it's marketplace infrastructure rapidly if it doesn't want to lose its place as a major platform for workers and requesters.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

It is clear that writing is a task that requires full attention and one that can trick even experts into making subtle mistakes. Hence, it is a very appropriate one to be addressed using crowdsourcing. Workers in this setting are people fluent in english who can aid a writer by reviewing their writing and suggesting/correcting their work so that texts are clear of mistakes when deliverd to their final purpose. There are good and bad sides when it comes to hiring workers to aid writing tasks. As an example, it is quite easy to imagine that no single person has 100% efficiency in any task, especially one that requires both real time attention and previous knowledge such as writing. So, it is a possible scenario that a worker may add errors to a sentence that already contain errors. Not only that, but they also may lead the writer in the wrong direction when it comes to syntax and to the flow of the text itself. So, the authors propose that a requester gets service not only from one turk worker, but from various and filter out bad contributions.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requesters in this type of task are usually well educated people looking forward to deliver perfectly formatted documents with various purposes. So, it is to their full advantage to have someone else equally skilled with verbal and grammar skills to proof read their texts before handing them out.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

Even though the reading material and videos gave us a solid understanding of the processes being handled in this task, we set ourselves with the mission of verifying how this affected the real world experience by requesters and workers. Therefore we went to reddit and explored how people actually interact via the MTurk platform and how their experiences could help us reshape our beliefs towards the user experience design behind the website, which we will list as follows:

In this example, we clearly see how the user experience could be redesigned for the better: an unexperienced requester makes an otherwise easy-to-fix mistake, but as the user interface of the MTurk application is not optimal, it leads to a very negative experience.

Here we see another rejection due to poorly designed workflows inside the marketplace: The requester made a mistake and it severely affects the worker's reputation even though he is not to blame. Not only that, but it is also complicated for the worker to request a review of his/her work so that he can get his status back.

In this topic, it becomes clear what he had seen before: The MTurk experience may be negative for beginning workers. In the example, the worker got rejected on his first day because of a misunderstanding of the instructions given by the requester, which could be because the marketplace does not enforce them to be precisely clear.

Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

Here we summarize the most important topics from our perception:

  • Workers need to have a minimum hourly rate for activities. It doesn't necessarily have to a high value, but as workers are using the platform as a primary source of income or at least a relevant one, it's important to make sure that the earnings won't have a huge fluctuation Although it's a discussion points, we believe that this would give a better security for workers and, eventually, increase their work quality and the whole environment.
  • Workers need to be able to provide feedback about the requesters and also to evaluate them. The method must be different from the workers' method of evaluation to guarantee the efficiency of the system.
  • Workers need to know why they failed to complete a task. The frustration about the failure sometimes is more related to work ethic than to actual money.
  • Workers need to have Amazon support. The platform needs to ensure some basic infrastructure for workers. Every economic system needs a "judicial system" and the necessary regulation, it's important for the financial health of the environment.

Requester Needs

  • Requesters need to be protected against fraud/bots/scams. There are a lot of possible sources for bad quality work. This is generally created by the fact that the platform doesn't provide a lot of support against it and the payments are really low, creating incentives for fraudulent systems to execute easy tasks in order to increase the total amount earned.
  • Requesters need access to the work quality of the "hired" workers. Although there are already some methodologies around it, it's necessary to improve the capacity of a worker to display knowledge about certain kind of work in order to improve the resource allocation of the platform.

Milestone Contributors

Our team:

  • Gabriel Bayomi - @gbayomi
  • Flavio Scorpione - @scorpione
  • Henrique Orefice - @horefice
  • Lucas Bamidele - @lucasbamidele
  • Teogenes Moura - @teomoura