Milestone 2 munichkindl

From crowdresearch
Revision as of 14:41, 11 March 2015 by Davidlechnquiones (Talk | contribs) (Requester Needs)

Jump to: navigation, search

Attend a Panel to Hear from Workers and Requesters

Panel 1

  • To new requesters, the options available when setting up an HIT are sometimes confusing. For instance, one requester thought the maximum time for an HIT was to be an estimate shown to the workers instead of a system-side maximum completion time
  • Some requesters say the ideal time for a survey on mTurk to last is between 8 and 10 minutes in order to get valuable data.
  • Furthermore, requesters cannot be certain that their uploaded tasks are completed (instead of not being taken at all or workers stopping halfway through)
  • The system makes does not make it possible to pre-select workers who match certain demographics or have a specific type of qualification which are verified.
  • Workers value the flexibility to work from any place, at any given time
  • Workers do not like that there is inconsistency in whether they will get enough tasks to work on
  • There are people working on Turk full time as well as part-time in addition to a 'real life' job
  • A group of Turkers created the 'Dynamo agreement' ( aiming to make requesters provide some kind of minimum wage and fair working conditions

Panel 2

  • The rejection rate is broken in Mechanical Turk. Requesters usually don't reject work even if is not useful, so when you look at worker's rejection rate you know it's not true.
  • In Mechanical Turk is always more useful to break down tasks in smaller tasks to get better results.
  • It's useful to first test the worker with a task which you have its ground truth to know if you can trust the rest of the results.
  • In Mechanical Turk, it would be useful to be able to choose a concrete worker for a concrete task.
  • It's hard to make personalized tasks in Mechanical Turk, such as Twitter suggestion algorithm tests.
  • The successful workers find jobs during unconventional hours.
  • For some research tasks would be useful to access to concrete demographic groups.
  • Mechanical Turk should have a better way to proof the worker's identitiy, such as where is the worker living, age, and so on. This is crutial for some social research.

Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Worker perspective: Turkopticon

The authors analyse the croudsourcing platform Amazon Mechanical Turk as a site of technically mediated worker-employer relations. They present the development and the final version of Turkopticon, a system that allows workers to publicize and evaluate their relationships with employers. Finally, they discuss the potentials and challenges of sustaining activist technologies that intervene in big socio-technical systems.

The main reason for the development of Turkopticon, is that the authors see an inequality between the right of the requesters and the workers, which is also caused by Amazon’s interests.

In the paper it is stated that the workers, also because they come from different countries, have different views on how well-payed and valued is their work (what is a lot of money in India is not enough to survive in the US!). Some of the workers just engage for fun in AMT and others are in need of the salary. The salary of the workers is below average, furthermore they give up every right about their intellectual property. Additionally, every work can be rejected from the requesters without the need of any reasons and the workers have limited options of dissent within AMT itself. Payment can take as long as 30 days. If the workers consider the system unfair, they have very little options. Before the forums about AMT became important and Turkopticon has been developed, there was not much communication between the workers possible, which was even worse because they come from different cultures and speak different languages.

On the other hand, it is easy for requesters to not see the workers as humans and instead as some sort of APIs. They can reject every work without having to justify it which includes also that there is no need of payment. If the workers want to complain, they don’t even have to answer.

Turkopticon aims to close the communicational gap between the workers, as well as between workers and requesters.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

Although this paper strongly focuses on possible advantages of and methods for standardizing crowdsourcing platforms, there is certain information about current behaviors of requesters and workers to be extracted.

Workers, have only very limited possibilities to filter tasks for desirable criteria such as skill requirements and difficulty or payment structure. In addition, they need to manually check each requester’s reputation beforehand in order to avoid possible fraud, as well as the level of quality expected by the requester. Due to non-standardized task user interfaces, they need to adjust to each one separately. The majority of workers works on low-skilled tasks and are said to perform better with very detailed instruction. At some platforms (such as they receive online training before being allowed to accept a certain type of task but this is not a common standard.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

On the requester side, much redundancy happens at the creation of tasks. Not having any or only a small number of ‘best practices’ to refer to, everyone defines their own user interface without yet knowing their effectiveness for good results. For comparison, in empirical research, there is a number of standardized interview questionnaires to choose from; such a thing for is not yet established for microlabor tasks. Furthermore, categorization of tasks is only possible on a very broad level. Considering task pricing, requesters do not have to possibility to adjust prices according to market demand without having to repost the task.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

The author of this post from 2010 proposes improvements on AMT based on the following observations:

It is difficult for the requesters to access the quality of the workers beforehand. Work can be rejected without justification what has bad effects on the workers reputation. However, if the platform is not user-friendly, good workers are tented to leave AMT, reducing the overall quality of the platform.

The requesters have the problems of starting to become popular on AMT, because workers are careful accepting tasks from new requesters, because they could easily be not paid or rejected. Also, it is difficult for them to manage the complex API or the execution time. The author claims that many requesters hire full-time developers to produce the microtasks. However, by doing so, the work gets less cost-effective.

The author states that without changes fitting the needs, AMT will not be successful for much more time.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

List out the observations you made while doing your fieldwork. Links to examples (posts / threads) would be extremely helpful.

  • This comment talks about one of the points that we got in the Panel 2. The platform needs a way to verify the identity of the worker, not only to trust them, but also to save time as is something that is asked really often in the different tasks (workers need to spend a lot of time on it).
  • Trust is a real issue for requesters, as we could see in the panels and Reddit. See thread

Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

  • Workers need to show their personal information because they are adding it in different tasks quite often. If there is a profile and the task can take that information automatically the worker will save time and they will be able to focus in the real tasks.
  • Workers need to own more rights about their intellectual property. Right now, requesters can use their work even in case of rejection.
  • Workers need to connect more with each other. For now, they can only get information about requesters or help each other on external platforms.
  • Workers need to be connected to the requesters. Until now they can only contact them via email and no answer is enforced.

Requester Needs

  • Requester needs to trust the identity and qualification of the worker. Right now Mechanical Turk doesn’t give the requester any way to confirm the information about the worker (such as where they live, age, gender and so on).
  • Requesters need to select concrete demographical groups for some tasks. If they want to do it now, they need to do some type of pre-survey. (It would be helpful to be able to select exactly what kind of worker they need specifying exact information and segment them like you can do in systems like Google Adwords.)
  • Requesters need to be cost-effective. Now sometimes they even need to hire full-time programmers to create the tasks for AMT.
  • Requesters need the ability to modify tasks. Now they have to create a new task, even if they only want to change one parameter.
  • Requesters need to be good communicators they should be able to communicate what their task is and if necessary justify why their payment is sufficient for the task.