Difference between revisions of "Milestone 2 munichkindl"

From crowdresearch
Jump to: navigation, search
(Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc)
(Worker perspective: Turkopticon)
Line 24: Line 24:
 
=== Worker perspective: Turkopticon ===
 
=== Worker perspective: Turkopticon ===
  
1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.
+
The authors analyse the croudsourcing platform Amazon Mechanical Turk as a site of technically mediated worker-employer relations. They present the development and the final version of Turkopticon, a system that allows workers to publicize and evaluate their relationships with employers. Finally, they discuss the potentials and challenges of sustaining activist technologies that intervene in big socio-technical systems.
  
2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.
+
The main reason for the development of Turkopticon, is that the authors see an inequality between the right of the requesters and the workers, which is also caused by Amazon’s interests.
 +
 
 +
In the paper it is stated that the workers, also because they come from different countries, have different views on how well-payed and valued is their work (what is a lot of money in India is not enough to survive in the US!). Some of the workers just engage for fun in AMT and others are in need of the salary. The salary of the workers is below average, furthermore they give up every right about their intellectual property. Additionally, every work can be rejected from the requesters without the need of any reasons and the workers have limited options of dissent within AMT itself. Payment can take as long as 30 days. If the workers consider the system unfair, they have very little options. Before the forums about AMT became important and Turkopticon has been developed, there was not much communication between the workers possible, which was even worse because they come from different cultures and speak different languages.
 +
 
 +
On the other hand, it is easy for requesters to not see the workers as humans and instead as some sort of APIs. They can reject every work without having to justify it which includes also that there is no need of payment. If the workers want to complain, they don’t even have to answer.
 +
 
 +
Turkopticon aims to close the communicational gap between the workers, as well as between workers and requesters.
  
 
=== Requester perspective: Crowdsourcing User Studies with Mechanical Turk ===
 
=== Requester perspective: Crowdsourcing User Studies with Mechanical Turk ===

Revision as of 12:25, 11 March 2015

Attend a Panel to Hear from Workers and Requesters

Panel 1

Panel 2

  • The rejection rate is broken in Mechanical Turk. Requesters usually don't reject work even if is not useful, so when you look at worker's rejection rate you know it's not true.
  • In Mechanical Turk is always more useful to break down tasks in smaller tasks to get better results.
  • It's useful to first test the worker with a task which you have its ground truth to know if you can trust the rest of the results.
  • In Mechanical Turk, it would be useful to be able to choose a concrete worker for a concrete task.
  • It's hard to make personalized tasks in Mechanical Turk, such as Twitter suggestion algorithm tests.
  • The successful workers find jobs during unconventional hours.
  • For some research tasks would be useful to access to concrete demographic groups.
  • Mechanical Turk should have a better way to proof the worker's identitiy, such as where is the worker living, age, and so on. This is crutial for some social research.


Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Worker perspective: Turkopticon

The authors analyse the croudsourcing platform Amazon Mechanical Turk as a site of technically mediated worker-employer relations. They present the development and the final version of Turkopticon, a system that allows workers to publicize and evaluate their relationships with employers. Finally, they discuss the potentials and challenges of sustaining activist technologies that intervene in big socio-technical systems.

The main reason for the development of Turkopticon, is that the authors see an inequality between the right of the requesters and the workers, which is also caused by Amazon’s interests.

In the paper it is stated that the workers, also because they come from different countries, have different views on how well-payed and valued is their work (what is a lot of money in India is not enough to survive in the US!). Some of the workers just engage for fun in AMT and others are in need of the salary. The salary of the workers is below average, furthermore they give up every right about their intellectual property. Additionally, every work can be rejected from the requesters without the need of any reasons and the workers have limited options of dissent within AMT itself. Payment can take as long as 30 days. If the workers consider the system unfair, they have very little options. Before the forums about AMT became important and Turkopticon has been developed, there was not much communication between the workers possible, which was even worse because they come from different cultures and speak different languages.

On the other hand, it is easy for requesters to not see the workers as humans and instead as some sort of APIs. They can reject every work without having to justify it which includes also that there is no need of payment. If the workers want to complain, they don’t even have to answer.

Turkopticon aims to close the communicational gap between the workers, as well as between workers and requesters.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

List out the observations you made while doing your fieldwork. Links to examples (posts / threads) would be extremely helpful.

  • This comment talks about one of the points that we got in the Panel 2. The platform needs a way to verify the identity of the worker, not only to trust them, but also to save time as is something that is asked really often in the different tasks (workers need to spend a lot of time on it).
  • Trust is a real issue for requesters, as we could see in the panels and Reddit. See thread

Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

A set of bullet points summarizing the needs of workers.

  • Example: Workers need to be respected by their employers. Evidence: Sanjay said in the worker panel that he wrote an angry email to a requester who mass-rejected his work. Interpretation: this wasn't actually about the money; it was about the disregard for Sanjay's work ethic.

Requester Needs

A set of bullet points summarizing the needs of requesters.

  • Example: requesters need to trust the results they get from workers. Evidence: In this thread on Reddit (linked), a requester is struggling to know which results to use and which ones to reject or re-post for more data. Interpretation: it's actually quite difficult for requesters to know whether 1) a worker tried hard but the question was unclear or very difficult or an edge case, or 2) a worker wasn't really putting in a best effort.