WinterMilestone 2 Intunity

From crowdresearch
Jump to: navigation, search

Attend a Panel to Hear from Workers and Requesters

Deliverable

  • Tuned into the professor's talk about how communication is a big deal. There needs to be a system in place that improves

the communication. To so do, there needs to be a way to have a requester know that the worker acknowledges/understands what's been said.

  • Workers don't have set schedules. Generally they follow the HITS. For instance, if a HIT notification alerts that's high in value

while the worker is doing daily activities (eating, etc.) then he or she generally will pause and focus on the HIT. There needs to be a system in place that allows workers to work on their own schedule while obtaining close to max benefits.

  • Workers generally see requesters as demanding and impossible to predict. This again all stems from the poor communication systems.
  • What's funny is that requesters describe as either negatively (troublesome, etc.) or in positive terms. The requesters have a more diverse

vocabulary when talking about workers. Not so much for workers. This again stems from the power dynamics.

  • Since workers desire to have as high an approval rating as possible, they make sure to get an idea of how fair/nice a requester is

before working for him or her. A worker's approval rating must always be high in order to succeed in the crowdsourcing world.

Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit. Observations we noticed about workers from the readings are that they generally don't cheat on jobs, many of them usually aren't high paid professionals (they usually work part time / don't have a full job), and that people new to Amazon Turk focus on boosting their scores (HIT count) as much as possible.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit. Observations we noticed about requesters from the readings are that they are generally have the power in disputes. It appears that if a problem occurs between a requester and a worker, the requester usually wins regardless of the situation. They usually believe that workers are enjoying their work (i.e. they think that it's a dream job, enjoyable job). Also what's interesting is that requesters get information about workers such as their approval rating (a main reason why new workers try to increase it asap).

Worker perspective: Turkopticon

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit. From the readings, we noticed a few potentially unfair things about Turkopticon about the power dynamics between workers and requesters. For instance, Turkers can easily be kicked off / removed from the site if they have low approval ratings, while there isn't any system in place to analyze requester's actions. What if a requester abuses the system? Also, if a requester does not like the job a Worker has done, then the requester can EASILY deny payment. That just furthur explains the power dynamics. Perhaps there are better solutions to this. Also, there seems to be some invisible culture that workers follow. This could be very problematic. There have been many events in history where culture (Alcoe for example) have caused poor keystone habits. These habits however end up being so strong that they ripple throughout the entire eco-system, generating many negative effects.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit. From the readings we noticed that it's difficult to improve as a worker. There isn't much of a feedback loop to let know workers know what they're doing incorrectly. Yes, there are approval ratings etc., but a score without explanation is almost useless if someone wants to improve. Also, since there isn't a "person" behind each worker in the eyes of the requester, they almost all seem the same. It's very difficult to treat someone fairly/with respect if you do not see the individual. I think this alone is one major key in a lot of problems with crowdsourcing.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit. Observations about workers that we drew from the readings were that not everyone tried to game the system. It seems to be a small minority of individuals. However, it seemed very possible to cheat the system by providing auto responses to similar studies. A way to avoid this is to simply reframe the questions in a way that makes it much more difficult to blurt out. For instance, require that questions need an open ended answer, rather than a logical yes or no answer.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit. Observations we drew from the readings were that requesters were generally limited in controlling the environment of workers. Also, requesters looking through responses one by one deciding which ones must be rejected. Also, in order to get great responses, requesters need open ended questions. Simply providng yes/no or other types of simple questions generally will result in simple responses. And simple responses are usually the ones that required little to no thought (easy to spam).

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit. Observations about workers that we drew from the readings were that they generally don't have much standing/power in the market. They are easily replaceable if the tasks they're doing have a large number of workers. Also, workers generally needed to spend a lot of time learning how to adapt to the guidelines of different employers.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit. Observations about requesters that we drew from the readings were that they don't have much reason to tell workers how to do a better job or what they're doing wrong. They can just find a new worker. Also they could not guarantee that workers would successfully complete jobs despite being trained to do them. Also, instructions must be given clearly and explicitly since workers can habitually do a slightly different job than you asked for.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

Workers need a guaranteed trustworthiness from the requesters. For instance, requesters should be able to pay on time

Workers also need a better interface. Workers cannot search for a requester easily. In addition, workers have a difficult time searching for available tasks of interest. Some proposed solutions for this are having task categories, better search engine, and a recommendation system to propose tasks for workers.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

There needs a better interface to post tasks.

Requesters need to know the reputation profiles of the workers. Not having this knowledge makes the requesters having a harder time to differentiate good workers from bad workers. This leads to good workers being paid the same amount as bad workers, and this results in good workers leaving the market. Such improvements to make the requester have a better understanding of the worker are: qualification tests, ability to track working history, etc.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

List out the observations you made while doing your fieldwork. Links to examples (posts / threads) would be extremely helpful.

Requesters need a way to handle abusive workers. These workers are getting paid for not doing the task correctly. https://rh.reddit.com/r/mturk/comments/2bc3cp/how_to_handle_abusive_workers/

Some workers may not complete the task on time, and the requester would like to compensate them for their efforts. https://rh.reddit.com/r/mturk/comments/2e0y6l/2_of_my_participants_ran_out_of_time_id_like_to/

Some requesters don't know how to compensate the workers. https://rh.reddit.com/r/mturk/comments/3uvpff/how_much_to_pay/

Some requesters want to give bonuses so that it will incentivize workers to actually complete the task correctly and prevent scams. Also, these requesters do not want to reject these workers because rejection rates may mean a lot to certain workers. https://rh.reddit.com/r/mturk/comments/3cler4/requestor_how_do_i_give_bonuses/


Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

A set of bullet points summarizing the needs of workers.

  • Workers need to be respected by their employers. Evidence: Sanjay said in the worker panel that he wrote an angry email to a requester who mass-rejected his work. Interpretation: this wasn't actually about the money; it was about the disregard for Sanjay's work ethic.
  • Workers may not complete task on time but the requester would like to compensate for their efforts. Evidence: The reddit post where

participants ran out of time. Interpretation: The system is very rigid like and doesn't allow for payments behind the scene. Also, it appears that this wasn't even a consideration in MTurk. Either they're not doing enough research on their customers or they don't care.

Requester Needs

A set of bullet points summarizing the needs of requesters.

  • Requesters need to trust the results they get from workers. Evidence: In this thread on Reddit (linked), a requester is struggling to know which results to use and which ones to reject or re-post for more data. Interpretation: it's actually quite difficult for requesters to know whether 1) a worker tried hard but the question was unclear or very difficult or an edge case, or 2) a worker wasn't really putting in a best effort.
  • Requests need to be able to handle abusive workers. Evidence: Workers are getting paid for not doing the task correctly on the reddit post.

Interpretation: There seems to be a lack of system management between requesters and workers.

  • Price for some tasks appear arbitrary. Evidence: Some requesters don't know how to compensate the workers (via reddit post).

Interpretation: It appears that the tasks are unique to the point that there isn't a market price on it. Need to at least have general price ranges for tasks that are correlated to time/difficulty.

Milestone contributors

Slack usernames of all who helped create this wiki page submission: @yenhuang, @jorghi, @ndubnov