WinterMilestone 2 hizai

From crowdresearch
Jump to: navigation, search

Template for your submission for Winter Milestone 2. DO NOT EDIT THIS DIRECTLY - instead, make a new page at WinterMilestone 2 YourTeamName or whatever your team name is, and copy this template over. You can view the source of this page by clicking the Edit button at the top-right of this page, or by clicking here.

Attend a Panel to Hear from Workers and Requesters

Deliverable

Turker Nation is a worker-maintained platform which turkers can communicate with others.

  • They help others to get involved and use the AMT properly.
  • They emphasize on ethical behavior. It's not a forum teaching people how to cheat or make quick buck in AMT.
  • They communicate with requesters so that they can get to a consensus of how to improve the experience during the process.

(It's kind of like a labor union to me because people in Turker Nation are trying to make AMT better and help turkers to be "successful" in AMT.)

Daily Thread is a daily thread that turkers will post some good HITs here, in order to share valuable information.

  • Turkers can start their day from Daily Thread, and keep on finding outstanding HIT opportunities.
  • The intention is to share work access.

Turkers' Life

  • UNPREDICTABLE. There's no way to plan ahead for when you are going to work because great HITs may come up at any moment.
  • It's a 24hrs cycle. Even deciding whether they should have their lunch first or have this HIT done first is a difficult decision to make.

Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Turkers turk for money.
  • Turkers values fair pay.
  • Turkers spend time seeking information to enhance their knowledge of how to earn more money by turking efficiently.
  • Turkers share information with others in order to form a collective intelligence community.
  • Turkers believe they can create a balance work community by deciding their requesters, so they don't want interference from government or academy which might cause changes or even closing up the whole crowdsource market.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Requesters will be evaluated in workers' forum such as Turker Nation.
  • Requesters who reject workers hard work with no obvious reasons, accusing workers with abusing comments, or identifying workers as robots would be recognized as bad requesters.
  • Requesters who are willing to communicate with workers and improve their behavior will be accepted by the workers as good requesters.

Worker perspective: Turkopticon

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Workers as individuals don't really have the power or access to confront the requesters or AMT.
  • Workers feel they've been rejected arbitrarily.
  • Workers want the requesters to be prompt paying them.
  • Workers feel ignored by requesters and AMT because they don't really face the problem that workers raised.
  • Receiving rejection from the requesters is not a small business since it's hard for workers to regenerate their rate.
  • Workers want a forum with no surveillance and they can have dignity and voice in it. (Turker Nation?)
  • Workers want to develop workers-requesters relationship. (Boomerang?)
  • Workers want to have a union. (Some workers strongly oppose to this idea.)
  • Workers value communicatively, generosity, fairness, and promptness of the requesters.
  • The standard of "generosity" varies in different countries.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • "Human-as-a-service" makes requesters easily ignore the effort workers devoted.
  • Requesters have the right to use the results of HIT even if they eventually rejected it.
  • Requesters don't necessarily have to reply the comments given by workers. The cost of replying thousands of comment is higher than the wage of HIT.
  • Requesters tend to change the algorithm when the voice of complain raise instead of revising the decision of rejecting HITs. (Interpretation: They like to prevent this situation from happening again, but not interested in compensating for what already happened. There's no rule to regulate them, anyway.)
  • Requesters have 30 days to evaluate HITs and decide whether they should pay the money.

3) Other: Amazon Mechanical Turk

  • AMT sees human labor as replaceable thing since the system is huge enough to handle the lost of workers. (Worker Invisibility)
  • AMT is accused for not creating a platform that provides a equal chance for workers and requesters to express their thoughts.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Some workers are taking advantages of the system when they can finish a task with little effort (eg. fill in non-constructive or semantically empty words).
  • Workers are all over the world, which might be beneficial for collecting user data with generalization.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Requesters, which are researchers in this paper, are looking for reliable, low cost, fast platform that can help them collect abundant user input.
  • Requesters(researchers) cost of human resources with evaluating and rejecting malicious users' work.

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Workers got flexibility of their works.
  • Workers face difficulty finding work.
  • Workers need to get accustom to different user interface from various requesters.
  • Workers need to get used to different quality request from various requesters.
  • Malicious workers make the whole market mistrustful.
  • Workers giving feedback to the requesters is additional work but beneficial to the market.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Requesters got flexibility of their requests.
  • Requesters might got spammers with their requests.
  • Requesters can't predict time span and task quality.
  • Requesters need to develop a template from scratch.
  • Requesters have no way to know the market price until they actually release their tasks and explore.
  • Requesters giving feedback to the workers is additional work but beneficial to the market.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Workers don't know requesters' object ratings such as payment speed, rejection rate, appeal rate, or total volume of posted work.
  • Workers don't have many access point to find tasks, so they tend to start with most recent HITs and HIT groups with abundant HITs.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Requesters need to build system from scratch, which cost to much time and money.
  • Requesters need to divide tasks into workflow.
  • Requesters can't rate workers.
  • Requesters can't estimate completion time of the tasks.
  • Requesters have no evidence of whether the worker is reliable or not other than "completed HITs amount" and "approval rate".
  • Requesters can only make sure the quality of a worker after many times of cooperation.
  • New requesters releasing a batch with huge amount of HITs might get a lot of bad results from spammers because experienced workers are still waiting to see whether this requester is good or bad.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

TurkerNation Daily Thread

carolyn: "That's pretty amazing. Paid immediately and offered invite for other other studies if you qualify."

podcayne: "I accepted your HIT, however I will not be submitting it. Ten to fifteen minutes for 40 cents with an unpaid screener?????? You should be ashamed of yourself. Would you work for $2.40 an hour?"

carolyn: "Instructions also say, if you do well, you may be invited to other studies."

feb1user: "Took me around 2 hours (watching the video, reading the instructions, tightly annotating the pills, reviewing it and making necessary changes). Would have returned it if it wasn't for lack of work. Extremely exploiting and frustrating. The pay should have been at least $2. Very unfair Requester, IMO. As if they don't know how much time and effort the task requires! While I agree that it's voluntary & they're not forcing us, but is that an excuse good enough to be this ridiculous?"

(HIT description: "Complete a short survey about banner ad design contests (around 3 mins)" ) feb1user: "Around 3 mins, as it says."

Requester: Joe Hanna Qualifications: Masters has been granted, Total approved HITs is not less than 1000, HIT approval rate (%) is not less than 97

chrisfuccione: "Six questions. Easy. No TO"

feb1user: "There were no PredictWallStreet.com HITs y'day, correct? Will there be any today? What's your prediction??!"

Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

  • Workers like the opportunity of being invited to another task if they did well. Evidence: carolyn mentioned invitation to other studies if the requesters approved workers' work quality.
  • Workers care about fair pay. Evidence: podcayne and feb1user argued about the low pay rate of requesters. The pay rate and the working time are not proportional.
  • Workers trust and value Turkopticon ratings. Evidence: when commenting a requester, workers would attached its' TO(Turkopticon) rate. chrisfuccione specifically mentioned the requester he/she said has no TO rate.
  • Workers are interested in specific requesters. Evidence: feb1user was asking about PredictWallStreet.com HITs.

Requester Needs

  • Requesters posting tasks with accurate time indication are good. Evidence: feb1user agreed that HIT takes 3 minutes as the description said. It gave workers a sense of how long will the HIT really takes.
  • Requesters are recruiting high quality workers. Evidence: the quality boundary of Joe Hanna is extremely high.

Milestone Contributors

Slack usernames of all who helped create this wiki page submission: @hizai