WinterMilestone 2 xThavy

From crowdresearch
(Redirected from WinterMilestone 2 xThavy)
Redirect page
Jump to: navigation, search

Attend a Panel to Hear from Workers and Requesters

Team xThavy

  • Thavy Thach - @thavythach

Attend a Panel to Hear from Workers and Requesters

Deliverable

The deliverable for the panel subsection: report on some of the observations you gathered during the panel.

Workers:

  • "This is my life, and i have to make this work for me".
  • Options are extremely limited
  • Indecisive: Eat or HITs first?
  • Turkers have alerts
  • No scheduling breaks
  • Job is 24/7
  • Daily goals
  • Flexible job
  • Tasks are risky
  • Worker rating is affected
  • Tracking wages: Look at HITs and I can estimate how much time i eed to allocate it
  • Tracking wages: Participating time estimates
  • Tracking Wages: Spreadsheets
  • Tracking Wages: Unpredictable on how much I can get a day from 1 penny to $100
  • Never risk going below a 99% worker rating

Requesters:

  • Wants a better management-worker engagement system?
  • Ratings are really important
  • Interesting emails from workers
  • Rapid live feedback from task design implmentations
  • Assumed good quality of work. Automatic Approval, but had some bad quality in the batch.
  • Wish: HIT on an iFrame?
  • Wish: Need support for workflows
  • Researchers like Chris

Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Majority of turkers are U.S. & Indian Based
  • U.S. Turkers around 56%
  • Indian Turkers around 36%
  • 1/3 of respondents/workers have a median annual salary of less than $10,000
  • Motivation is for fun
  • Motivation is for money
  • Workers have a reputation System
  • Workers have worst information on turkers than requesters do on turkers
  • Want to improve task designs
  • External tools are used: Turk Alert, Turkopticon, scripe sides, blogs, Journalism, Crowdsourcing sites.
  • Turkers use TurkerNaton as the main source of information-sharing
  • Turkers can make as much as much money they want for how much work/time they put in
  • Can be a full-time job
  • Why a certain wage?: Extra money
  • Why a certain wage?: Boss & Company Market Depression
  • Why a certain wage?: Experiences vary
  • "AMT is vital to living"
  • Relating to Requesters: "Find good requesters, avoid bad ones"
  • Turkers wage expectation: "attractive wage" or "$7.25/hr"
  • Turkers don't like the media
  • Turkers don't like the government and regulation


2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Requesters don't have a reputation system
  • Requesters have better information on turkers than turkers do on requesters
  • External tools are used: Turk Alert, Turkopticon, scripe sides, blogs, Journalism, Crowdsourcing sites.
  • A good requester: workers talk good about requestors
  • A bad requester: workers talk bad about requesters
  • Relationships: An engaging requestor in a community can receive praises and be trustyworthy
  • Relationships: Evolution of a Requester can be bad -> decent -> okay -> good
  • Workers telling: Don't be too quick to damn; Be objective in accepting/rejecting tasks

Worker perspective: Turkopticon

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Using Turkopticon to "evaluate their relationships w/ employers"
  • Real worker opinions of "hard" workers on Turkopticon
  • Motivation: fun - kill time
  • Motivation: money - pay rent or pay for groceries
  • Technically not a full-time job since title is freelancer
  • Workers have limited options
  • 67 Participants: Workers felt work regular rejected unfairly; 35/67
  • 67 Participants: Workers demand faster payment; 26/67
  • 67 Participants: Minimum wage or minimum payment was noted; 7/26
  • 67 Participants: 8/67 expressed dissastification w/ requesters and Amazon's lack of response to their response
  • Workers lose income/time from complaints
  • Workers have risks: accidental download/malware
  • Workers have risks: Reduced worker ratings
  • Workers have a hard time raising their rating if no one is accepting their work
  • Workers are taken advantage of
  • Users of Turkopticon use worker ratings to evaluate who they want to work for
  • Workers want to have a say in what they can actually do.
  • Workers have very limited options

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Exchanging email as a requester equals time wasted
  • More privilege over the workers
  • Task design is up to the requester
  • Experience varies and Turkopticon rates requesters
  • Requesters have the option of now losing more workers than usual from this external tool

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Interaction design plays a huge role for the worker
  • Workers are able to complete these tasks whereas computers can't
  • No idea if these answers are good quality or not
  • Workers don't want to spend time reading
  • Workers are time-consumed on every task the do


2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Interaction design plays a huge role for the requester
  • Error-prone for accepting/rejecting tasks
  • Requesters love to do research: subjective tasks vs. objective tasks
  • Requesters ask: What's a good answer?

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Workers don't require real world skills
  • Worers can acquire real world skills, but not required
  • Workers are confused from baffling tasks
  • Disorganized Market: Reputation is a mess
  • Disorganized Market: Scalable Human-Powered Systems
  • AMT Criticism: difficult of searching for work
  • AMT Criticism: difficult of predicting completion times
  • Workers see task design and want improvement
  • Standizing: Trading commodities - queue of tasks w/o reputation in mind
  • Standizing: True Market Pricing - Prioritizing tasks & whatever is highest is on top
  • Working for a worthwhiled tasks, then automate to receive the task later if accepted


2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Requesters give baffling tasks
  • AMT Criticism: difficult of pricing work
  • AMT Criticism: difficult of predicting completion times
  • AMT Criticism: difficult of gaining quality work
  • Curated Garden Approach: Standardize tasks for workers who are trained/recruited
  • Curated Garden Approach: Cultivating qualified workers to do my work and not bad quality workers
  • Standardizations are easily accessible: surveys, transcribing, etc.
  • Standardizing: Reusability - reflections & reuse

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Time to grow AMT from a beta tester
  • Users complain to Amazon and compare AMT's potential to Amazon's marketplace
  • Responsibile for everything they do as a worker
  • Better reputation system
  • Better task search interface
  • Difficult for new workers to grow to big guys
  • Workers should have more qualification tasks to vary the quality of work
  • Disconnect payment from rating: acceptance & Reputation together is not a great idea
  • Seperate HITs and ratings by type
  • Make an API for workers to be more effective
  • Marketplace is broken since we have tools: Turkopticon and/or Hall of Fame/Shame on turker Nation
  • Workers should not be afraid of worker for a particular requester

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Problems never addressed (identified years ago): Scaling up
  • Problems never addressed (identified years ago): Managing complex API
  • Problems never addressed (identified years ago): Managing execution time
  • Problems never addressed (identified years ago): Ensuring Quality
  • Users love to complain to Amazon
  • Responsibile for everything they do as a requester
  • Complaints on better interface to post tasks
  • A requester trustworthiness guarantee
  • Hard time creating tasks using API, CMD-LINE, etc.
  • AMT encourages requesters to build their own interfaces and own workflows from SCRATCH
  • New users would basically suffer using these interfaces
  • New users require previous knowledge to use AMT
  • Keeping track of working history; 'previous work'
  • Allow rating of workers by requesters
  • Requesters abuse power depending on their intent: good or malicious
  • Should have an objective rating system of the work

Soylent: A Word Processor with a Crowd Inside

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Major Populations of AMT: Well Educated, Moderate-Income Americans
  • Major Populations of AMT: Well Educated, Less Wealthy Indians
  • Externals tools to maximize performance: attractiveness of tasks
  • W/O clear guidelines/instructions, work is ambigious
  • if given an error, workers add on to the error making it more of an error

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • AMT costs money
  • AMT is error-prone
  • Best quality is wanted
  • Externals tools for finding workers: worker availability, demographics, etc.
  • Have a set of our own worker pools


Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Note: Please give me lots of feedback here!

Workers Needs

Example 1:

  • Need: Workers need to be actively doing tasks 24/7 to be able to live in the world
  • Evidence: During the panel, one individual said, "This is my life, and I have to make this work for me".
  • Interpretation: It's not just about her life and making it work for her. It's about living her life as much as possible; it was something keeping her in this world as AMT doesn't pay well. She has to be constantly doing HITs/tasks to continually be on par with the life pitted against her.

Example 2:

  • Need: Workers need to rate on a reputation system for requesters
  • Evidence: During the panel, 5 words said: "Anonymous frustrating complicated trick [very trick]" on describing work on AMT
  • Interpretation: Workers have it bad. Everything is confusing at first glance. They don't have a lot of leverage against the requesters who have more power than them. Workers need to be experienced to continue working for a long time as worker ratings are always going down because of inexperience.

Example 3:

  • Need: Workers need to use real skills to complete tasks
  • Evidence: In an article, workers don't require world skills, but using/acquiring them is a leg up from the competition on AMT
  • Interpretation: Workers need to know how to use a complex system because without knowledge, there is no power. Without power, there is no profit. Why aren't workers using these skills to qualify for quality pay.

Example 4:

  • Need: Workers need to have access to faster payment
  • Evidence: In the article "Being a Turker", individuals point out attractive wage as well as need for real life survival: groceries, rent, etc.
  • Interpretation: This isn't about the wage; it was about the pay cuts given by current employer.

Example 5:

  • Need: Workers need to be happy
  • Evidence: One individual on the panel said they were indecisive: Eat or HITs first?
  • Interpretation: Something is holding them back; Should I do this first or that first? Which will benefit me more? Maybe I can get more performance/be more effective if I ate. Maybe if I did this HIT, then I'd be getting more money for the day.

Requesters Needs

Example 1:

  • Need: Requesters need a better interface to create HITs in.
  • Evidence: One individual during the panel said to create HITs in an iFrame. Another individual said need support for workflows.
  • Interpretation: It's quite difficult for a new requester to understand the interface and all its tricks. It can be still be difficult for an experience requester to understand how to get the best quality of work.

Example 2:

  • Need: Requesters need an engagement system with workers
  • Evidence: In A Plea to Amazon paper, users love to complain to Amazon. Amazon says requesters and workers are responsible for everything they do.
  • Interpretation: The current system on AMT drives people mad. It doesn't have any engagement. Only the ratings are on the workers; not on the requesters. What if we added an engagement system? Would anything change in the process of these requesters?

Example 3:

  • Need: Requesters need quality work
  • Evidence: One of the individuals during the panel said: we have an automatic payments, but some of the work we get are pretty bad.
  • Interpretation: What if we can increase high quality work? How could we get that high quality work we expect? If we expect work already, then what if we switched around to work that we don't want. Would we get a different system?

Example 4:

  • Need: Requesters need worker profile to assess quality
  • Evidence: In Soylent and A plea to Amazon articles, the individuals said that using previous work in a workers' lifetime can play a role in getting better quality.
  • Interpretation: Having a full-time job or doing other tasks or being on different crowdsourcing sites might make us understand what their motivation is in picking AMT

Example 5:

  • Need: Requesters need to be trustworthy to workers
  • Evidence: Turkopticon allows requesters/employers to be rated fairly like workers
  • Interpretation: It's very hard for a worker to know if the requester is going to reject my work or not. It can be good intent or for malicious intent.