WinterMilestone 2 by Team - witty

From crowdresearch
Jump to: navigation, search

Team members: @vrinda1994 @witty123

Attend a Panel to Hear from Workers and Requesters

Deliverable

Workers

  • Workers discussed how a day is like in their lives and most of them described working on crowdsourcing platforms as 'unpredicatable' as they never know when HITs of their interest will be posted; so one always has to be on 'edge of her seat'. Due to such nature of working, one cannot solely rely on crowdsourcing to pay their bills and it is difficult for people who budget their incomes.
  • They described crowdsourcing as a 24 hour cycle, 7 days a week. It can never be scheduled; people jumped even in middle of the night to do some HITs.
  • We also got to talk to some workers like Laura, who are unable to work in office jobs, due to their special conditions. For such workers, crowdsourcing is a blessing and they are pretty excited to work on these platforms.
  • Most of the workers work on HITs keeping in mind the effect on their ratings. So many of them are hesitant to work with new requesters. They make sure that they are never 'putting too many eggs in one basket'. They verify whether their is a human behind the task posted by sending emails to the requester. They get a mixed bag of responses.
  • Time estimates given on platforms are often misleading, according to workers. They use 'worker forums' to find the true time to finish a task and then they decide whether to accept a HIT or no, although their is a large variation depending on the individual. So, it can't be generalized. There are also some tools to estimate time of completion.
  • Workers tell that their earnings are hugely variable, from hundred dollar one day to couple of dollars the next day. Working method depends on individuals- some spend their 'slow day' working on small tasks which pay few pennies and they do many such tasks to complete their daily goal. Others like to spend their time doing some other fruitful tasks than turking.
  • Workers summarized requesters as anonymous ,frustrating ,complicated ,tricky, terrible.

Requesters

  • Requesters designed HITs for academic purposes which are helpful in designing an experiment, testing a hypothesis and see whether a tool works by deploying it live.
  • Many a times workers don't understand the HITs properly, which is illustrated in the results. Workers are always emailing them and requesting for clarification.
  • Some requesters post HITs designed for 'productivity' like 'label book on bookshelves'. In such cases, a quick response is needed.
  • When automatic acceptance of HIT is done, they often find 'crap results'. To tackle this they often revoke the qualification of a worker if their work falls below some quality percentage.
  • Requesters found it difficult to interact with workers when they scaled HITs to larger volumes. They had to hire some undergraduates as assistants to clarify the queries of workers.
  • Rejecting a task is difficult, especially when the task results are between a 'bottom threshold' and a 'upper threshold'. Fairness is important, so can't reject a task randomly.

Reading Others' Insights

Worker perspective: Being a Turker

  • What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.
  1. Turkers turk for money i.e money is their primary motivator.
  2. Median income of 1/3rd turkers present in US is <10,000$.
  3. Turkers value market ethics. Relationships matter.
  4. Primarily focused on turkers based in US.
  5. Turkers work together to gain knowledge from each other.
  6. Turkers get offended after they have worked on a particular HIT for some considerable time and that HIT gets rejected.
  7. New turkers focuses on getting more and more of their HITs accepted.
  • What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.
  1. If a requester is willing to communicate with the workers politely, then he is considered as a good requester.
  2. Requester might reject someone's work without giving any concrete reason.
  3. Believe that the task they post is fun which might be a misconception in some cases.

Worker perspective: Turkopticon

  • What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.
  1. Workers want faster payments.
  2. They treat them as their normal jobs.
  3. They might have to leave the platform if their HITS don't get accepted.
  4. They want to have some kind of relationship between them and requesters.
  • What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.
  1. Requesters are like GODS who:
    1. can reject HIT result without any concrete reason.
    2. can use the results of HIT even after rejecting it.
    3. may or may not reply to a particular worker for certain issues.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Workers like to do tasks which require less time or the tasks in which they can give any kind of random answers.
  • Workers take this advantage of the system multiple times.
  • Workers also take advantage by providing false information about themselves.


2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Requesters spend a lot of time in checking and rejecting the answers submitted by workers.
  • Requesters try to design tasks such that workers have to spend the same amount of time in completing the task irrespective of their authenticity.
  • Requesters don't have any control on who is going to perform their tasks.

Requester perspective: The Need for Standardization in Crowdsourcing

The paper summarizes that currently the crowdsourcing platforms are flexible but there it needs to be standardized for better performance and quality.

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Workers find it difficult to perform a new task properly due to incomplete/unclear instructions.
  • Searching for tasks is an effortful process for workers.
  • Many workers are spammers, or are submitting tasks without any effort.
  • New workers have a hard time adjusting with the interface due to bad design.
  • Workers feel that adjusting to requirements of each new requester is hard.
  • Workers give a lot of importance to reputation of requesters.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Some requesters can be recruiting accomplices for nefarious activities.
  • They find it difficult to price tasks.
  • The longterm requesters learn from their mistakes and fix the design problems.
  • New entrants have difficulty fitting in.
  • They cannot fluctuate the price of tasks without removing and reposting the tasks.
  • Requesters have to maintain their reputation.
  • They cannot priortize their tasks and get them completed early.
  • Requesters don't always get high quality results and hence they are often unsatisfied.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

This article is a brilliant, sarcastic take on what all is wrong with Amazon's mechanical turk by one of its many dejected user. It couldn't be explained in better words!

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Workers have difficulty in searching for a task belonging to a particular category.
  • There is no guarantee that a particular requester can be trusted by a worker.
  • No working history of workers is available.
  • Workers are not rated by the requesters.
  • They are not paid if the results do not meet requester's expectation, even after putting all the effort.
  • Many workers use Turker Nation and TurkOpticon to learn more about the requesters.
  • Workers can't review requester's rejection or acceptance rate.
  • They cannot see the volume of posted work to decide whether it makes sense to invest the time to learn the tasks of the requester.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Difficult for requesters to post task because of poor interface, which is not user-friendly. They build their own interface and workflow systems from the scratch.
  • Small guys/newcomers cannot grow.
  • Requesters have a hard time differentiating good from bad workers.
  • They use redundancy to improve the quality of results, i.e one task is completed by many workers.
  • Requesters serve as 'slave masters'- they are free to reject any work , they might not pay on time.
  • Newcomers get low quality results as good workers are wary of completing their tasks and as a result only spammers and inexperienced workers complete their tasks.

Soylent: A Word Processor with a Crowd Inside

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Workers with a basic knowledge of written English can support both novice and expert writers.
  • Workers can solve problems that Artificial intelligence cannot.
  • Different sets of workers are employed to work on different stages of Find, Fix and Verify.
  • There is high variance in amount of effort that turkers put into a task. They are classified as Lazy Turker(whos does as little work as necessary to get paid) and the Eager Beaver(who go beyond the task requirements in order to be helpful, but create further work for the user in the process).
  • Output of 1 worker is sometimes used as input for other, which can compound errors.
  • Turkers like to give suggestions more than doing some original work.
  • Turkers do not have expertise in domain knowledge.
  • Workers do not gain any legal rights to the document they edit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • It might be difficult for requesters to create tasks that are scoped correctly for a Mechanical Turk worker.
  • The requester or user of Soylent may spend money on a buggy command of Human Macro.
  • Requesters might have a difference in stylistic opinion with the workers.

Synthesize the Needs You Found

Worker Needs

A set of bullet points summarizing the needs of workers.

  • Workers need a stable minimum income through turking.
  • Workers need a proper reputation system for requesters.
  • Workers need an interface to communicate with requesters easily.
  • Workers need a search engine to browse tasks which are more suited to them.
  • Workers need a recommender system to recommend HITs to them based on their previous HITs or based on their likes and preferences.
  • New workers need a better on-boarding system.
  • Workers need to be paid fairly and on time. There should be an incremental task based payment system so that a worker doesn't feel disappointed if his work is rejected after he spent a lot of time on it.
  • Workers need a system which verifies the specification (like estimated time of completion, authenticity of task, authenticity of requester etc) on HITs.
  • Workers need to have control, to decide whether their work is being used or not after being rejected.

Requester Needs

A set of bullet points summarizing the needs of requesters.

  • Requesters need an automatic acceptance/rejection tool which verifies the result of tasks fairly.
  • Requesters need a better user interface to post tasks easily, one which does not use command line tools.
  • Requesters need to give a feedback to workers on their results if it does not meet their expectations.
  • Requesters need to post their tasks with proper, unambiguous instructions and they need to clarify any doubt a worker has.
  • Requesters need to be able to choose the workers for their tasks.
  • Requesters need to have a proper system which suggests fair pricing of tasks.
  • Requesters need to be able to priortize the tasks which are urgent/important.
  • Requesters need to have a better system to differentiate good from bad workers.