WinterMilestone 2 Despicables

From crowdresearch
Jump to: navigation, search


This is the deliverable for winter milestone 2 for the week spanning January 19th, 2016 to January 25th, 2016.

Team Despicables

Team members

  • @varshine
  • @kr.prastut
  • @afreen

Attend a Panel to Hear from Workers and Requesters


Report on some of the observations you gathered during the panel.

The observations are listed as per the questions:

1. What the daily rhythm feels like? What do the hours look like? How much are you sprinting? How much are you popping in and out?

  • Identifying which jobs to take: Workers use a script to identify jobs to start off with, they rely on forums such a TurkerNation - where different workers have gathered together to help each other out to find tasks and complete them & encourage ethical behaviour between the workers.
  • Forums:Someone starts a daily thread and puts up tasks that are worth working on. Workers need a place where they can get reliable time estimates, nature of requester, etc.
  • Unpredictable: The rhythm is unpredictable and very often there are some tough decisions to take (I need to go get lunch, but something that has value is up – Do I take the job or go eat lunch). There could be HITs posted 24x7. Workers with a schedule need some consistency and stability

2. At requesters : What is your process for creating and iterating tasks?

  • Illustrated by results: Requesters send a small batch out, look at the results and then iterate on the design of the task. Requesters try to translate the task into clear instructions that convey what needs to be done.
  • Engagement with workers: Some workers email and suggest changes to task instructions/design. Requesters and workers find it difficult are trying to communicate with each other and bridge the gap

3. At workers : Batch tasks vs small tasks - How do you view such tasks – is it based on personal preferences?

  • Approval rating and risks: Never risk going below a 99% approval rate - worker uses script to tell her worst case scenarios - very difficult to work back up the approval rates. Worker relies on reputation of the requesters - hesitant towards new requesters - tries assessing them by doing a few HITs and seeing how the requester responds - sometimes emails them and takes a decision based on the email response.
  • Effort-intensive: The worker makes a lot of effort to find standard tasks to do - with no guarantee that it will get accepted

4. At requesters : what type of work do you put up? Batch tasks , small tasks?

  • Checking quality: Use an internal qualification to approve and reject work.
  • Engagement - Batch tasks: Engagement while giving out large number of tasks becomes difficult as there are a large number of workers and tasks running. Engagement is point of improvement for the existing system. Requester tries to manage the different tasks and interactions with the workers.
  • Reputation of requester: Requesters try to maintain a good reputation about themselves to ensure good quality workers take up their tasks.
  • Inhumane nature of MTurk: Requesters see workers as a set of serial numbers and forget that there is a person at the other end. Requesters are consciously trying to be fair to the workers

5. At requesters : What are the types of categories do you want to see? What can’t you do on crowd-sourcing platforms?

  • Hits inside of an iframe: Requester doesn't use the templates provided by Mturk, creates custom to get hits within the iframe.

6. At workers: How do you track your wages?

  • Experience: With experience on the platform, looking at the requester and nature of the task, workers can evaluate to some extent.
  • Budget a day: Difficult to budget a day due to the highly unpredictable nature of when HITs get posted and what type of HITs get posted.
  • Estimating time: Workers try estimating time it would take to complete a task by either looking up at the forums or doing a task or two. These estimates vary from one person to another cannot be predicted.

Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Looking at short term fixes as opposed to better long-term fixes: This is evident in the way they have created a thread to act as a hall of fame/shame.
  • Do not want to compromise for the future: The workers only seem to be in it for purely selfish reasons i.e. do not think of turking as a respectable/permanent job therefore quick fixes work much easier than concrete changes.
  • No organised effort at change: It may so happen that a turker does have a bad experience with a certain requester, but that does not matter to another turker as long as there is pay & work, leaving the worth of the turker as replaceable.
  • Anger on being a part of discussion & debates: The attention focused on crowdsourcing as a profession is causing more researchers to come up with ideas, ideas that will overrule the existing hacks used by some of the unethical and perhaps some ethical turkers in making money. Getting caught up in rules and regulations will needlessly bring about disruption in the short term, which is unacceptable for people turking hand-to-mouth.
  • A missing code of conduct: Amazon themselves are not actively involved with building an ethical code for both turkers & requesters i.e. the requesters & providers are not legally bound to be respectful of one another. There are no laws & rules governing them, except a weird state where a requester can choose to block a turker for no valid reason at all!
  • Turker loses his morale: The unfairness & uncertainty may lead a turker to not produce quality work like they would once have done. Emotional damage of the turker is high and costs heavily to the requester.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Turkers' mutual influence: The majority seems to understand that the quality of work you put in matters and probably causes issues with requesters, at the same time, there are a few that take advantage of this generally ethical environment by venting and taking actions that may affect the quality of work received by requester by fellow ‘informed’ turkers.
  • Demonized requesters: Regardless of how good a requester is in general, they are portrayed as ruthless individuals with no regard for a turker's circumstances & hard work by certain bad apples in the community. They're the 'bad guys' who are in charge and control the money.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Efficiency: Workers seek to minimize the time spent by them on identifying the right requestor & understanding the requirement of the task, such that their efficiency can be improved and they complete more tasks in a day.
  • Identify the right job: Workers try to identify the right job for them - in terms of pay, time , skill required.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Optimization: Requestors try to design tasks such that they receive the most impactful results with minimal investment in terms of money and time.
  • Verifiable results: Requests try to build tasks such that responses can be verified easily - this helps in separating the malicious workers from the honest ones. The given approach helps especially in user studies, where the answers can be different across different users.

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • A standard assembly-line model for workers is missing: Micro-tasks are not haphazardly defined and not consistent, the concept of reusability in terms of workers is nil for similar tasks, as there is no categorization of work.
  • No incentives for good performance: If a turker is good at his work, there seems to be no celebration of that i.e. goes to say that in this organization, the benefits that a person can get from a full-time job in the real world is not open to turkers.
  • Curated garden approach: Since there is a common point of redressal, workers can be assured of timely payment, consistency in micro-tasks etc. Although there can be the occasional well-paying requester that workers would like to work for, as opposed to a constant source of income.
  • Learning about a requester & their tasks from scratch: When the requesters do not heed to certain principles, it is a trial and error process for workers. A lot of time is spent on the workers' side on experimenting with new requesters on the market and figuring out if they are good to work with. Time is money for turkers, and this is an obvious cost that they cannot ignore for their own long-term benefit.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • People work better under controlled environments: The concept of incentives introduces a new edge, one that allows requesters to pick certain people who deserve incentives for their good work. In the current system, there is no way to prevent poor quality of work with the help of these incentives.
  • Pricing of work: No scientific methods used in understanding of how to price a micro-task, if it requires a higher compensation because it may require a little more than regular classification & may classify as actual problem-solving.
  • Curated garden approach: A consistent set of workers that will provide you with HITs you need. This may work out to be costly for a requester, and they will prefer to go back to the open-market concept. There is still certainty in this concept, and multiple requesters can bid off for the set of workers.
  • Reinventing the wheel with every micro-task: There is no fixed categories of work, rules of qualification for the category of micro-tasks or any quality rules followed. It is redefined time and again by every requester.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Attributes for the requestor: Workers want to be able to identify “good” requestors with attributes such as - time on the platform, volume of work posted, speed of payment, rejection rate etc.
  • Assurance: Workers need to be assured that they’d be paid fairly for the time and quality of work submitted by them – even if it gets rejected.
  • Efficiency: An usable and pleasant interface can help workers find jobs they want quickly, assess requestors, understand the task better and get started in no time – this would lead to increase in number of tasks per day
  • New requestors: Workers especially need assurance from a new requestor.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Simple process saves time: Requestors seek a system that brings down the burden that requestors handle, which includes taking care of quality assurance, managing workers according to their qualifications, breaking down tasks into workflows etc.
  • A broader scope to find workers: Requestors would like to see more attributes other than acceptance rate to differentiate between spammers and honest workers. These attributes could include - qualification tests, portfolio of work done on the platform, ratings etc
  • Being fair to workers : Requestors want to be fair to honest workers whose work didn’t meet their expectations.
  • New requestors: New requestors want to show their credibility to get good workers.

Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

A set of bullet points summarizing the needs of workers.

  • Workers need to be able to evaluate a requester. Example : Rochelle mentions being hesitant to take tasks under a new requester. The existence of a forum also indicated the need to know the requestor and establish trust before taking up a job
  • Workers need some consistency in terms of how their day pans out. Example : Words frequently brought up includes unpredictable.
  • Workers need a way to communicate with requesters who reject and accept their tasks. They need to feel a human is on the other end. Example : Rochelle mails requester to just get to know them and understand how they'd respond.
  • Workers need a usable interface they helps them sort tasks out based on their preferences. Example : They use a forum or run a script

Requester Needs

A set of bullet points summarizing the needs of requesters.

  • Requesters need an interface to easily manage and scale tasks on the marketplace. 1) Chris (panel of requesters) mentions that he finds it difficult to manage tasks 2) Author mentions in 'Plea to fix Mturk'
  • A automated scoring/evaluation that differentiate spammers from honest workers. 1) A mention across most papers 2) Requesters talk on the panel regarding the system they have set up to assess quality of work they receive.