WinterMilestone 2 mayankkhanna

From crowdresearch
Jump to: navigation, search

Attend a Panel to Hear from Workers and Requesters

Deliverable

Worker Observations

  • Timings for tasks posted on AMT is unpredictable and hence, workers need to keep on checking when relevant work is available.
  • Workers rely on other platforms like Turker Nation and other forums to know more about the Requester behavior in terms of how frequently he rejects tasks, how quickly he/she processes payments, etc.
  • Workers pick up tasks based on the feedback from other turkers on how they are able to progress with those tasks wherein such feedback is gained through outside forums
  • Experienced workers usually have daily earning goals which they try to achieve and hence keep on taking tasks until they accomplish that goal.
  • Workers are skeptical in picking tasks from new requestors as it would hurt their system reputation if the task is unethically rejected or considered spam.
  • For a lot of workers, the income through AMT acts as their primary income and for others it can act as a necessary secondary income.

Requestor observations

  • Requestors prefer creating their own task templates and integrating them using iframe on AMT.
  • Requestors appreciate the opportunity to clarify their tasks for workers when things are unclear to them.
  • Requestors would appreciate the opportunity to select workers for a task based on their skill set.

Combined Observations

  • There is a strong lack of trust among both parties.
  • Both parties at times are competing against each other in order to serve their interests instead of collaborating to benefit each other (lack of symbiosis)

Reading Others' Insights

Worker perspective: Being a Turker

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • 80% of tasks are carried out by 20% most active Turkers
  • Main incentive for the turkeys to do the work is the payment they receive for the turkers.
  • If Turkers start accepting tasks for low payments on a regular basis, the notion that turkers work for fun would affect turkers negatively, as requesters would not be considerate in assigning appropriate renumeration for tasks.
  • While turkers work on low payments, the collective earnings can be equivalent to minimum full time wages of turkers in the US and in India.
  • While turkers appreciate the secondary income through work on AMT, they would prefer having better paying full-time jobs than spending time working on AMT.
  • Indian workers tend to accept even very low paying tasks as they would still provide substantial income for their livelihood while American workers might not pick up very low paying tasks.
  • Workers tend to spread the good word about a requestor if he ethically accepts task results and processes payments in a timely manner.
  • Workers tend to accept their faults if the rejected tasks provides relevant rationale for rejection.
  • Workers tend to avoid picking tasks from requestors that have a bad reputation that is spread through forums.
  • Since genuine workers tend to avoid tasks from bad requesters, the only workers completing such tasks are spammers which results in poor quality work and establishes an iterative loop of poor reputation for requestor and worker.
  • Workers tend to acknowledge improvement in requestor behavior and take up tasks from a requestor which had a poor reputation initially but has improved his behavior.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Requesters tend to have a lower trust towards workers.
  • Requesters at several times assign unfair payments for tasks that can take a long time to accomplish.
  • Requesters are burdened to go through every result, even if has been poorly performed by a spammer.
  • Requesters want to reach out to same workers that did a great job in the requester’s previous tasks.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Workers perform in a better manner if they are provided certain guidelines to follow when performing the tasks.
  • Workers can be trained to work as experts if they are tasked to do the work in a manner that follows a process similar to the working style of an expert.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Lack of demographic information about workers make it tough for requesters to setup certain tasks on AMT.
  • Lack of skill and expertise information about workers also make it tough for requesters to feel confident about the output received from workers.
  • Requesters can ensure a higher quality result by structuring the work into smaller steps instead of just providing the task that needs to be done.

Requester perspective: The Need for Standardization in Crowdsourcing

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Workers tend to complete even very low paying tasks if they feel they will be able to improve their system reputation.
  • Lack of standardization of payments for different kinds of tasks results in workers picking the high paying tasks only.
  • Workers need to adapt to the task and interface settings for each requester as there is no standard format to specify tasks of different kinds.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Lack of standardization of payments for different kinds of tasks makes it tough for requestors to ensure timely completion of their tasks.
  • Lack of standardization of task formats forces the requesters to find their own workarounds to create tasks in the AMT system and hence involves high redundancy.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • High quality workers tend to leave AMT overtime due to lack of regular high paying tasks.
  • Workers need to trust the requesters that their output will not be rejected and that they will be paid in a timely manner.
  • Workers need to be able to trust the Requester by knowing the review progress of the requester.

2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

  • Requesters cannot increase wages for high quality workers as they lack the ability to distinguish high quality workers from those that produce low quality results.
  • Requesters cannot assign specific kind of tasks to specific users(that are good at a task) due to lack of any skill and background of workers.
  • Requesters are unable to identify high performing workers due to lack of a rating system.


Synthesize the Needs You Found

List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker Needs

  • Workers need to build trust with requesters in the system by having the ability to have meaningful interactions with requesters.
  • Workers need clear guidelines about the task that needs to be done by having clearly laid out expectations for input and output formats.
  • Workers need the ability to know when the tasks they prefer and are skilled to perform are rolled out on AMT
  • Workers need the ability to send the task for review when guidelines are ambiguous and need more clarification from the requester.
  • Workers need the ability to track and search tasks of specific types and from specific requesters.
  • Workers need the ability to know the progress of task output review by requesters and approximate timeline of when they will receive the payment.
  • Workers need to feel pride in the work they do by receiving genuine positive or negative feedback from requesters on their task outputs.
  • Workers need the ability to indicate that an unfair payment is associated with the task to the requester.
  • Workers need the ability to train themselves to improve upon the tasks they do and to learn new skills to effectively do other types of tasks.

Requester Needs

  • Requesters need the ability to review workers for the tasks they do.
  • Requesters need the ability to rollout tasks to specific workers on a prior basis.
  • Requesters need detailed profiles of workers that showcase their skills, the number of tasks they have effectively solved and their error rate.
  • Requesters need the ability to pilot the task with a worker to improve the instructions and identify an appropriate time taken to complete that task.
  • Requesters need the ability to provide incentives to good quality workers in order to lure good workers to complete their tasks on a priority.
  • Requesters need standard templates for specific tasks in order to lay out clear instructions for workers to understand.
  • Requesters need a robust method to punish spammers and provide opportunity to improve work done on a low quality output instead of simply rejecting the work.