WinterMilestone 2 BPHC

From crowdresearch
Jump to: navigation, search

Attend a Panel to Hear from Workers and Requesters

Some observations:

  • There exists significant emphasis on ratings - both for worker and requesters.
  • The decision to take up work depends on risk of rejection which affects ratings.
  • Communication between the requester and workers is important
  • A steady income is not guaranteed fom the platform
  • Communities and information exchange between workers occurs frequently
  • Time estimates are frequently inaccurate


  • The necessity to maintain high worker ratings compels workers to take up jobs only on the basis of potential risk and reward, judging from the fact that workers are using scripts to make these decisions.
  • There is an element of unpredictability, workers have to be able to take up work at the right time when it is requested, to be able to make money. If workers are unavailable or busy at the time, they lose out on high paying work.
  • Requesters are forced to set parameters to guage if the work is of bare minimum quality, to decide if the workers' work is tobe accepted or rejected.
  • The haphazard nature of the work being posted requires a strong community such a Reddit or other forums, where workers can discuss high paying HITs and other issues related to the platform
  • Requesters are either unable to estimate the amount of effort and time required for a task or inaccurately represent such details to attract workers.


  • Workers require a sense of structure and predictability about the scheduling of work, to generate steady income.
  • Requesters need a better way to reject or accept work and resolving disputes, especially in a instances where judging the quality of work can be relative.
  • A medium is needed to commuincate effectively in such platforms, amongst workers and between requesters and workers.
  • An effective tool to accurately guage the effort and time required for a task, for a given worker's abilities is needed as well.

Reading Others' Insights

Worker perspective: Being a Turker

1) There are varied reasons for why people take up this work, but money for survival and extra cash for groceries are some of the big reaons.

  There is a strong community amongst the workers that helps other workers seek out the best requesters, which is critical to the process of   making money.
  Although it is slowly chnaging, many workers seem to feel a little at the mercy of requesters, in terms of acceptance or rejection of their work.

2) Requesters seem to be very wary of fraudsters, bots and the likes. There also seems to be soem disconnect in terms of reaching out to the workers to adequately describe and resolve issues related to the task.

Worker perspective: Turkopticon

1) Turkopticon can be seen as an attempt to balance the scales, to give some voice and a mechanism to provide feedback about requesters, allowing workers to make informed choices about the work they take up. Allowed a strong community based system to develop, where requesters are rated, helping workers avoid troublesome and controversial requesters - helping them avoid wasting time.

Allowing workers to choose the right work helps them reach their daily or weekly money goal faster without wasting their time.

2) Turkopticon made requesters slighlty more accountable. It shows that a system that evlautes and requires both sides of the contract to be vigilant both in terms of quality of work performed and justified acceptance or rejection of work.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

1) Some of the work available is difficult and the 'right' answers can be relative and dependent on the requester's perspective.

2) Not all tasks are suitable for the platform and such crowdsourced humans often perform poorly on certain tasks.

Requester perspective: The Need for Standardization in Crowdsourcing

1) Workers will benefit from standardization by reducing unpredictability in money earned and controversy about acceptance/rejection of work.

2) Companies and requesters can greatly benefit from this by making it easy to compare, judge and set costs to tasks and to evaluate quality of work.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

1) Workers find it difficult to stand out in a sea of similarly qualified workers.

2) Requesters deal with issues related to the design of the tasks and have concerns about the quality of the workers.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

We found several posts about workers attempting to find high paying tasks and posts about workers trying to resolve issues related to rejection of work, simple issues about the platform itself and several guides welcoming new entrants to the industry/field.


Worker Needs

  • Workers require an effective way of estimating effort and time needed for a task. Evidence: One of the panelists said that estimates given by requesters were often inaccuate. Interpretation: Poor estimate impair workers' ability to plan ahead and make informed decisions about taking up work.
  • Workers need a predictable schedule of work. Evidence: The panelist stated that the amount of money generated and the decision to take up the work depended on her availabiity at the time of posting. Interpretation: Predictability would ensure a steady flow of money for workers allowing them to plan in advance.
  • Workers need a superior platform/tool to effectively rate and provide feedback about requester and the work offered by them. Evidence: Turkopticon is a popular tool. Inference: Requesters need to be rated as well, and this feedback influences workers'choices.
  • Workers need a way to stand-out in a crowd, to market and prove their employability. Evidence: The panelists stated that she ran a script to ensure that he doesn't take up too much work that could possibly lower her rating below 90%. Inference: There is too much focus on a worker's rating as a number. It has become almost gamified, where worker try to mximize the risk they can take without damaging their rating.
  • Workers require an effective way to resolve disputes about rejection or acceptance of work. Evidence: Several posts in subreddits indicate that there is frustration about work being rejected, which the worker believes to be unfair. This affects the worker's ratings. Inference: Unilateral rejection of work hurts workers.

Requester Needs

  • Requesters need a better way to filter out poor quality work and fradulent work. Evidence: The panelist stated that he added questions designed to trp fraudulent workers seeking to quickly make money without properly completing the task. Inference: Trust is still an isue between workers and requesters.
  • Requesters need better tools to evaluate workers. Evidence: From milestone one's reports, it is clear that some of the work was rejected outright because of its unsuitability. Inference: Choosing the right workers before they commence work is a challenge.

Compiled By

Sree Nihit, Aditya Nadimpalli