WinterMilestone 2 yashovardhan

From crowdresearch
Jump to: navigation, search

Team yashovardhan

Stanford cr.png
  • @yashovardhan : Yashovardhan Sharma

Attend a Panel to Hear from Workers and Requesters


Some of the observations that I gathered during the panel are listed below. I've separated them out into the Workers and Requesters perspectives.

Worker's perspective :

  • This is serious work for them, i.e, their livelihood, hence they are looking for job security but not in the traditional sense. They want there to be some representation of their views and thoughts inning the community, and a change from the current paradigm where requesters hold all the power.
  • Wages. They want fair wages for their time.
  • Good communication with the requester. This way they can clarify the task if it is unclear, or find out exactly why their work was rejected.

Requester's perspective :

  • Reliability from the worker community in general. Currently requesters cannot rely on the community as a whole, but maybe only on a fraction of workers that are dedicated and sincere in their work.
  • Assurance of quality. They need some quantitative method to verify the quality and correctness of the work that has been submitted. Currently MTurk doesn't provide any such mechanism and most major requesters resort to creating their own systems and methodologies to judge the quality of work provided to them.
  • Humanisation of the entire process. Requesters (especially large ones) tend to forget that it's people who are doing their work and not just a bunch of Serial Nos. being shown on their dashboard. The current dehumanisation of the entire process makes it easy for requesters to be much harsher while rejecting a worker's work.

Reading Others' Insights

Worker perspective: Being a Turker

Observations about Workers :

  • As mentioned earlier, this is a lot of workers primary occupation. Hence, they take it seriously.
  • Some workers work for fun, but mostly everyone is there to earn a living and support their family.
  • Workers are human too. They like it when they are treated as such by the requesters.
  • Workers are an extreme bunch of people. They can appreciate requesters who are fair and just, but can be very critical of requesters who are unreasonable and reject their work without adequate reason.
  • Wages are a complicated topic and there seems to be no general consensus on what an hourly rate should be. Views vary a lot with change in geography. Although, most agree on what shouldn't be the wages - something very low like $1/hr.
  • Workers form a strong community and tend to help each other out a lot.

Observations about Requesters :

  • There is a lot of variation in the kind of requesters (according to the workers).
  • Some of them were fair and paid promptly on the completion of a task.
  • Some paid well and even clarified several concerns that the workers had regarding their HIT's. They generally seemed to care about the workers.
  • A bunch of requestors didn't provide constructive feedback and unfairly rejected the work done by the workers.
  • A lot of requesters paid poorly. Their HIT's had rewards widely accepted by the worker community as very low and insufficient.
  • Requesters also faced problems while using MTurk. The interface wasn't the friendliest, the kind of tasks they could design were limited etc.

Worker perspective: Turkopticon

Observations about Workers :

  • Most workers were there on MTurk to earn a livelihood. Some were there for fun.
  • Most workers were unhappy with the payment system. They wanted their wages to be a certain minimum (and not as low as requester can set them), and also wanted prompt payments when their work was accepted.
  • Workers believed Amazon was turning a blind eye to the numerous problems they were facing and the concert that they had raised.
  • There was a lot of unnecessary focus on approval ratings, instead of quality of work and work ethic.
  • Workers found no way to way establish a longer relationship with a requester that they liked.
  • Among some workers there is a fear that by giving a requester a very critical review, they may be blacklisted or face some sort of retribution.

Observations about Requesters :

  • Requesters try to find ways to minimise friction in passing data output from HIT's to their systems that actually use that data.
  • By focusing too much on the results, some requesters subconsciously stop treating their workers as humans.
  • Too ensure quality of work, requesters use qualifications and other criteria that a worker must meet before they can perform a task.
  • Requesters had complete authority on the wages to pay a worker for a particular task. They could quite literally dictate the terms of the agreement.
  • According to the guidelines of MTurk, workers are not obliged to respond to a workers concerns, thereby perpetuating a cycle of mistrust and skewing the balance of power in the favour of the requester.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

Observations about Workers :

  • Workers tend to work on more recently posted HIT's. The older the HIT gets, the lesser attention it attracts from workers.
  • Questions should be posed in an appropriate manner, keeping in mind the workers answering the question. Questions which can be very long or very vague attract meaningless and uninformative responses from the workers.
  • Some workers will always try to cheat the system.
  • The réponse of the workers greatly depends on the structure of the HIT provided. While some led to greater attention to detail and better quality, the others did not.

Observations about Requesters :

  • Requesters see MTurk as a massive enabler for getting large-scale work done in a time and money efficient manner.
  • Most requesters are cautious about the approach with which they formulate a HIT.
  • Since most requesters put in time designing their task, they expect the result from the workers to be of equally high quality.
  • Worker interest and response time vary greatly with structure of the task that it is provided. Hence the task design and structure has to always be kept in mind.

Requester perspective: The Need for Standardization in Crowdsourcing

Observations about Workers :

  • Workers face difficulties in learning how to use the platform interface, and also while trying to comprehend the instructions provided by the requester.
  • Quality has no uniform meaning across the platform. Hence it is difficult for workers to judge what is "good" quality for a particular task.
  • Workers are not bonded to a platform or task. They can choose their work and timings freely.
  • Workers continuously worry about their reputation on the platform, since it the equivalent of a credit score. A bad reputation might well destroy their career on that platform. This also opens doors to potential for abuse and hence needs to be taken more seriously.
  • Wages are a big question mark. No one can seem to decide and reach a consensus about anything.

Observations about Requesters :

  • Requesters, not unlike workers, learn from their mistakes and become better with experience. The design and structure of their tasks improves with time.
  • The learning curve for new requesters is pretty steep and hence they waste a significant amount of time setting everything up initially.
  • Prices for a task remain static on a dynamic market. Requesters have no way to change the price of a task on the basis of popularity or demand.
  • Requesters lack ways to screen, train and incentivise workers to perform tasks for longer durations of time.
  • Requesters have complete autonomy on deciding the wages for a task.
  • They lack ways to control the estimated time it will take to finish their task completely and also face uncertainty regarding the quality of results received.
  • Requesters in general enjoy significantly more power on the platform as compared to the regular worker.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

Observations about Workers :

  • Worker require external websites and forums to learn about the "rating" of a requester.
  • It takes time for a worker to trust a requester. This leads to a situation of distrust between the two for a long period of time.
  • Workers worry about mass rejections and unreasonable requesters.
  • There is not time frame provided to workers int high they are guaranteed to receive their rewards.
  • They cannot see the rejection rate of a requester.
  • Workers cannot appeal against the rejection of their work by a requester.
  • They are unable to search for requesters or specific tasks they find interesting.

Observations about Requesters :

  • They find it hard to design, create and post a task on MTurk.
  • Creating and designing a task is a time and money consuming process.
  • They have no way to distinguish between good and bad workers. Rates have to be set assuming every worker is bad.
  • Have to use iterative processing, i.e, repetitive tasks to ensure the quality of results received.
  • No way to see the past history of a worker. They find it hard to trust a potential worker.
  • Complete autonomy over wages provided per task.
  • They have no way to classify or categorise similar tasks together.
  • Experienced requesters split work into smaller batches, over different intervals of time.
  • No way to deal with bad quality of work apart from leaving the platform or seeking professional help.

Synthesize the Needs You Found

Listed below are my most salient and interesting needs for workers, and for requesters. Each one is backed up with evidence: at least one observation, and also an interpretation that goes along with it.

Worker Needs

Needs of Workers :

  • Workers need to be respected by their employers. Evidence: Sanjay said in the worker panel that he wrote an angry email to a requester who mass-rejected his work. Interpretation: this wasn't actually about the money; it was about the disregard for Sanjay's work ethic.
  • Workers worry about the fact that requesters cannot be held accountable. Evidence : Several people in the above papers point out how their work can be rejected point blank by a requester even though they may have put in a lot of effort into it. Interpretation : Workers are looking for equality and fair representation in the decision making process. They do not want to be overruled without having a say in the matter.
  • Workers need to know larger picture and objectives behind the micro-tasks created by requestors. Evidence: It is hard for workers to deal with payment uncertainty as is evident in the paper "In A Plea to Amazon: Fix Mechanical Turk!", where the author presents the case where small-task requestor can take advantage of the worker by the hiding details related to task. Interpretation: Workers cannot see the total volume of posted work and decide whether it is worth of time and money to learn the new task from the requester. This creates uncertainty about income and job. Knowing what lies ahead will help workers to organize their activities and make significant money for living.
  • Workers want fair pricing of tasks. Evidence : As is pointed out by several irate workers above, requesters hold complete autonomy on the rewards associated with a task. Interpretation : Workers want fair wages for fair work. They do not want to be underpaid or overpaid, just adequately compensated. This seems a far reality in the current system where the requester holds all the power, while the worker has none.

Requester Needs

Needs of Requesters :

  • Requesters need to trust the results they get from workers. Evidence: In this thread on Reddit (linked), a requester is struggling to know which results to use and which ones to reject or re-post for more data. Interpretation: it's actually quite difficult for requesters to know whether 1) a worker tried hard but the question was unclear or very difficult or an edge case, or 2) a worker wasn't really putting in a best effort.
  • Requesters need to be able to judge the integrity of a worker. Evidence : The research conducted by "Kittur metal 2008" shows that it is hard for requestors to trust the workers. The authors prove that spammers can significantly degrade the quality of work. Interpretation: Workers have limited opportunity to directly interact with requestors and understand the tasks. This isolation makes it harder for them to establish trustful professional relationship with the requestors. Due to the small percentage of bad workers, requestors believe that all the other workers are bad.
  • Requesters need a more cost and time-effective way of designing tasks. Evidence : In the paper "A Plea to Amazon: Fix Mechanical Turk", the authors show that several requesters waste tremendous time and money trying to navigate the interface and create tasks. Interpretation : There needs to be a system that makes it much easier to create and design any type of task effortlessly. Understanding the user-interface shouldn't take ups lot of time and effort, instead it should be intuitive to the point of obviousness.