Milestone 2 pentagram

From crowdresearch
Revision as of 13:20, 11 March 2015 by Karthiksenthil (Talk | contribs) (Both requester and worker needs)

Jump to: navigation, search

This is the submission page for Milestone 2 by Team pentagram.

Attend a Panel to Hear from Workers and Requesters

Morning panel

Some observations/ideas noted down in the morning panel session :-

Worker perspectives

  • some sites to find list of HITs- easy way to find tasks undertaken by workers
  • the joy of giving back to community is a motivating point for turkers
  • HITExploiter - a script to automate and rate tasks using Turkopticon
  • Motivation for workers in MTurk
    • money
    • wanting to help people
    • social concepts
  • the worker community is very friendly and helpful ; more or less like a Facebook group discussing about anything under the sun
  • worker <-> requester interaction
    • shoot emails directly (instructions not clear)
    • invite requesters to forums
    • verify credibility of requesters
  • many a times pro workers help newbie requesters on how to use GUI,API and all
  • the biggest hurdles for newbie workers
    • finding a matching job - very hard
    • poor UI in Amazon Mturk
    • requires scripting knowledge to make decent money
    • in case of non-US countries ; more difficult to find work
    • frustration due to poorly paying jobs
    • some very important suggestions --> ask questions
  • the dropoff rate on mTurk --> severe, because of no feedback and rejections

Requester perspectives

  • Some common thoughts
    • prevailing wage 8$-10$ per hour - why ? (demographically and ethically minimum wage)
    • main problem -> was task taken seriously ?
    • how to detect cheating -> using open-ended questions
    • time vs. money
      • balancing act
      • give incentives/bonus
    • requesters dont have time to follow forums to know about their tasks
    • requesters prefer personal email
    • no assurity about completion of tasks
    • very complicated to get a task done by a specific worker
  • threshold for rejecting HITs adopted by many requesters
    • use the open ended questions - if responses are direct from Wikipedia or gibberish implies bad work
    • use timers
    • feel that workers didn't read instructions completely
  • problem in India and outside US
    • proxy accounts
    • Indians using USA-based MTurk account to turk and make money
    • account selling - very common pre-2012
  • Some mistakes by requesters-
    • give very less time (no tutorial from Amazon side)

Evening Panel

Some observations/ideas noted down in the evening panel session :-

Worker perspectives

  • They feel the peak HIT time is slightly different from typical working hours. i.e. 7AM to 3PM. Some others however, do feel that weekends and unusual hours like midnight-3AM are more productive, as there are lesser people working at that time on better HITs.
  • Workers are equivocal about the system of assigning ratings to workers. Since there is no concept of task-specific rating, a person who has a high rating in transcription might be misconstrued as skilled at translation, for example. oDesk overcomes this problem by a method of recommendation where requesters can send private messages to other requesters, recommending a worker.
  • While MTurk can also be used for some volunteer work like filling free surveys, workers generally prefer working for money.
  • Income from doing micro-tasks on MTurk provides a highly variable income. What takes 2 days to earn in a certain week might take 6 days in another.
  • Workers would prefer questions having better tagging, and better description of tasks. They would not want to see nonsensical tagging like "Hey! This work is fun!" for something like 40 minute surveys.

Requester perspectives

  • Requesters would like to see a platform which is conducive for social science research, where demographics of the population are known. This would ensure that people don't fake survey data.
  • Personalised tasks are really hard to convert to turk-able tasks. So, some tasks are can't be put up on MTurk at all.
  • Single-person tasks are also hard to request work for. For example, analysing audio files is a single person task.
  • Requesters would love to have a system where they can request for small trivial tasks to be completed, and based on the outcomes of that, handpick the productive workers to assign the harder task to.
  • In order to check the correctness of the user input, it is not necessary to give questions which have answers already. There are other ways like attention checks or honesty checks.
  • They prefer breaking down tasks to the smallest possible extent and then requesting workers to complete them.
  • Requesters themselves feel the worker rating system is bad. They feel workers are defensive about their rating, and that defers them from working properly. Also, at the end of the day, it results in a bad relationship between the requester and the worker.
  • Also, as the requester, there's always a chance that a worker with a high rating might not be good, as other requesters also might have been reluctant to reject a HIT.
  • If a requested HIT is not adequately responded to, requesters go to forums and find out from workers if the pricing of the task is appropriate, and change the pricing appropriately.

Reading Others' Insights

Worker perspective: Being a Turker

In the paper, the frequently discussed issues and solutions of requesters on crowd sourcing platforms has been augmented with, the not discussed issues of the Turkers. It clearly outlines one sidedness of the current crowd sourcing platforms as

  1. Information assymetry
  2. Imbalance of power between the requesters and turkers

The research seems to be elaborate and realistic,in terms of the opinions gathered.

Observations about workers

  1. Clearly, even though there are some workers who do Turking as a source of entertainment/experience, most would want monetary gains from spending their time and effort.The most matured workers would aim at not only high paying jobs but also interesting jobs.The pay expectations seem to have a varied perspective with some satisfied of earning something extra and others not by doing excessive work for small pay.
  2. There are various range of workers making various amount of cash on crowd sourcing platforms.It has to be noted that the cash made is usually not enough to rely upon as a constant source of income.The interest common to all turkers is to make more cash than their previous attempt.
  3. Turker nation and similar forums are typically used by turkers for reviewing Requesters.Though on the crowd sourcing platforms themselves there are hardly any review system for them.
  4. An interaction between a turker and the requester will be a great platform exchange of necessities and making tasks more viable from both their perspectives.This also comes with the prospect of amateurs turkers/requesters being guidelined by pro requesters/turkers
  5. It has also to be considered that forums such as these have become stagnant accusation portals, after some point.Though there are instances of turkers owning up to individual mistakes,it largely seems to be a blame game.

Worker perspective: Turkopticon

The paper attributes the development of Turkopticon to overcome AMT's shortcomings.

Observations about workers

  • Their jobs are highly temporary
  • The work culture is going back to the olden days where they were treated as machines - use whenever required to cut costs, pay them meagre amounts without caring about health of workers or fair remuneration.
  • The need to work on crowdsourcing ranges from poverty to laziness and boredom.
  • Workers are defined as contractors subject to laws that tend to bypass the Minimum Wages Act. This practice is not admissible in a court of law.
  • Workers are termed 'resources' or who promise to obey the will of the employees.
  • They are considered ignorant of the big picture and concerned only about their work.
  • Workers are being advertised to requesters like commodities.
  • They and their contributions are invisible.
  • Dispute resolution is very hard and not worth the workers' time.
  • Workers are not provided the opportunity to interact with requesters or other workers and build solidarity.
  • When provoked, the workers responded strongly highlighting their problems.
  • Their work was regularly rejected unfairly or arbitrarily.
  • Very late payment.
  • Requesters are not expected to respond to the workers dissatisfied with a requester's work rejection and many do not.
  • Dispute resolution is very hard and not worth the workers' time.
  • Labour surplus.
  • They want forums to voice their concerns.
  • They want to build long-term work relationships with requesters and fellow workers to bring in more accountability and visibility.
  • They want short response time.

Observations about requesters

  • They exploit poor working conditions in developing countries to reduce expenses.
  • Workers can be accessed through API.
  • Their needs are given more importance compared to that of workers'.
  • They are seen as innovators, harbingers of change and not concerned with working conditions.
  • Employers give all instructions to workers without using their skills/knowledge input, set price without negotiating with them, provide approval ratings to workers that decides their proficiency, expects them to take exams on that field.
  • Once a worker submits work, the employer can choosewhether to pay for it. This discretion allows employers toreject work that does not meet their needs, but also enables wage theft.
  • Because AMT's participation agreement grants employers full intellectual property rights over submissions regardless of rejection, workers have no legal recourse against employers who reject work and then go on to use it.
  • Requesters are not expected to respond to the workers dissatisfied with a requester's work rejection and many do not.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

This paper talks about how Micro-task markets like Amazon Mechanical Turk provides a potential paradigm for engaging a large number of users for low time and monetary costs.

Further, it superficially explains the advantages and disadvantages of such a paradigm with examples of experiments conducted on users, and talks about measures to be taken by requesters to ensure correctness of input provided by the workers.

Advantages of micro-task markets

1. They are really good for getting small human-intelligence tasks done.
2. For tasks which require a lot of human participation like surveys, design testing, rapid prototyping, ratings, performance measures etc, it is very useful.
3. The potential of having a lot of people from diverse background providing input is very appealing.
4. Cost of acquiring user input is very low.

Disadvantages of micro-task markets

1. Inability to ensure authenticity of work.
2. Effort needed by requester to validate the input obtained.
3. Workers try to game the system by providing nonsensical responses in a short span of time.
4. People may be deterred by the low pay of the tasks.

In the paper, they explain two conducted experiments:

Experiment 1

Make people rate Wiki articles exactly like Wiki experts would rate them. (7-point Likert scale) The ratings were to be based on how well written, factually accurate, neutral, well structured, and overall high quality the article was. Also, user was supposed to fill a free-form text box for feedback.

RESULTS Very weak similarity between experts' reviews and crowd's reviews. Lots of responses were irrelevant. Free-form text had semantically empty content in some cases.

Experiment 2

Same as above, except that there was also some Quality Control questions to ensure that the people answered some questions which had a definite, known answer before attempting the actually rating. Also, they were supposed to write 4-6 keywords describing the article, which ensured that they read the article. This resulted in much more similarity to the experts' ratings.

Guidelines to requesters

The paper suggests that requesters should add verifiable questions to the task, to ensure worker honesty, and make the worker believe that the answers would actually be scrutinised.

Requester perspective: The Need for Standardization in Crowdsourcing

Observations about workers

  1. Tasks are chosen by workers through a online spot market.
  2. Workers are not sued or sacked for unsatisfactory task completion except that they don't get payed for HITs they completed.
  3. Tasks which are high demanded generally requires low-skilled workers.
  4. Workers need to strictly and constantly follow the rules in case of standardised tasks.
  5. Workers are free to choose any tasks( all of which differ in terms of level of difficulty and skill set required for each task).
  6. Method for suitable task retrieval by workers is inadequate and inefficient.
  7. Users of crowdsourcing platforms often get mixed results, which is quite fumble.
  8. In "curated garden approach",practitioners gain the scalability and cost savings of crowdsourcing.

Observations about requesters

  1. Can post any kind of tasks he/she wants and conducive to their terms and conditions and pricing.
  2. Reputations are weak and subverted.
  3. Occasionally we encounter scammers too.
  4. Some simply recruit workers(as accomplices) for criminal or offensive activities.
  5. Don't price the tasks nominally.
  6. Cannot rely on the quality of the task performed.
  7. Requesters evaluates the answers independently.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

The author of this blog is an experienced professor at the Department of Information, Operations, and Management Sciences at Leonard N. Stern School of Business of New York University.

At the time of writing, he had an experience of using AMT for almost 4 years. He gives a critical analysis of what's been missing in this platform through the blog post.

Some thoughts of author

  • A need to evolve
  • author is stressing on the fact that Amazon has completely alienated itself from the working and policies in mturk (the hands-off approach of Amazon)

Observations about workers

Trustworthiness guarantee for requesters
  • requesters on Mturk are serving like slave masters
  • some common problems with requesters
a) reject good work
b) not pay on time
c) incomplete info on tasks
  • new requesters tend to leave the market if they are not guided by experts on how to post tasks
  • some objective characteristics that workers should look for in a requester before working for him
a) speed of payment
b) rejection rate for requester
c) volume of work posted
  • these call for a system which can present all this information in a format that is accessible to every worker
  • a trustworthy market environment reduces the search costs for both requester and worker
A better user interface
  • make task finding an easy process for workers
  • workers have no means of navigating through the sea of tasks to find those that match their interests
  • this forces the workers to select tasks based on some priorities ; this inturn leads to an uncertainity in the completion time of the posted tasks on Mturk
  • some solutions proposed by author
a) an interactive browsing system
b) improvised search engine
c) a recommender system to post HITs to workers

Observations about requesters

A better UI to post tasks
  • less technical overhead ==> better online marketplace
  • requirements that every requester must satisfy
a) quality assurance for submitted HITs for a task
b) proper allocation of qualifications
c) break tasks into a feasible workflow
d) classify workers
  • Author points out an external API for running iterative tasks, Turkit, which has been very user-friendly for requesters especially
  • Mturk is requiring the requesters to build the app from scratch to orient it according to their needs
A better and true reputation system for workers
  • current reputation system uses no. of HITs completed and approval rate which are easy to manipulate
  • why a good reputation system ? because if requester can't differentiate a good from a bad worker, he tends to assume that every worker is bad
  • suggestions from author for a new reputation system
a) More public qualification tests
b) Track working history of workers
c) Rating of workers
d) Disconnect payment from rating
e) Classify HITs and rating
f) API for all the above features

A critical fact stressed by the author:

 A labor marketplace is not the same thing as a computing service. Even if everything is an API, the design of the market still matters.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc


We explored r/mturk subreddit to get some real life observations

  • Users are very apprehensive about the approval/rejection system,feel that they are succeptible to scam Link
  • The turkers are not positive about the current mail communication channel,task specific chat rooms are what they are looking for.Some turkers do communicate Link
  • Mobile device compatibility is also being under consideration.
  • Turkers in turn demand for a requestor rating mechanism and work statistics(For ex:microWorkers provides a small job description of how musch time approximately the job would consume,if that was statistically gathered it would be more information)
  • Turkers want to find skill specific tasks and without wasting much time on it.
  • Some of the requesters find the need of adjusting requirements for the tasks dynamically depending on the response rate and quality.
  • The earnings should be kept track of.If it exceeds the tax filing minimums,then it should be handled appropriately.
  • Some frustration has been gathered from turkers and requesters(in unison) towards the absolute unaccountablity of the host of the platform(Ex:Amazon for AMT).There are issues of unspecified-bans/not user friendly signups/problems in posting HITs etc Link


By going through TurkerNation posts and trying to indulge in chat, the following were observed:

  • The workers are motivated by money and do not enjoy taking surveys unless they are being compensated.
  • They feel their work is unfairly rejected many times and the requesters never reply to their queries. Such conflicts tend to take more of workers' time and they give up or shift to a different platform.
  • They complain about late replies, late payment, low payment.
  • They want to know and rate the requestors just like they are being rated.
  • They feel that the platforms give more importance to requestors than workers.
  • They want a way for them to select good quality work/requestors.
  • They feel that the rating system and number of hits can be easily fabricated to gain good appoval, so they want solidarity and honesty among workers

mTurk Forum

Log of an IRC chat with a turker at #mturkforum on freenode File:IRC_chat_mturkforum.pdf

Synthesize the Needs You Found

General Needs

  • The host organisation of the platform(Amazon) should not be alienated from the requester-worker community
    • Evidence :
      • Technological growth of the platform in terms of new features(as per the changing requirements) is absolutely stagnant.
      • No accountability for user problems or issues on the platform
      • No feedback or interacting mechanism with the community
    • Interpretation : Role of the host organisation is critical in the healthy functionality of the community using the platform

Worker Needs

  • There is a need for better classification of tasks and a corresponding inbuilt recommender system of tasks to a worker
    • Evidence : Workers spend more time searching for a task rather than completing the same. This is observed commonly for newbie workers. Sometimes this frustrates the worker leading him to drop-off the platform.
    • Interpretation : The platform should be as user-friendly as possible to a worker. Worker should not spend more time figuring out what/where to find tasks.

Both requester and worker needs

  • There is a need for an inbuilt system to prevent scammers both as requesters or workers.
    • Evidence : Presence of external mitigation techniques. Requesters adopt methods like repeated tasks, setting timers on tasks, open-ended questions, etc to thwart fraudulent workers. On the other hand workers have various forums and complaining mechanisms, along with tools like Turkopticon to rate tasks.
    • Interpretation : A guarantee about the satisfiable completion of tasks and the guarantee of doing a useful/rewarding work is needed.
  • There is a need for flexibilty in the platform for both requesters and workers.
    • Evidence :
      • For a Requester : Once a task has been posted, there is no provision for him to modify the specifications of the task based on the changing requirements or the responses obtained.
      • For a Worker: There is no method to resubmit or edit a submission for a task made by a worker based on reviews or expectattions from the requester.
    • Interpretation : Flexible environments are a must for a healthy market