Milestone 2 Triple Clicks

From crowdresearch
Jump to: navigation, search

Attend a Panel to Hear from Workers and Requesters


Observations gathered during the March 9, 2015, 8:30AM panel.

What are the biggest hurdles faced by first-time Turkers?

  • Finding work. Turkers can’t imagine using AMT’s interface to make decent money.
    • Interpretation: Turkers are limited in the manner in which they can find work, due to lack of experience
    • Need: Turkers need to be able to search for high-paying HITs without having to resort to using tools outside of AMT. But, ultimately, they need to be able to make ends meet, to be able to pay bills.
  • Not working with requesters who underpay consistently.
    • Interpretation: The initial steps as a Turker are hard and destined to lead one into working for low pay. It’s hard to get one’s footing as a new Turker. Low paying requesters take advantage of this and know that their HITs will get done, even for little money.
    • Need: New Turkers need to learn how to negotiate the ways of AMT without bringing down the stock of all Turkers in the process of building experience and higher approval ratings.

What do you do to communicate with other Turkers? What do you talk about?

  • Community is critical to keeping people going and getting through the uneven workload. But it’s still a question what gets people going day to day.
    • Interpretation: Daily life can be hard for a Turker. The work is often unsatisfying and doesn’t pay well, or evenly.
    • Need: Turkers need to feel valued, even though their reason for being there (the money) fluctuates over time and the unevenness is hard to deal with.

How do you interact with requesters?

  • You can write them emails. Can write suggestions regarding qualifications, money, invitation to the forums.
    • Interpretation: There is no native method in AMT to communicate with one another.
    • Need: Turkers and Requesters need to be able to share information with one another and have open avenues of communication.

How do you choose how much to pay?

  • Try to pay a prevailing wage. Doing this for ethical reasons. Also because it’s the local minimum wage.
  • Money often paid in bonuses.
  • There is some back and forth with payment amounts. Sometimes.
    • Interpretation: Setting one’s payment isn’t easy and is fraught with problems, such as being a responsible Requester and being seen as fair. There are ways to pay more and ways to judge how much a Turker can make on a HIT, but it’s not perfect.
    • Need: Requesters need to learn how to structure their HIT payments easily and with a sense that these payments will be fair and appreciated.

Requester frustrations?

  • Can’t require someone to download something. Against Mturk terms of service.
    • Interpretation: The fear of those acting in bad faith as Requesters has limited the ways in which Requesters can utilize AMT.
    • Need: Requesters need to be able to use tools that would make the fulfilment of their tasks most effective. They need to feel trusted by the platform.

Requesters, what is the biggest mistake you’ve made in authoring a task?

  • AMT doesn’t give any learning tools for authoring HITS. Problems with measured amount of time per HIT.
    • Interpretation: AMT is a bare-bones platform that only offers a way to exchange tasks for money. It doesn’t seem to take into account the fact that there is a need to learn how to use it and use it well, by both Requesters and Turkers.
    • Need: Requesters need to be able to learn how to properly author HITs for greatest efficacy. Requesters need to feel that they are properly compensating Turkers for the amount of work they are doing by knowing metrics such as task duration.

What is your most recent frustration with AMT?

  • (As a Requester) Amazon doesn’t talk to them about anything. There is no communication regarding upgrades, how to deal with rejections and blocking Turkers, nor any method of making suggestions.
    • Interpretation: While AMT has grown as a viable platform for Requesters to get tasks completed for relatively (or even in absolute terms) little money, the usability of the platform itself hasn’t kept pace with the needs of the users. Long-time Turkers have mentioned that the UI is essentially the same as it was 10 years ago. Suggestions on how to improve AMT abound on subreddits and blogs, but Amazon’s thinking on AMT seems rather opaque.
    • Need: Requesters (and Turkers) need to be able to communicate with Amazon. They need to feel that Amazon is taking their needs seriously, as users and as customers.

Observations gathered during the March 9, 2015, 6:00PM panel.

Turker on balancing the workload

  • Turker (Didn’t catch name) talks about being invited to interviews and the great struggles in balancing work hours and communications: “sometimes it doesn’t work out well...I’m not able to talk on phone on normal hours”.
    • Interpretation: It seemed as if she was frustrated by the lack of a seamless / cohesive fit between her work online and offline, having to juggle the conflicting expectations and schedules. It seems that this comes from not enough work opportunity in one realm (online or offline) to help her stay in a single context or not having the flexibility to accommodate work or inbound communications when in a block of time allotted to a specific client, project or platform.
    • Needs: More opportunity online (or offline) to be in a single work context, buy-in or support from clients and stakeholders to support flexibility between her role, alternative channels or modes to streamline communications, greater autonomy in her work environment.
  • Niloufar - Does research on mTurk, a PhD student talking about how she goes about controlling for quality: “Get the right people for the right task instead of trying to get the task done and see if they’re a fit…[I] use qualification tasks to see if they’re a fit...If it’s a programming task, qualifications for a little bit of programming. If it’s a micro tasks, the best way is to give out the task that you have already answered and see how they do.”
    • Interpretation: She recognizes that certain tasks have qualifications to ensure quality - and its a matter of finding people who (1) meet those qualifications and (2) make sure that the tests for qualifications meet the complexity, effort, or challenge level of the task. There’s a strong sense of being reasonable.
    • Needs: Possibly a way to search for and filter workers by measured/tested expertise in skill and/or interests, guidelines and standards (or templates) for designing qualifications for tasks, recommendation system or matching system for task and workers.

Turker on quality control

  • Dahn - “There are tasks that are clearly to establish if person is paying attention, there are many questions that can be asked...attention and honesty tasks...some part of the question is a known attention check location by IP or previous work (since we’ve worked together, see if you’ll answer the same)...If you’re not going back to an anonymous pool, you can weed out who isn’t paying attention or who isn’t conscientious.”
    • Interpretation: Requesters feel that there’s a need to take it upon themselves to check for attention and honesty - this implies some inherent distrust of the platform and the worker, which possibly generates an uneasy working relationship between requester and worker. Similarly, the obviousness of an attention or honesty check may be distracting or disheartening for a worker. It seems that these tasks/checks assume no existing relationship or trust between worker and requester, and the system does not support an opportunity for the future to establish that trust.
    • Needs: Behind the scenes or more elegant systems for reputation, quality control, and relationship building so that workers and requesters are not anonymous to each other if they want to continue working together.

Workers and Requesters on the length of a HIT (time spent) vs. quality of HIT sent back

  • Requester, Niloufar - “...A mutual understanding of what the task is and communicating to the worker [is important]…finding the right people with the right skills is the second biggest challenge.”
  • oDesk Worker, Denis - “It time it takes to build up credibility and working hours...some sort of testing or tasks to complete without an employer” would help new oDesk workers get started quickly.
    • Interpretation: For requesters, in the absence of a reputation system, sheer volume of HITs completed and percentage accuracy is better than using a qualification test, which lends to the behavior of picking workers who have been on the platform more or longer rather than being the best fit for the tasks.
    • Needs: Filtering or a better measure/indicator of quality/accuracy, skills and qualifications to help match a worker to opportunities.
    • Interpretation: For workers, a similar challenge of standing out from other workers who have been on the platform longer or more than new workers is a barrier to actively working and participating on the platform.
    • Needs: An opportunity to certify and verify their skills for credibility would help them to get started on systems easier and get recognized by prospective clients/employers, also, connecting new and qualified workers to opportunities and prospective clients/employers.

Workers on incentives beyond monetary

  • MTurk Worker, SpamGirl - “No, not on mTurk. Sometimes you’ll learn or contribute to science, but there’s money always there. Not many requesters submit free tasks. Back then there was a project to remove spam from the community (something to benefit the community).”
  • MTurk Worker, Rachel - “A task that doesn’t pay is online volunteerism. One of the biggest issues is the lack of tests. I have to pass in-depth tasks first. No one knows how you get Masters and there’s no difference in the tasks sometimes. mTurk needs to set up tasks to prove you’re good at something. Gaming the system would take a long time. Re-take delay would help prevent the gaming system. No segmentation of task types for different skill areas to set up a better task.”
    • Interpretation: Workers have a financial need first that must be fulfilled on the platform. Any tasks that do not provide a reward or incentive (monetary or for the greater good) do not support their needs. Most requesters, likewise, do not submit free tasks. If there is a task to benefit the community or a cause, these tasks are not necessary distinguished or rewarded publicly. Recognition of skills and expertise is also much needed.
    • Needs: Fair compensation/wages for monetary reward, better recognition of workers to donate their time for the good of the community, better distinction between tasks/HITs based on their reward (monetary vs. recognition), better recognition and distinction of workers by skills and skill levels, better matching of workers to opportunities based on interests or expertise/skills.

Workers on variability of earning money

  • MTurk Worker, SpamGirl - “It’s about 50/50 in terms of academic support and use of the system. Time of year, day of week. The whole system is very variable.”
  • oDesk Worker, Nicole - “I review a client’s daily nutrition and give him feedback on his diet (I’m studying nutrition) by looking at his MyFitnessTracker and giving him encouragement. I also work for a company in Australia and look for research for them for articles. But yes, highly variable.”
    • Interpretation: Work is highly variable and dependent on the needs and patterns/habits of requesters. It seems that seasons and times of year impact the availability of work because of the nature of the type of tasks/work that can be posted to these platforms. There may also be a cultural aspect to this (different holidays and seasons by countries as well as being able to translate tasks between languages).
    • Needs: Possible need for more work to be made available on the platform by opening up platform to more markets, more types of tasks, and ability to translate across language and timezones.

Workers on what makes it easier to find interesting work

  • MTurk Worker, Rachel - “Better tags. Saying ‘Survey, task, fun’ is not a helpful tag. More helpful descriptions that is true to the task would be good. Categorization tasks have pretty accurate tags.”
    • Interpretation: Tasks are poorly tagged or defined. Tagging is often times the forgotten or “lazy” part of preparing digital documentation or content for others.
    • Needs: There may be a need to build a system to autotag tasks or to recommend/default a series of tasks to the requester to streamline the tagging process.

Reading Others' Insights

Worker perspective: Being a Turker

Observations about Workers


  • Being able to work from home without having to conform to the restrictions of a workplace (“sitting in my sweats at home”) can make being a Turker more enticing.
  • Finding HITs that pay well is a job as well, just unpaid. Sometimes it’s just easier to do low-paying HITs rather than “spend an hour of unpaid searching.”
  • Novice Turkers will engage in low-paying work in order to bring up approval rating and to not have to deal with the searching necessary to find higher pay.
  • Turkers are wary of federal legislation on AMT. They feel that their own collective actions are more fruitful than outside legislation would be.
  • Turkers put great value in the power of their community.
  • Turkers seem to distrust the specter of outside influences on AMT.
  • “Turkers also orient to 'positive invisibility' – the freedom from surveillance, control, and intervention in their personal affairs.”
  • There is an orientation towards fairness amongst Turkers, whether in accepting wages or judging Requesters.
  • Turker go outside of AMT for information on Requesters and how to be more successful.


  • Despite the low wages, Turkers engage in turking, by and large, for the pay.
  • Generally, if a HIT is described as “fun” it is also a HIT that pays relatively well. Though there are some exceptions where a Turker will accept lower pay for a HIT that seems enjoyable.
  • Turkers might do it as a side job to make up for pay cuts.
  • Some might be doing it to make extra money for a purchase.
  • More experienced Turkers will only take higher-paying HITs.
  • Turkers will set themselves targets.
  • Fair pay is viewed in relation to the US Federal Minimum Wage of $7.25/hr. But also in relation to what is possible on AMT. Thus, a $6/hr wage is seen as better than average, even though a such a target can perpetuate the “digital sweatshop” image of AMT. Even so, some Turkers can make ~$10/hr and will not touch jobs that fall below ~$7/hr.

Relationship with other Turkers

  • Turkers will share all kinds of information with each other.
  • Turkers will often compare themselves to others to determine earning potential and what it takes to earn more.
  • Turkers will post their grievances and problems in daily life, such as not being able to make ends meet, or problems with their full-time work outside of Turking.

Relationship with Requesters

  • A dominant idea in Turker Nation is that “ ...Turkers’ actions en masse send messages to Requesters and that Turkers are responsible for promoting fair pay.”
  • Turkers will use emotional language when talking about Requesters.
  • Rejection of work by Requesters is seen as opaque by Turkers.
  • Turkers need a high approval rating (90%+) to get higher paying, more professional HITs. This is a valuable metric for Turkers.
  • If a Requesters blocks a Turker or rejects payment, it is complicated for a Turker to prove his/her innocence.
  • Turkers are happy to discuss good HIT design with Requesters. They will work with them to design HITs that are less likely to fall prey to scammers.
  • Turkers will reference Turkopticon for reviews on Requesters.
  • Turkers value good communication from a Requester: fast, polite, informative. Lack of communication seems to foster an “adversarial tension.”


  • Turkers are concerned with being labelled as a collection of bots, scammers, etc.

Worker perspective: Turkopticon

Need: Employees need to feel human

  • AMT effectively makes crowds of workers, infrastructure, sources of computation
  • Employees want to feel like a partner not like a resource for churning out work
  • AMT has prioritized employers over workers

Need: Turks need to receive helpful information

  • Since communication between Requester and Turk isn’t always clear, a relationship where communication is open should be defined as it benefits both parties

Need: Turks need to feel confident that their work meets the criteria of the requester

  • Getting rejected is demoralizing and can get the requester in trouble
  • Exchanging email for turks is more expensive than completing the work

Need: A way to express dissatisfaction without penalization

  • Workers cannot express dissent within AMT
  • They are not heard
  • Will have to leave system if dissatisfied

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

Observations about Workers

  • There are Turkers who will game the system and submit bad work on HITs, if possible. But most Turkers will complete HITs properly.

Observations about Requesters

  • Requesters must be creative in designing HITs in order to keep Workers from gaming the system and doing bad work for quick pay.
  • Requesters must figure out how to properly design a HIT on their own. It appears Amazon provides no help.
  • Properly designing a HIT to weed out bad actors can be an iterative process.
  • Requesters will use different methods, such as task duration, or checking for verbatim text copies, to see how well HITs were completed.

Requester perspective: The Need for Standardization in Crowdsourcing

  • Industrialization meant task standardization for the mass production of physical goods - building block tasks (set prices, spread best practices, build meaningful reputation systems, track quality).
  • Current crowdsourcing is like an open bazaar, people come and go as they please, receive or make offers on tasks, different rates of pay and different pricing structures, weak reputations, disorganized but flexible.
  • How tasks are allocated, how reputation is managed, how tasks are presented are important areas for standardization.
  • Curated garden (uTest, MicroTask, CloudCrowd, LiveOps) - recruit and train workers for standardized tasks. Excerpt:
      • Every employer has to implement from scratch the “best practices” for each type of work. For example, there are multiple UI’s for labeling images, or for transcribing audio. The long term employers learn from their mistakes and fix the design problems, while newcomers have to learn the lessons of bad design the hard way.
      • Every employer needs to price its work unit without knowing the conditions of the market and this price cannot fluctuate without removing and reposting the tasks.
      • Workers need to learn the intricacies of the interface for each separate employer.
      • Workers need to adapt to the different quality requirements of each employer.
    • Reusability
      • Trading Commodities - Simple tasks can be traded, match and fulfill buy and sell orders, queue of tasks to complete at any time, don’t have to think about reputation of requester or familiarize with task.
      • True Market Pricing - Have a very liquid market, tasks get completed by workers, priority order based on price, highest paying unit gets completed first (set market price).
      • Automated market markers (prediction markets) - workers ID the price they’re willing to work for, automated market maker tasks into consideration he “ask” (quote) and the “bid” (price of the task) and bridges the difference. Fee for the bridging transaction can cover the subsidy.
    • Quality Assurance - basic work unit (screen and test), higher quality guarantee

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

  • Key problems: Scaling up
    • A marketplace needs to reduce overhead, friction, transaction costs, search costs.
  • Requesters Need: Better Interface to Post Tasks: Last updates for requesters was the UI to submit batch tasks in summer 2008. Every requester would have to build a QA system from scratch, ensure proper allocation of qualifications, learn to break tasks into a workflow, stratify workers according to quality.
  • Requesters Need: True Reputation System for Workers: Better distinction and identification of good workers and bad workers that empowers good workers to stay in the market (as good and bad workers get paid the same). A need to verify skills (qualification tests and certifications), keeping track of work history (past projects and HITs, payments, etc), rating of workers, disconnection of payment from rating (rejecting work should not be used against honest and good workers), separate of HITs and ratings by type (filtering), and better API would help support requesters and workers.
  • Workers Need: A Trustworthiness Guarantee for Requesters
    • Requesters are free to reject good work and not pay for work they get to keep. Some do not pay on time. Requesters should not be afraid of worker for a requester. Some characteristics:
    • Speed of payment
    • Rejection rate for requester
    • Appeal rate for requester
    • Disallow ability to reject work that is not spam
    • Show total volume of posted work
    • Make all the above accessible from an API
  • Workers Need: A Better Interface
    • Unpredictable completion times
    • Having a browsing system for task categories
    • Improve the search engine
    • Use a recommender system to propose HITs to workers

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

Needfinding on Reddit

Turkers utilize bots in order to find out the status of HITs

  • hit-bot - ‘HIT Info: Requester's TO Profile - Review Requester. I am a bot, and this action was performed automatically. I am designed to help determine when HITs are no longer available. Please contact the mods of this subreddit if you have any questions or concerns. Support your local programmer!...’ - Link to post

Need: With so many HITs out there, Turkers need to be able to quickly find out what HITs are still active. Works need to save time and not waste it on looking at dead HITs.

Turkers will report on the legitimacy of a Requester for the benefit of the group

  • CyanideandMadness - ‘I have one more Longitudinal follow up survey to finish from this requester, not sure if there will be follow ups for this one but if there are do them. It's 7 questions about your debt, you get a $1 each month and $10.00 for doing them all. Had some problems with the last 3 months of not paying but he fixed it(Amazon froze his account because he is Canadian and demanded a copy of passport) Just thought I'd mention in case anyone gets a follow up. He is legit :)’ - Link to post

Need: Turkers need to know if a Requester is not only willing to pay well, but also if the Requester values them as people.

There are monetary awards given to contributors to the subreddit. This money is taken from a pool created through donations from the community. Contributors are also rewarded with Gold Coin icons next to their names, which are visible on all of their posts.

  • ‘Also, if you don't feel like donating, another way you can help support /r/HITsWorthTurkingFor is by using our Amazon Associates link located in our sidebar! Some of the money for the sub's monthly rewards come from referrals to a survey site called Pollbuzzer that pays $1 for answering a single emailed question. As of late that income has been petering out a bit. If you haven't already, take a look at signing up under the community referral link to see if it interests you. Further details can be found in the FAQ... For every four polls that a referee answers, the subreddit also gets $1.’ - Link to post

Need: Turkers need to feel appreciated and to gain and sense of accomplishment. They also need to contribute to the wellbeing of others in the community.

There are Turkers who see mTurk as a better alternative to other forms of work and see the Turker community as a large part of what helps them do the work.

  • petepostlethwaite - ‘…Thanks! Without you wonderful people, I'd probably have a stupid real job! This place is a godsend.’ - Link to post

Need: Turkers need to feel comfortable in their lives, even outside of the online world of AMT and its numerous workaround sites and their collective members.

Needfinding through Planet Money's podcast The People Inside Your Machine

  • There is uncertainty in:
    • How much work
    • How much pay
    • What to expect
    • Lack of promise of future work
    • Intense competition for the work

Needfinding on TurkerNation

  • Privacy is a concern for workers and Amazon does little to support their privacy (it is easy to find public name, location, photo, product reviews, etc using worker ID) - Link to Post
  • And personal information in surveys. - Link to Post
  • Requesters might ask a worker to download or open something (or go to a site). mTurk does not provide support in ensuring the safety of workers (against malicious sites, viruses, etc) nor are there standards or protections/policies in place regarding external sites or executable software. - Link to Post

Synthesize the Needs You Found

Worker Needs

  • Workers need to be able to easily and quickly find tasks that match their skills.
    • Evidence:
    1. Monday evening panel workers cite the challenge in identifying or distinguishing tasks because of poor tagging.
    2. Reddit discussion also cited the exorbitant amount of time that they spend trying to find tasks and do the mental calculations to find the opportunities that match them best (e.g. time to complete the task on average, average $ per minute on task, requirements to complete the task).

  • Workers need to be able to verify skills (credibility and trust) quickly to compete with established workers.
    • Evidence:
    1. Monday evening panel workers from oDesk cite that most employers will not work with them until they have enough feedback or past work on the platform .
    2. Panel requesters talked about looking specifically for workers with the highest number of past HITs since they were better indicators of quality than the effort of building and using qualification tests on mTurk.

  • Workers need to feel they are being fairly compensated for their work.
    • Evidence:
    1. Panel workers talk about the distinction between paid tasks and tasks for the common good, but platforms don’t make a distinction for when they complete tasks that are online volunteerism.
    2. Reddit discussion cites that the payment system for HITs is not adaptive and does not take into account changing marketplace conditions (supply/demand) and pricing of tasks based on those changes.

  • Workers need to be able to directly communicate with requesters.
    • Evidence
    1. Reddit discussion mentioned the need to get clarification from requesters .
    2. Reddit discussion cited concerns with being blocked (for no reason) or having work rejected.

  • Workers need to have a voice and support from the platform in their interactions with requesters.
    • Evidence
    1. Reddit discussion cites a need to rate requesters and/or block and flag requesters that have a history of mass rejection, not paying, or scamming.
    2. Reddit discussion cites that the lack of a universal standard for quality makes it difficult for them to do work that meets a standard (varying standard levels across requesters) and a lot of time lost (unpaid time) spent clarifying a task.

  • Workers need to have flexibility in how they do their work in order to produce their best work.
    • Evidence
    1. Reddit discussion cites interest in being able to do HITs from their phones.
    2. Panel discussion cites the challenges with juggling various deadlines, client contexts, expectations, and modes of communication.

Requester Needs

  • Requesters need to be able to find workers based on specific, verified skills.
    • Evidence:
    1. Monday evening panel with Edwin and Niloufar discussed their need for a single worker specialized in specific type of work or relying on past workers that they’ve worked with.
    2. Monday evening panel also discussed how requesters currently determine if a worker is qualified, but primarily relied on number of HITs total (a worker who has been on the platform for a long time or often) which was not segmented or broken down into HITs of specific categories or skill areas.

  • Requesters need to have help with appropriately pricing HITs.
    • Evidence:
    1. Reddit discussion cites requesters interested in being able to adjust their HITs to a higher payout value to draw more workers or to compete with other HITs for completion.
    2. "The Need for Standardization of Crowdsourcing" article talks about the liquid market / trading commodities and the need to have a platform that serves as a broker between requesters and workers to identify a fair price for tasks.

  • Requesters need to have guidelines for designing clear HITs.
    • Evidence
    1. "The Need for Standardization of Crowdsourcing" article talks about providing design patterns for tasks and workflows.
    2. Panel discussion cites the lack of consistency across requesters in definition of quality.
    3. Reddit discussion talked about the frustrations of workers who had to spend unpaid time and effort to attempt to contact a requester to gain clarification on tasks and/or not having a sense of a universal standard on quality to set fair expectations.