WinterMilestone 2 duka

From crowdresearch
Revision as of 18:09, 24 January 2016 by Lorettale (Talk | contribs)

Jump to: navigation, search

Attend a Panel to Hear from Workers and Requesters


Report on some of the observations you gathered during the panel.

Reading Others' Insights

Worker perspective: Being a Turker

Explaining the significance of the relationship between worker and requester. Comparing and contrasting each individual’s behaviors and how it can affect future performances.

Worker Observations:

  • we focus on their behavior with requesters- emotionally and ethically
  • value positive experience with a requester to avoid rejection
  • value their experience and perspective
  • prefer to have fun, unique tasks even if pay maybe low
  • goal-getters
  • have their own interests and goals
  • some turkers work in their free time. they turk as a supplement to their regular job
  • AMT is their labour marketplace
  • Main motivation is money with some secondary experiences that affect the behaviour in selecting and executing tasks.
  • It's hard to make money in AMT
  • Worker/Requester relationship goes beyond doing HITs and getting payed
  • Workers use an external platform to communicate between themselves
  • Doubts about trusting on requesters and how to select HITs are frequent

Requester Observations: - Amazon - has close working relationship with Turkers and AMT - need reputation - some shady and negative people out there - do not give fair pay or do not even pay at all - can be rude and give unfair ratings to turkers, therefore affecting their future work - unjustified act of blocking others

Worker perspective: Turkopticon

Observations: Activist system that allows workers to publicize and evaluate their relationships with employers. It enables workers to engage one another in mutual aid. -Allows workers to create and use reviews of employers when choosing employers on AMT, maintaining the communicating system with workers.

Employer defines criteria that candidate workers must meet to work - including approval rating, self reported country, whether the worker has certain specific skill. It is a work based community that allows people to work together to get the best results with ethical behavior. -It is used upon a daily thread and this allows them to share work access

They now introduced: -moderators to engage in debates and movements in worker forums -an option for workers to take on screen names : identifying themselves

Advantages: -Brings people together around practical and shared concerns -Mutual aid -Social interchange and interdependency -People don’t have to be experts so people have to understand how to make things work

Disadvantages: -Minimum wage -Payments are slow -Amazon has little reason to prioritize workers needs in a market -Too much competition -Fair compensation -Lots of rejections; downgrade people too fast -Workers feel like their work is rejected too often -Dissatisfaction with employers -Competition increases -Conflict between workers/ requesters - Lack of understanding of each other -Chats aren’t consistent -Time consuming -Lack of proof on who did the work


Positives: The dynamic society of Turkopticon allows people to unite and work together. This allows them to learn easily from each other. People from different background of knowledge are able to come together and do the similar tasks. It is enjoyable and people are able to have easy access to it. Creates a sense of a community because of interdependence

Negatives: People get affected when their work gets rejected. It also harms their reputation and it is hard to get chosen when their work has been rejected. This does not give everyone a chance to show their talent. There are a lot of people wanting to do the same job and a lot of people who have the same expertise. Hence, decreasing the possibilities of jobs for others. The pays are very slow because Amazon has little priorities to workers even when the workers do not earn enough for a standard living. There is miscommunications between the worker and requester. This could be because both parties have different ideas but are not able to express those ideas in language, especially when it is a language that is not their mother tongue. It is difficult to know how much time one task will take. Sometimes it takes too long or too short and because of this people are enabled to do other tasks. Since it is a 24/7 job and competition is high, people would take that opportunity and forget about the other things that they have planned during the day.

Needs: Communicator Generosity Fairness Promptness Greater visibility to press, policy makers, organisers Labour’s concerns Identify hazards of crowd work Prioritize workers

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

Micro-task Markets: A system where practitioners can access large user pools and collect data quickly, while users are compensated with micro-payments by completing small tasks.

Observations about workers:

-Some workers do not care about completing tasks effectively, main objective is to make money quickly -Some workers “game” the system -Try to make more money by providing nonsense answers to complete tasks quickly -Diverse group of workers from all around the globe -Small group of workers “game” the system, majority of workers do not -Don’t spend much time on tasks, usually can complete them within minutes

Observations about requesters -Must design tasks effectively to prevent workers from “gaming” the system -Requesters who spend more time to make an effective task include explicit verifiable questions -Requesters try to encourage workers that completing task accurately will require the same or less work as mindlessly completing task -Try to find multiple ways to detect random and careless responses

Advantages: -Collects data quickly -Ability to complete simple tasks that computers can’t -Large diversity of users -Anyone connected to the internet can participate

Disadvantages -Lack of demographic information and unknown expertise of users -Finding, removing, and rejecting bad responses takes extra time =No control over experimental setting -Can reduce the amount of workers who “game”, but can not prevent

Requester perspective: The Need for Standardization in Crowdsourcing

Observations about workers: Workers are able to have great flexibility and trade any kind of labor on their own terms. Usually the work that is being done are low-skilled and workers need to follow close instructions. Workers are able to come and go for their tasks. There is no reputation that you need to uphold or any real work hierarchy.

Observations about requesters: Requesters also have great flexibility in their work. They are able to propose, make, and accept any offer for the task given. Requesters also do not have a reputation to uphold on the online market.


More standardization practices such as a curated garden approach where a firm recruits and train workers for standardized tasks while settling the prices for both sides of the markets where it may not be “free” but scalability and cost of savings for crowdsourcing is ensured. Evidence: Ipeirotis (2012) discusses how this will help both parties to have more meaningful, informative feedback and if rewards are available then it will change people’s perspectives to be more motivated in doing their work. Interpretation: Having a standardization practice will enable people to want to do the best they can so they can get a reward otherwise they will simply do another task elsewhere.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

Observation about workers:

There is an assumption that workers are either bad or good. The workers that are good at their job are getting paid too little so they leave the certain task leaving only the “bad” workers to stay. Workers are cautioned by new requesters, they only do a couple to see if the requester how the requester behaves such as if they are legitimate and pays on time. Workers also usually use priority queues to pick the tasks they want to work on.

Need: To ensure the quality of workers, Ipeirotis (2010) talks about having more public qualification tests (writing skills, proofreading), keep track of working history, allow rating of workers, and rejecting work to only spammers. Interpretation: This will help both requesters and workers to see who is fit for the job and who aren’t. Workers also need an easier way to find possible work because currently they pick the work by priority queues. Ipeirotis (2010) recommends implementing a browsing system under task categories, improving the search engine, and a recommender system where MT will propose similar HITs to workers. Interpretation: The solutions presented will be an easier way for workers and making everyone happy.

Observation about requesters:

Mostly small requesters for tiny tasks so there is no room to grow with Mechanical turk encourages people to built their own interface but requesters do not have time or the resources to build their own from scratch. Requesters are free to reject good work and not pay for the work they keep. They do not have to pay on time. New requesters may not know understand that workers are hesitant to work on their batches so if a new requester post a large batch, it may be disastrous.

Soylent: A Word Processor with a Crowd Inside

The combination of AI and crowdsourcing. Explaining the Soylent system, the breakdown of the features, connection to the workers, and the amount of pay.

Advantages: - recognize and organize information, usernames, files, etc - the “Shortn” button to revise and shorten papers - Crowdproof is used to locate grammar errors with explanations and recommendations to reword the text - Human Macro is an extension that allows quick work and feedback to be done in “human language”, therefore allowing better relationship between computer and human - can identify patterns - proofreading algorithm beats microsoft word’s - allows users to have feedback directly from other users/humans - potential for other creative domains in the image editing and programming field

Disadvantages: - does not understand specific organization methods to match the requesters’ preference - Crowdproof costs money to use - the requester will not know who is working behind Soylent - no credibility/reputation behind the work

Worker Observations: - workers work behind the scenes of soylent, revising anything necessary as per request of the requester

  • The “Shortn” button
  • assists the requester in shortening their paper to avoid mistakes, repetition,

- the lazy turker vs. eager beaver

  • lazy turkers, as the title implies, give very few, even pointless feedback
  • eager beaver may not always be a good thing, for they may create more work for the user

- worker performance is dependent on the pay and task

  • opinions seems to have a higher percentage of accuracy and intention as well as decent pay

Synthesize the Needs You Found

Worker Needs

  • Workers need to get paid for the work they've done. Evidence: larak56 said "(...) While I do find some of the HITS fun and actually learn an incredible amount by doing HITS, I do it for the cash." Interpretation: Even though the main motivation for a worker is money, there are some secondary experiences affecting the behaviour in selecting and executing tasks.
  • Workers need to feel safe when performing transactions. Evidence: Foldergirl asks about a requester's reputation: "Does anyone know if he is a good requester to work for? He has 1 review on TO. (...)". Interpretation: Workers may avoid new requesters because they are not sure if they will get paid or not. It's hard for them to foresee how their relationship with requesters will happen.
  • Workers need to be treated fairly. Evidence: A user from AMT stated "I would also like workers to have more of a say around here, so that they can not easily be taken advantage of, and are treated fairly, as they should be. (...)". Interpretation: Workers fell like they are not taken into account and that the platform gives preference to requesters. When their work is rejected, they demand a proper explanation. If the rejection is fair, they will generally admit fault.

Requester Needs

  • Requesters need to have tasks done. Evidence: The existence of AMT itself shows the need of requesters to use actual people to do tasks that computers are not capable yet.
  • Requesters need to trust the quality of tasks submitted. Evidence: Crowdsourcing User Studies With Mechanical Turk mentions: "The results from Experiment 1 provided only weak support for the utility of Mechanical Turk as a user measurement tool. Furthermore, they demonstrated the susceptibility of the system to malicious user behavior.". Interpretation: It's time consuming for requesters to validate workers submissions and, without validation, they feel unsure about the quality of the work.
  • Requesters need to get assistance on creating tasks to increase the chances of high quality submissions and to improve their relationship with workers. Evidence: modgirl said "Great requester, honest, good communicator, cares about what is best for us as much as is best for him. Highly recommended. You’ve proven to me that you are a good requester because you’ve asked here for help –you have my respect.". Interpretation: It's easy to make a mistake as a requester when creating tasks because they do it from their own point of view, without considering others.


Sally Chung @sallyxchung, Loretta Le @lorettale, Diane Phan @diane, Kimberly Le @kimberly35, Renan Castro @renan, Samarth Sandeep @samarthsandeep