Milestone 2 PixelPerfect

From crowdresearch
Jump to: navigation, search

Team Pixel Perfect's submission for Milestone 2.

Attend a Panel to Hear from Workers and Requesters

Observations gathered from the Panel

The Panels were highly informative and gave us an insight into the MTurk community as well as other Crowdsourcing platforms like oDesk. We summarize our team's observations here :-

Observations from Worker Experiences

  • Finding work on MTurk is not easy. Veteran Turkers use multiple tools (scripts/scrapers) to find HIT's.
  • First-time turkers face hurdles as there are no tutorials for AMT as such, the interface doesn't really help and thus many people tend to give up even before they get started.
  • Money primarily motivates people to Turk, some do it part-time for a partial financial support and for others, their main source of income.
  • The workers at AMT have a strong bonded community, they help each other finding HIT's, helping new turkers get up to speed etc.
  • The communication between requesters and workers takes place on forums in general.
  1. There seems to be a perception among workers that requesters should read up on the forums.
  2. Some requesters expect communication regarding their HIT's to be done through email only, as they don't have the time to go through forum posts regularly.
  • Times at which new HIT's are uploaded is variable, the general trend is that the top earners generally turk at odd hours, also the pay is highly variable, one may make more than expected in one week, and then much less in subsequent weeks.
  • It has been observed that much of the HIT's on Mechanical Turk are academic/research oriented, the amount of work available during summer/spring break or holiday season is drastically lower than that during normal days.
  • For the people outside the United States, other platforms are available like oDesk, CrowdFlower but again work there is close to zero.
  • There is a lot of competition for work on oDesk.
  • A feature on oDesk which is absent on Mechanical Turk is that once someone develops a good profile on oDesk work seems to find them instead of them trying to find work.

Observations from Requester Experiences

  • Generally, requesters pay workers atleast the minimum wage or the prevailing wage in the area for their HIT's , it can be more depending upon the quality of work required by the requesters.
  • The requirement of a Social Security Number for registering on Mechanical Turk is due to tax payment regulations imposed by the IRS in the United States.
  • Requesters try to make their HIT's as short and quick as possible, as the quality of the work decreases proportionally with the length of the HIT. BubbleHell is a term prevalent in the MTurk community referring to long tasks.
  • Some task which requesters have problems deploying on the AMT platform are
  1. Tasks whose completion require some software . According to the terms and conditions of Mechanical Turk ,software cannot be uploaded on the platform as a security measure against malware ,but this practice seems to hurt the genuine community more than it hurts the spammers.
  2. Deployment of tasks which are personalized for workers ,for example tasks which depend on the relevant details of the worker.
  • Requesters don't have the guarantee that their HIT will be completed before some specified deadline.
  • Another problem that the requesters face is weeding out the frauds doing their tasks . Requesters generally apply various techniques like embedding timers, putting open-ended question in surveys to segregate the spammers from the genuine workers.
  • Requesters generally never reject the workers' work, even if it's not upto the mark, this is due to the broken rejection system in Mechanical Turk, even one rejection can highly decrease the rating of the worker.
When faced with rejection, workers either ask to requester to overturn the rejection or sometimes even slander the requesters on forums which might affect their Turkopticon rating.

Reading Others' Insights

Worker perspective: Being a Turker

Turking is the new Laissez-faire economy controlled by the collective individual actions of requesters and workers. Crowdsourcing work sounds easier and cheaper for a requester but what encourages workers to do it is explained and analysed in depth in this paper. Different aspects of being a worker are explored - needs, demands and their communications with each other and the requesters reveal the true nature of crowdsourcing.

Observations on workers

  1. Some workers agree to working besides the low wage because the tasks are fun and have scope for learning.
  2. Workers enjoy their anonymity, the flexible working hours and the freedom to choose work.
  3. In many instances, workers accept their fault and make it their responsibility that their mistakes don't reflect upon the requesters.
  4. A strong and supporting "union" of workers look out for each other, speak up against fake, unprofessional requesters and help each other understand the ways of turking so that the community can expand.
  5. Turking is perceived as a free labor market by most workers and government involvement is dreaded.
  6. No minimum wage per hour makes workers vulnerable to exploitation.
  7. Workers who turk full time have no "job security" as such.
  8. Beginners and novice workers find it difficult to establish their credibility.
  9. Being a global phenomenon, wages that maybe considered low in one country might be more than sufficient in another. Thus, increasing the competition.
  10. Lack of an official "Bill of Rights" for workers.

Observations on requesters

  1. No proper review and rating system for requesters.
  2. Requesters often have the upper hand over turkers. They have better and more information about the turkers than the turkers, requesters.
  3. Requesters don't hesitate to exploit their powers in certain cases and there is no system to stop them.
  4. Some Requesters do try to communicate with the workers and change for the better on the worker's suggestions.
  5. Requesters do benefit from the lack of proper "official" structure.
  6. Genuinely good Requesters with proper HITs stand out amongst the crowd.
  7. Communication between requesters and turkers is a very important aspect of crowdsourcing which should be encouraged.
  8. Lack of enough information on requesters is a major concern which hampers growing success of the crowdsourcing forums.

Worker perspective: Turkopticon

The paper describes the development of Turkopticon, an activist system designed for intervention into the AMT crowdsourcing network. Its main goals are to address worker invisibility, improve and publicize worker-requester relations and act as a platform for mutual aid. The design process involved conducting surveys on AMT for a “Workers’ Bill of Rights”, challenging the ethics of human computation. The implementation required a "critical mass" of workers to contribute reviews, moderators for comments and reviews and providing a at-a-glance system for workers to assess employers. Turkopticon owes its success to keeping a vigilant watch on employers, holding them accountable and inducing good behavior. This helps repair AMT into a more transparent marketplace.

Observations on workers

  1. Workers actively respond to a "Worker's Bill of Rights" when brought to their attention by an external source.
  2. A significant minority of workers rely on their income from the AMT platform to pay for household expenses.
  3. Legal definition of workers are contractors subject to laws designed for freelancers and consultants. Framing attempts to strip workers of minimum wage requirements in their own countries. Workers, even in the US, are paid below minimum wage in many cases.
  4. Workers have no legal recourse against employers who reject their work, and then use it. Thus they do not enjoy intellectual property rights over their own work.
  5. Dissatisfied workers have little option other than to leave the system altogether.
  6. Due to huge worker supply and interchangeability, AMT can sustain the loss of workers.
  7. Workers are divided in how they evaluate employers and the privacy they wish to enjoy.
  8. Turkopticon has now become a staple tool for AMT workers. Supports a thriving collective of workers engaged in mutual aid, brought together by browser extension and web technology.

Observations on requesters

  1. Requesters enjoy the right to deny wages to a worker if work done is deemed unsatisfactory. However, this may be abused for "work theft".
  2. Requesters do not usually respond to worker's emails, adding to the worker-requester barrier.
  3. There are high incentives for requesters to write good reviews on Turkopticon for themselves, and indeed they engage in such practices.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

The given paper evaluates the use of Amazon Mechanical Turk as a platform for conducting user studies, which tend to be technical in nature. It was demonstrated via two experiments that, technical evaluation can be done by low-cost crowdsourcing, in spite of variation in expertise and demographics. Carefully designed HITs with task-relevant, verifiable questions can help deliver results which approximate expert judgement.

Observations on workers

  1. Every task attracts malicious workers who try to "game" the system and provide nonsensical answers to decrease time spent and increase pay. Non constructive and uninformative responses originate as a result.
  2. Many invalid responses to tasks are due to a small minority of users who try to take advantage of the system multiple times.
  3. Inspite of large variations in demographics, expertise and evaluation criteria among crowdworkers, it is possible to approximate expert panel judgement. In cases of collecting many varied data points like user measurement or prototype testing, crowds may do a better job than experts.

Observation on requesters

  1. Requesters often have to engage in trade-offs between sample size, time requirements and monetary costs. Economics of collecting user input is important to them.
  2. Requesters trust fewer expert users with complex tasks, whose demographic details and expertise levels are known. They also prefer more experimental contact. These are absent in case of a crowdworker scenario.
  3. Badly designed tasks will cause gamification of the system and invite non-constructive answers. Such answers consume requester resources in their scrutiny and removal.

Requester perspective: The Need for Standardization

This blog post basically deals with how standardization of labor can benefit all the participants in an online labor market: the employer, the worker and the market owner. The problem with the existent online labor markets is the need for more coordinated, organized operations. There are differences in quality of work, wages, uncoordinated work-flow (in cases of complex tasks) and cases of fraud. Standardization would improve if not revolutionize the online market system.

Observations on workers

  1. There is a lot of untapped potential in the worker crowd. Even technically savvy workers have difficulty finding relevant jobs with decent pays, and if the online labor market is used long enough, only then the workers gain sufficient experience which boosts their efficiency. Adapting to different interfaces and requirements takes time.
  2. If we use ideas from the prediction market systems, workers may demand for wages higher than their actual expectations because automated market systems bridge the high ask of the workers and the low bid of the requesters, leading to a wage lower than the one demanded. The trend is vice-versa for requesters.
  3. Workers would prefer if the standards of the work to be done is specified. This improves competition and motivation, since the requesters would offer higher pay for better quality and less time taken to complete the tasks.
  4. This would also lead to price fixation: a constant price for a set of almost-identical tasks. This could be beneficial to the general worker crowd especially in times of high demand for labor, which would lead to a significant wage rise.

Observations on requesters

  1. Requesters find it very taxing to specify the expected quality of work each time from their workers. The flexibility that a disorganized online labor market system offers is beneficial in the sense that the task can be personalized according to the tastes of the requester. But most of the times, it is much easier to standardize the quality and price of labor.
  2. It is not easy to manage complex tasks consisting of smaller tasks with a definite workflow. The pricing fluctuates and is unpredictable, the time taken to finish is high, and the quality is not consistent, so requesters would rather hire from a physical labor market where communication and coordination would not hinder the job completion.
  3. Standardization would involve the market owner to impose certain quality standards that would reduce low-quality entries. It could lead to higher costs incurred by the requesters but a guarantee of quality could possibly be worth it.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

This blog post talks about how Amazon Mechanical Turk has not been seeing any major improvements for many years now despite having identified the problems years ago. The problems are faced by both the workers and the requesters. These problems include:

  • Lack of a good interface to post tasks: The requesters who have to accomplish a task of high complexity have to create their own interface and track the work flow.
  • Lack of a fair reputation system for workers.
  • No guarantee of trustworthiness of the requesters before working for them.
  • Lack of a good interface for the workers.

Observations on workers

  1. Lack of trust due to unreliable reputation/rating system of the workers can lead to dilution of quality because efficient workers would go elsewhere for better wages than work in a place where efficiency does not yield extra wages.
  2. Lack of a guarantee of trustworthiness of the requesters also makes the workers more skeptical about working for them. Workers would find it more relieving if they knew when the payment would be released, the rejection rate, the volume of posted work etc.
  3. Workers find it very difficult to find tasks as they can sort and see either the most recent HITS or the HIT groups with most HITS. They work in a priority queue, so the popular tasks just receive more and more demand before workers find themselves with no tasks again.

Observations on requesters

  1. The requesters who are serious about the tasks and have a knowledge of the command line (or hire someone who has the knowledge) are the ones who post big tasks which are immediately attended to. The smaller requesters cannot grow in such conditions. A standardized, default user interface should be used in AMT to ease the process of posting tasks and keeping a track of the workflow.
  2. The execution of a task takes way more time than initially estimated and this is a major drawback of AMT. This happens due to inefficiency. The work history, the type of work successfully done, money made etc., if made available to the requesters, the process could be hastened due to increased efficiency. Building trust by familiarizing oneself with the workers before hiring helps a lot.
  3. The problem mentioned above about how workers find it difficult to look for new tasks also affects the requesters whose tasks get subdued and they remain unaccomplished and the time to accomplish it would be indefinite.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

“Needfinding” broken into simpler terms, is an art of discovering needs - either explicitly stated, or hidden beneath the surface. As a part of Milestone 2, we were allotted the work of going through various blogs and forums for various crowdsourcing platforms. Based on the thorough readings of these resources here are some of the observations that our team made.

Observations on Worker needs :

Workers who have worked on MTurk for a while know, it is very important to keep track of HITs. But as a matter of fact there is no such functionality on MTurk which would help workers to sort this matter out. As a result most of the workers rely on 3rd party tools by trading their precious time. MTurk doesn’t offer suggestions based on the previous contributions. Job searching could be made easier for workers by improving tracking and filtering options.
Sufficient amount needs to be paid for quality output. Why would any worker invest his/her precious time to HIT which is underpaid? Workers basically evaluate the minimum wage based on the time required for the task. On an average it is about $0.10 for a minute. (Stats based on Thread here).

Observations on Requester needs :

The biggest problem that requesters face is “How to attract workers?”. To overcome the above stated problem the requester needs to “Make HITs available”. Workers use search to target HITs and rarely review beyond the first page of results. So it’s critical to select descriptive titles, keywords and rewards making it easier for Workers to discover the HIT. The other aspect on which the requester needs to work upon is “How to make HIT attractive”. What person sees in the very first glance makes a big difference, so it's important to make your HIT attractive while at the same time providing a title and description that accurately describe your task. At the first place exaggeration may attract workers but it would eventually backfire by tarnishing requesters reputation. Offering competitive pricing based on the length of time and complexity of the HIT directly affects the quality of the output.

Useful Blogs and Threads :

http://mturkforum.com/showthread.php?28581-Why-nobody-answers-our-hit-to-get-0-25D-Could-you-help-us-to-figure-it-out
http://www.mturkgrind.com/threads/how-to-improve-my-hit-and-attract-more-workers.27461/

Synthesize the Needs You Found

Worker Needs

Evidence Needs
Observations Interpretations
Workers realise the importance of a "Worker's Bill of Rights" when brought to their attention by an external source. Workers lack the technical skill to develop a mutual aid platform on their own. They cannot help themseleves in this respect. Workers need to have a platform to protect their rights and recognize their interests.
Workers are divided in how they evaluate employers and the privacy they wish to enjoy. Cultural and demographic diversity of workers pose different requirements and expectations. Workers need to have methods to evaluate employers and comfortably exert their free speech.
Workers have no legal recourse against employers who reject their work, and then use it. Thus they do not enjoy intellectual property rights over their own work. Workers need to be respected and their work valued more.
Finding work on AMT is tough,Veteran Turkers like SpamGirl and Manish use various tools/scripts to find work.
As David discussed in the morning panel for first-time Turkers it almost impossible to find work. The non-user-friendly interface is an obstruction in itself when one starts turking,
and one needs to have various tools to make reasonable money . Workers need to get Job suggestions according to their skills or previous HIT's.
"iambob" on Turker Nation was unjustly blocked by a Requester. This is about what the workers deserve. Workers should be given certain rights and they should be guarded.

Requester Needs

Evidence Needs
Observations Interpretations
Requesters often have to engage in trade-offs between sample size, time requirements and monetary costs. Economics of collecting user input is important to them. Requesters need to look for low-cost, high-volume sources for user input.
Requesters trust fewer expert users with complex tasks, whose demographic details and expertise levels are known. They also prefer more experimental contact. These are absent in case of a crowdworker scenario. Researchers value technical work done by a supervised and trusted body. Requesters need to understand the potential of crowdsourcing and trust them better.
Badly designed tasks will cause gamification of the system and invite non-constructive answers. Such answers consume requester resources in their scrutiny and removal. The temporal cost of submitting a good and bad interface should be approximately the same. Requesters need to design their tasks to discourage uninformative answers.
Serge , Gordon and Dahn as requesters generally don't tend to reject other worker's work , even if it's not up to the mark. Workers go against requesters who reject their work. Requesters feel it's not worth the effort to deal with the worker. The rejection system in Amazon Mechanical Turk is broken.
 Even one rejection can cause a drastic change on the rating of a worker ,which is very strict. Requesters need to have a better rejection system.
Serge and Gordon tend to put open-ended questions ,embed timers in their tasks to identify those who didn't read the instructions properly. The requesters have to come up with ways to stop spammers from interfering with their work. The AMT interface provides no help in this matter. Requesters need to let the platform handle the burden of weeding out the frauds , instead of doing it themselves.
Due to the terms and conditions of AMT,Serge feels requesters can't upload additional software to help complete tasks.Edwin talked about the assignment of a separate task to each worker if personalization is needed. Requesters can't ask a pool of workers to do their HIT. The security reasons (malware) due to which uploads are not possible, seldom stop the spammers from uploading it,but the work of the genuine requesters gets hampered. Amazon Mechanical Turk does not provide enough options for requesters to customize their tasks. Requesters need to choose the workers to do their jobs,give personalized tasks,upload resources to help workers complete tasks etc, on a flexible platform
As was discussed in the Evening Panel ,since the requesters almost never reject the workers's work, workers with a high rating don't always guarantee high quality work. Requesters still need to go through each workers' work. The requesters have to perform the screening of the responses themselves to filter the responses good enough for their work. Requesters need to trust a screening mechanism provided by the platform to filter workers best suited for their HIT.