WinterMilestone 2 SneakyLittleHobbitses

From crowdresearch
Revision as of 21:36, 24 January 2016 by Natasha (Talk | contribs)

Jump to: navigation, search

Group : Sneaky Little Hobbitses

Group members : Natasha Hervatta (@natashahervatta), Rajashri Venkatesh (@rajashri92)

Learn about Needfinding

On watching the video lecture by Scott Klemmer, we noted the following points :

  • What do people do now?
  • What values and goals do people have?
  • How are these particular activities embedded in a larger ecology?
  • Similarities and differences across people
  • … and other types of context, like time of day


Studying the Panel of Workers and Requesters

We watched the video on the panel discussion with Workers and Requesters. We also went through the notes uploaded to Wiki by Dcthompson and Willtchiu. We then came up with an extensive list of observations:

  • Worker uses TurkerNation’s daily threads to find HITs every day.
  • Worker works through the list, finished HITs, revisits update queue. “Rinse and Repeat” process.
  • Worker can sometimes sense patterns in HITs.
  • If worker finds a well paying HIT, they will stop the work they’re doing at that moment and do the HIT instead.
  • Worker has to prioritize. Sometimes, they have to choose between making lunch and doing a HIT and earning money.
  • Sometimes a worker will jump up in the middle of the night to work on a HIT.
  • Workers set alerts, notification systems, Snapchat groups.
  • Worker keeps TurkerNation chatroom open at all times.
  • Worker doesn’t have a fixed schedule.
  • Worker sets a daily “goal” and stops only when it is achieved (within reason).
  • Workers email and give helpful suggestions to Requesters.
  • Worker picks which kind of HIT to tackle (novice or expert level) based on how it affects their worker rating.
  • Worker NEVER risks going below 99% worker rating.
  • Worker will work for a new Requester, but only until it doesn’t affect their worker rating. At that point, they will wait for approval.
  • Worker makes sure not to “put too many eggs in one basket”.
  • If worker comes across a new Requester, they send an email even if they don’t any specific questions just to”know that there’s a human at the other end”.
  • Worker really appreciates a positive response from Requesters.
  • Worker estimates wages based on forums, participating, and experience. Another worker estimates wages based on trying a HIT for 2 mins, online communities, experience, how they determine hourly wages personally.
  • Worker’s earnings are highly unpredictable
  • Worker’s earnings depend on how much time they have and how they want to spend it
  • Workers find it v. v. difficult to budget.
  • If they are unhappy with the rejection, worker sends a screenshot to the Requester and asks for an explanation (v. politely)
  • Requester’s process for designing and iterating on HITs varies a lot.
  • Requester considers it most important to give clear and concise instructions on HITs.
  • Requester often posts a small batch as a trial to see if instructions make any sense depending on the results they get back.
  • Requester would get emails from non-native English speakers just saying “Sir, I’d like to work on your HITs”.
  • Requester employs a person to manage their email and respond on their behalf.
  • Requester makes sure that they aren’t rejecting workers unfairly and that their tasks are designed clearly.
  • Requester has created mental models of quality of work they get. The work between the bottom threshold level and reasonable work level is the area where they decide rejections.
  • Requester interacts with a group of workers (who work on their HITs) on online communities.
  • Requester doesn’t use built-in templates on MTurk.
  • Workers and Requesters see their experience with each other and AMT in vastly different lights.


Reading Others' Insights : Observations on Workers

  • Money is primary motivator for Turkers/Workers. Other factors are considered ‘side benefits’ alongside to earning money. Some found the HITs to be fun, interesting, or educational but money was more important.
  • For each worker who reported needing the money to pay for rent or groceries, there was another who did it for fun or to “kill time”.
  • Turkers make judgements about whether the rates for HITs are worth it and in what circumstances. When a job becomes available, Turkers often do an initial investigation; what does the HIT comprise, how long does it take, how quickly until I become proficient, and how much will I earn? These calculations help them to decide if it is worth it for them.
  • Turkers are always apprehensive of new Requesters. They often turn to forums for advice in this matter.
  • Workers use 2 main sorting mechanisms : See the most recent HITs, or see the HIT groups with the most HITs.
  • Turkers set themselves targets while working. For example, to make $10 per day or to double the last years’ amount.
  • Turkers discuss a number of issues on forums apart like HITs, Requesters, problems, tips and guidelines, and even share their problems and look for advice (‘Prayers and Good Vibes’ forum).
  • Direct, open, polite, and and respectful communication with Requesters is highly valued by Turkers.
  • Workers dissatisfied with a requester’s work rejection can contact the requester through AMT’s web interface. Amazon does not require requesters to respond and many do not; several requesters have noted that a thousand to one worker-to-requester ratio makes responding cost prohibitive.
  • Dissatisfied workers’ within AMT had little option other than to leave the system altogether.
  • Numerous responses said that requesters ought to respond to questions from workers, that requesters ought to justify their rejections, and that workers have the right to confront employers about those rejections.
  • If Turkers are given proper explanations, they are willing to admit fault. They are more tolerant of genuine mistakes, especially when the Requester seeks to sort them out.
  • While it is considered important for Requesters and Turkers to adhere to Amazon’s terms of service, they do not seem to want the US government to legislate and regular AMT. They are not very pleased with the idea of academia being involved either.
  • Currently, the workers are highly restricted by the current interface, in their ability to find tasks. They cannot search for a requester, unless the requester put their name in the keywords. Also workers have no way to navigate and browse through the available tasks, to find things of interest.


Reading Others' Insights : Observations on Requesters

  • An important distinction between online and offline is that once a worker is hired off an offline, traditional market, they are not allocated to tasks via a spot market.
  • With task standardization, hired workers could complete those tasks easily, predictably and in a way that training was easy to replicate for new workers. To return to paid crowdsourcing, most of the high demand crowdsourcing tasks are relatively low-skilled and require workers to closely and consistently adhere to instructions for a particular, standardized task.
  • Disadvantage : Among both buyers and sellers, one can find scammers; some buyers are simply recruiting accomplices for nefarious activities.
  • Advantage : The upside of such a disorganized market is that workers and buyers have lots of flexibility. There are good reasons for not wanting to just recreate the on-line equivalent of single-firm factory.
  • The efficiency of the market can increase tremendously if there is at least some basic standardization of the common types of (micro-)work that is being posted on online labor markets.
  • One helpful way to think about the role and incentives of online labor platforms is to consider that they are analogous to a commerce-promoting government in a traditional labor market.
  • Task standardization will probably require buy-in from on online labor markets and intermediaries. * Setting cross-platform standards is likely to be a contentious process, as the introduction of standards gives different incentives to different firms, depending upon their business model and market share.
  • This is a positive development, particularly because paid crowdsourcing gives people in poor countries access to buyers in rich countries, enabling a kind of virtual migration.


Reading Others' Insights : General observations

  • AMT design prioritizes the needs of employers.
  • Once a worker submits work, the employer can choose whether to pay for it. This discretion allows employers to reject work that does not meet their needs, but also enables wage theft. Because AMT’s participation agreement grants employers full intellectual property rights over submissions regardless of rejection, workers have no legal recourse against employers who reject work and then go on to use it.
  • Because AMT treats workers interchangeably and because workers are so numerous (tens of thousands by the most conservative estimates), AMT can sustain the loss of workers who do not accept the system’s terms.
  • The Turkopticon system allows workers to make their relationships with employers visible and call those employers to account.
  • Turkopticon is not an expression of our own values, or even the values of the users we interviewed, but a compromise between those values and the weight of the existing infrastructural norms that torqued our design decisions as we intervened in this powerful, working real world system. This attention comes not only in the crowdsourcing community, but also in broader public fora.
  • A major task of a marketplace is to reduce overhead, friction, transaction costs, and search costs. The faster and easier it is to transact, the better the market. And MTurk fails miserably on that aspect.
  • The current reputation system on MTurk is simply bad. “Number of complete HITs” and “approval rate” are easy to game.
  • Early user input can substantially improve the interaction design, and input after development can provide important feedback for continued improvement.
  • The diversity and unknown nature of the Mechanical Turk user base is both a benefit and a drawback. Since many users are sampled with a pool drawn from all over the globe, results found using the Mechanical Turk population have the potential to generalize to a varied population more than the small user samples and limited geographic diversity typical of more traditional recruiting methods.
  • The tasks and the summarization activity of keyword tagging raise the cost of generating non-obvious malicious responses to at least as high as producing good-faith responses.
  • Hundreds of users can be recruited for highly interactive tasks for marginal costs within a timeframe of days or even minutes. However, special care must be taken in the design of the task, especially for user measurements that are subjective or qualitative.


Synthesizing the needs

A set of bullet points summarizing the needs of workers and requesters.

Worker Needs

  • Workers need a steady source of income as their livelihood depends on it. Evidence (From the panel) : 1) If worker finds a well paying HIT, they will stop the work they’re doing at that moment and do the HIT instead. Worker has to prioritize. 2) Sometimes, they have to choose between making lunch and doing a HIT and earning money. 3) Sometimes a worker will jump up in the middle of the night to work on a HIT. Interpretation : For some workers working on AMT is more important than daily chores or other work, because they rely on it heavily.
  • Workers need better clarity on HITs. Evidence : Section on ‘Standardizing basic work units’ in The Need for Standardization in Crowdsourcing Paper. Interpretation : SWU can help to get a predictable outcome.
  • Workers need to establish a good relationship with the Requesters. Evidence (From the panel) : Workers email and give helpful suggestions to Requesters. ) Interpretation : Workers are always keen to help out Requesters and prevent the risk of Rejections.
  • Workers need to be able to trust the Requesters. Evidence (From the panel) : 1) Worker picks which kind of HIT to tackle (novice or expert level) based on how it affects their worker rating. 2) Worker NEVER risks going below 99% worker rating. 3) Worker will work for a new Requester, but only until it doesn’t affect their worker rating. At that point, they will wait for approval. Interpretation : Workers are apprehensive about picking HITs from new Requesters as it has a direct impact on their Worker’s Ratings.
  • Workers need an organized system to decide if the HITs are worth their time. Evidence (From the panel) : 1) Worker estimates wages based on forums, participating, and experience. 2) Another worker estimates wages based on trying a HIT for 2 mins, online communities, experience, how they determine hourly wages personally. Interpretation : There’s no guaranteed source of information on how much wages they will make.
  • Workers need to know what they’re doing wrong so that they don’t make the same mistakes again. They also need to make sure that they weren’t rejected unfairly. Evidence (From the panel) : If they are unhappy with the rejection, worker sends a screenshot to the Requester and asks for an explanation (v. politely). Interpretation : Getting a rejection directly affects the Worker’s ratings.


Requester Needs

  • Requesters need to maintain a good reputation. Evidence (From the Panel) : 1) Requester makes sure that they aren’t rejecting workers unfairly and that their tasks are designed clearly. 2) Requester has created mental models of quality of work they get. The work between the bottom threshold level and reasonable work level is the area where they decide rejections. Interpretation : The Requester’s reputation (on online forums) depends on the number of unfair rejections they make, so they try not to make harsh decisions.
  • Requesters need to establish a good relationship with the Workers. Evidence (From the panel) : Requester interacts with a group of workers (who work on their HITs) on online communities. Interpretation : Requester wants to know and understand their Workers better