WinterMilestone 2 SneakyLittleHobbitses
Group : Sneaky Little Hobbitses
Group members : Natasha Hervatta (@natashahervatta), Rajashri Venkatesh (@rajashri92)
Learn about Needfinding
On watching the video lecture by Scott Klemmer, we noted the following points :
- What do people do now?
- What values and goals do people have?
- How are these particular activities embedded in a larger ecology?
- Similarities and differences across people
- … and other types of context, like time of day
Studying the Panel of Workers and Requesters
We watched the video on the panel discussion with Workers and Requesters. We also went through the notes uploaded to Wiki by Dcthompson and Willtchiu. We then came up with the following Observations, Interpretations, and Needs.
- Observation / Interpretations / Needs
- Observation / Interpretations / Needs
Reading Others' Insights : Observations on Workers
- Money is primary motivator for Turkers/Workers. Other factors are considered ‘side benefits’ alongside to earning money. Some found the HITs to be fun, interesting, or educational but money was more important.
- For each worker who reported needing the money to pay for rent or groceries, there was another who did it for fun or to “kill time”.
- Turkers make judgements about whether the rates for HITs are worth it and in what circumstances. When a job becomes available, Turkers often do an initial investigation; what does the HIT comprise, how long does it take, how quickly until I become proficient, and how much will I earn? These calculations help them to decide if it is worth it for them.
- Turkers are always apprehensive of new Requesters. They often turn to forums for advice in this matter.
- Workers use 2 main sorting mechanisms : See the most recent HITs, or see the HIT groups with the most HITs.
- Turkers set themselves targets while working. For example, to make $10 per day or to double the last years’ amount.
- Turkers discuss a number of issues on forums apart like HITs, Requesters, problems, tips and guidelines, and even share their problems and look for advice (‘Prayers and Good Vibes’ forum).
- Direct, open, polite, and and respectful communication with Requesters is highly valued by Turkers.
- Workers dissatisfied with a requester’s work rejection can contact the requester through AMT’s web interface. Amazon does not require requesters to respond and many do not; several requesters have noted that a thousand to one worker-to-requester ratio makes responding cost prohibitive.
- Dissatisfied workers’ within AMT had little option other than to leave the system altogether.
- Numerous responses said that requesters ought to respond to questions from workers, that requesters ought to justify their rejections, and that workers have the right to confront employers about those rejections.
- If Turkers are given proper explanations, they are willing to admit fault. They are more tolerant of genuine mistakes, especially when the Requester seeks to sort them out.
- While it is considered important for Requesters and Turkers to adhere to Amazon’s terms of service, they do not seem to want the US government to legislate and regular AMT. They are not very pleased with the idea of academia being involved either.
- Currently, the workers are highly restricted by the current interface, in their ability to find tasks. They cannot search for a requester, unless the requester put their name in the keywords. Also workers have no way to navigate and browse through the available tasks, to find things of interest.
Reading Others' Insights : Observations on Requesters
- An important distinction between online and offline is that once a worker is hired off an offline, traditional market, they are not allocated to tasks via a spot market.
- With task standardization, hired workers could complete those tasks easily, predictably and in a way that training was easy to replicate for new workers. To return to paid crowdsourcing, most of the high demand crowdsourcing tasks are relatively low-skilled and require workers to closely and consistently adhere to instructions for a particular, standardized task.
- Disadvantage : Among both buyers and sellers, one can find scammers; some buyers are simply recruiting accomplices for nefarious activities.
- Advantage : The upside of such a disorganized market is that workers and buyers have lots of flexibility. There are good reasons for not wanting to just recreate the on-line equivalent of single-firm factory.
- The efficiency of the market can increase tremendously if there is at least some basic standardization of the common types of (micro-)work that is being posted on online labor markets.
- One helpful way to think about the role and incentives of online labor platforms is to consider that they are analogous to a commerce-promoting government in a traditional labor market.
- Task standardization will probably require buy-in from on online labor markets and intermediaries. * Setting cross-platform standards is likely to be a contentious process, as the introduction of standards gives different incentives to different firms, depending upon their business model and market share.
- This is a positive development, particularly because paid crowdsourcing gives people in poor countries access to buyers in rich countries, enabling a kind of virtual migration.
Reading Others' Insights : General observations
- AMT design prioritizes the needs of employers.
- Once a worker submits work, the employer can choose whether to pay for it. This discretion allows employers to reject work that does not meet their needs, but also enables wage theft. Because AMT’s participation agreement grants employers full intellectual property rights over submissions regardless of rejection, workers have no legal recourse against employers who reject work and then go on to use it.
- Because AMT treats workers interchangeably and because workers are so numerous (tens of thousands by the most conservative estimates), AMT can sustain the loss of workers who do not accept the system’s terms.
- The Turkopticon system allows workers to make their relationships with employers visible and call those employers to account.
- Turkopticon is not an expression of our own values, or even the values of the users we interviewed, but a compromise between those values and the weight of the existing infrastructural norms that torqued our design decisions as we intervened in this powerful, working real world system. This attention comes not only in the crowdsourcing community, but also in broader public fora.
- A major task of a marketplace is to reduce overhead, friction, transaction costs, and search costs. The faster and easier it is to transact, the better the market. And MTurk fails miserably on that aspect.
- The current reputation system on MTurk is simply bad. “Number of complete HITs” and “approval rate” are easy to game.
Synthesizing the needs
A set of bullet points summarizing the needs of workers.
- Example: Workers need to be respected by their employers. Evidence: Sanjay said in the worker panel that he wrote an angry email to a requester who mass-rejected his work. Interpretation: this wasn't actually about the money; it was about the disregard for Sanjay's work ethic.
A set of bullet points summarizing the needs of requesters.
- Example: requesters need to trust the results they get from workers. Evidence: In this thread on Reddit (linked), a requester is struggling to know which results to use and which ones to reject or re-post for more data. Interpretation: it's actually quite difficult for requesters to know whether 1) a worker tried hard but the question was unclear or very difficult or an edge case, or 2) a worker wasn't really putting in a best effort.