- 1 Worker perspective: Being a Turker
- 2 Requestor perspective: Being a Requestor
- 3 Worker perspective: Turkopticon
- 4 Requester perspective: Crowdsourcing User Studies with Mechanical Turk
- 5 Requester perspective: The Need for Standardization in Crowdsourcing
- 6 Both perspectives: A Plea to Amazon: Fix Mechanical Turk
- 7 Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc
- 8 Synthesize the Needs You Found
Worker perspective: Being a Turker
While turking might just be a fun pastime and a quick way to earn some bucks,many turkers are doing AMT as a means of living ‘hand-to-mouth’. Money does not seem to be the only reason for participation. Some find it fun, interesting and even educational. The joy of giving back to the community is a strong driving force for the turkers.The most matured workers would aim at not only high paying jobs but also interesting jobs. However it is required that workers must often decide between a job of their satisfaction and an appropriate pay. There also seems to exist a trade-off between HIT counts and approval rating. Novice workers seem to be more focussed on increasing their HIT count while experienced seem to be concerned about the approval rating as it is bi-directional. A better filtering algorithm must be implemented so that Turkers may do work in the subject of their interest.Workers would also prefer if the requestors were to respond quickly to them and engage in communication in forums etc.
Requestor perspective: Being a Requestor
Requests are often posted where there is a huge difference between the complexity of the task versus the pay offered for it. This often is the cause of dissatisfaction among the workers as they have to pick between a task of their liking or one which pays well. Requestors often have huge expectations of the results for workers. However to be fair to the workers they must be motivated by a reasonable pay to express interest in it
Worker perspective: Turkopticon
Requester perspective: Crowdsourcing User Studies with Mechanical Turk
1. Observations about workers a. Natural human tendency to go for the shortcut Most workers would want to make the most money spending the least effort and as such, they will be on the hunt for ways to complete tasks like user studies, which can be subjective, in an arbitrary manner without any real effort put in. Thus, when micro-tasks add the element of subjectivity, the risk of “nonsensical answers” is higher.
b. Pooling in of diverse human experience to match expert opinion When presented with a more structured way of capturing the abilities of a diverse group into doing a job similar to the admins, the workers are able to produce good quality work. This illustrates the potential of the doing an “expert” task through large less-qualified group of individuals, through their experiences.
2. Observations about requesters a. Need for careful division of tasks and framing of questions Given the much better response from experiment 2, which had lesser invalid responses and better closeness to admin ratings, it is clear that poorly divided tasks that can elicit arbitrary responses leading to severe degrading of quality. Yet, a better understanding of the workers’ perspective and attitude, can lead to a major improvement in the responses. It also shows a limitation that the requestor needs to spend more effort.
b. Design of tasks to draw “true” responses We can observe that when the questions for the user studies were restructured such that the effort required to complete the task ‘in good faith’ and that to do it arbitrarily is similar, which would act as a motivation for workers to do it rightfully. This can indeed be a tricky task.
Requester perspective: The Need for Standardization in Crowdsourcing
1. Observations about workers a. The modelling of the prices for a task like the stock market prices adds a different dimension to the perspective of the worker, who is able to define the tasks in terms of the pay he thinks he deserves. This creates a possible motivation for the worker for better quality work yet can also lead to lesser efficiency due to the introduction of liquidity in the prices.
b. Workers are not held accountable in a formal way like suing for unsatisfactory task completion
c. The idea of an online “curated garden” can be a good addition to any crowdsourcing platform, which helps it to adopt some of the pros of the regular work done in formal offices.
2. Observations about requesters
a. Allows flexibility to post different categories of tasks that is acceptable to them based on pricing and conditions of work. The impact of reputation can be avoided is required.
b. The pricing of the tasks may not be in concordance with those expected by the workers due to reservations about the quality that they may get in return. The varying price model can prove troublesome if not structured in an effective and transparent way.
Both perspectives: A Plea to Amazon: Fix Mechanical Turk
1. Observations about workers
a. Trustworthiness guarantee for requesters The current state of workers on MTurk, in which the author has good experience in, is said to be quite bad due to issues of high rejection rates, uncertainty about payments and lacking information about the tasks. This has led to some alienation of the workers from the requestors. The current state of the ‘reputation’ used has several flaws that do not help the issue of rejections being high for spurious requestors
b. A better UI The matching of tasks with workers is inadequate in terms of matching their interests and priorities. This is critical to finding the right group of workers, who want to do the work, to complete the tasks. The process also needs to be made easier.
2. Observations about requesters
a. A better UI to post tasks The requestors need a platform that helps them to distribute the work in the most effective way. The platform needs to understand the workers’ perspective so that it guides the requestors to frame tasks in a way that can be more efficacious.
b. A improved reputation system for workers
a. The current reputation system uses number of HITs completed and approval rates, to judge which has led to a poorer differentiation between the “good” and “bad” quality of work. Through more qualification tests, working history tracking and classification of the HITs, the system can be improved
List out the observations you made while doing your fieldwork. Links to examples (posts / threads) would be extremely helpful.
Synthesize the Needs You Found
List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.
A set of bullet points summarizing the needs of workers.
- Example: Workers need to be respected by their employers. Evidence: Sanjay said in the worker panel that he wrote an angry email to a requester who mass-rejected his work. Interpretation: this wasn't actually about the money; it was about the disregard for Sanjay's work ethic.
A set of bullet points summarizing the needs of requesters.
- Example: requesters need to trust the results they get from workers. Evidence: In this thread on Reddit (linked), a requester is struggling to know which results to use and which ones to reject or re-post for more data. Interpretation: it's actually quite difficult for requesters to know whether 1) a worker tried hard but the question was unclear or very difficult or an edge case, or 2) a worker wasn't really putting in a best effort.