WinterMilestone 2 spontaneous
We worked on reading more about needfinding and crowdresearch, going through some of the optional recommended readings.
- 1 Attend a Panel to Hear from Workers and Requesters
- 2 Reading Others' Insights
- 2.1 Worker perspective: Being a Turker
- 2.2 Worker perspective: Turkopticon
- 2.3 Requester perspective: Crowdsourcing User Studies with Mechanical Turk
- 2.4 Requester perspective: The Need for Standardization in Crowdsourcing
- 2.5 Both perspectives: A Plea to Amazon: Fix Mechanical Turk
- 2.6 Soylent: A Word Processor with a Crowd Inside
- 2.7 Recommended Readings (Optional)
- 3 Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc
- 4 Synthesize the Needs You Found
- 5 Milestone Contributors
Attend a Panel to Hear from Workers and Requesters
Here is a report on some of the observations we gathered during the panel:
- During the panel discussion, it was apparent that the requesters and the workers share a selfless interest in the upliftment of the community.
- The workers and requesters are both in search of a platform that addresses their needs in a better direction.
- Both the workers and the requesters need to weed out dishonest people who are there to merely game the system.
- The workers and the requesters need a platform where they can interact and instil a sense of trust and confidence in each other's jobs.
- A better platform can help both the concerned parties in better service and consensus on monetary deals.
- The support that community members provide to each other instil confidence to work dedicatedly even after facing a rejection.
- The requesters who post tasks that are realistically entertaining have a higher probability of getting completed.
- The workers need to browse through the performance of requesters before deciding to work for them. The requesters also need to make sure that they catch fraudsters.
- The workers first need to understand how they can use their real work skills in crowdsourcing efforts.
- The requesters and the workers should be provided with appropriate training materials.
Reading Others' Insights
Worker perspective: Being a Turker
- Majority of workers are US Based, although recently Indian based workers' percentage has increased but social desirability is still prevalent.
- 80% of tasks are done by 20% of active workers
- Different workers are motivated to work for different reasons; not necessarily monetary benefits (though money remains the primary factor), some value interesting tasks more than higher pay.
- Workers' complaints include slow and low payment, scams, requesters' rejection of work while reaping benefits of said work.
- No method to rate requester i.e. imbalance of power whereas requesters can rate workers which can cause privacy issues.
- Turk Nation allows workers to discuss requesters and collect experiences to allow them to identify the reliable requesters.
- Allows engaging of requesters with workers.
- Many workers use AMT to earn money on the side as a safety net or for extra cash and sometimes set monetary targets.
- The flexibility i.e. no set or regular hours attracts workers.
- Workers accept responsibility for their mistakes.
- Workers can be quick to judge based on one bad experience.
- Articles on AMT and interference by journalists worries workers since it may harm their source of income through AMT.
- Workers believe they can influence AMT through their reactions and don't appreciate any outside influences.
- They want sufficient information, opportunity and choice.
- Requesters tend to 'mass reject' and write demeaning comments.
- Good requesters are fair and approve payments on time and take care about their interaction with requesters being professionally correct.
- Approval rating of a worker can be severely damaged by an unfair requester, if a worker is unlucky to meet several of such requesters, his account can be permanently banned by AMT.
- Requesters also interact on Turker Nation to understand workers' point of view and feedback.
- Some requesters take feedback positively, and improve on those points, which attracts more workers to work on their tasks.
- Workers may not be fair and objective towards requesters.
Worker perspective: Turkopticon
- Workers are dependent on AMT for wages while some may just do it to kill time/for fun, but pay tends to be below minimum wages.
- Workers have no legal rights over their work due to AMT's intellectual property rights regardless of rejection.
- Workers can contact requester if unsatisfied by work rejected by requester but requesters are not liable to respond.
- Common issues workers point out include:
- Work rejected unfairly
- Slow payment
- Minimum wage enforcement
- Lack of fair compensation
- No response to feedback on requesters and AMT site
- Solutions suggested by workers themselves to above mentioned issues include:
- A forum for discussion and feedback without any consequences.
- Build long term work relationships with requesters.
- Turkopticon allows workers to rate and review a requester by his unique AMT id by collecting quantitative ratings.
- Requesters choose workers based on approval ratings.
- Requesters have the power to approve or reject payment for a task.
- Requesters are unable to spend time communicating with workers due to time constraints.
- Requester can flag reviews on Turkopticon they feel are incorrect.
Requester perspective: Crowdsourcing User Studies with Mechanical Turk
- Worker pool is a diverse population: which can be detrimental or advantageous for the requester based on the task assigned by the requester.
- Some workers try to cheat the system and don't produce quality work: some traits include short time duration and same comments on tasks
- Workers have varied quality of responses and malicious workers were more responsive in the first experiment.
- Requesters need to be careful in identifying malicious users trying to game the system.
- Requesters compare workers' responses to expert groups.
- Requesters receive large number of responses for relatively low costs.
- Requesters can get user responses in a rapid speed.
Requester perspective: The Need for Standardization in Crowdsourcing
- Workers tend to work on similar tasks but payments for these might have a wide difference
- Workers have to adapt according to a requester
- Standardising simple tasks helps requesters in reusing the basic interface for future tasks, determining pricing and market value.
- A requester has to set a price without knowing the market rate which can not be changed later.
- Workers are not evaluated on a regular basis and requesters do not trust the quality of work.
Both perspectives: A Plea to Amazon: Fix Mechanical Turk
A Trustworthiness Guarantee for Requesters
- Work can be rejected even though work is kept by the requester and withheld payment
- Legitimate(experienced) workers tend to avoid new requesters in the market due to unreliability.
- Good workers would be more likely to spend time on new requesters with small tasks to test their reliability
A Better User Interface
- Sorting mechanism is faulty with limited filters .
- No ability to search for a requester or particular task according to their preferences or interests respectively.
A Better Interface to Post Tasks
- Requesters are responsible for their own quality assurance and defining the workflow.
- It takes a lot of effort for the requesters to ensure high quality results from workers.
- MTurk encourages building of the requester's own interface and workflow systems from scratch which may mean hiring a person just to design tasks by the requester
A True Reputation System for Workers
- No reputation system for workers hence unreliable and no assurance of high quality results.
- Requesters assume all workers are bad.
- Requesters tend to get the same task completed by many workers to ensure accurate results.
- Requesters seem to have all the power: they can get the work done and reject it later which rejects the payment for the worker
Soylent: A Word Processor with a Crowd Inside
- Workers summarise section of text which the user requests. Several workers work on the same task for the user to pick up the best result.
- Crowdproof: (Crowdsource Proofreading):
- Workers perform tasks that computers currently can not and require human intelligence like correcting grammar
- The Human Macro: (Natural Language Crowdscripting):
- The natural language command interface allows the user of Soylent to specify a task for a worker in human language, but may choose to use that work or not
- Workers are largely involved in all sections of the task and spend a great deal of time and effort to produce required results
- Lazy Turker and Eager Beaver workers may provide highly different solutions to the same task highlighting the variance in the delivery of a task by different workers
- The variance and human involvement of workers is an invitation for a high degree of error in results.
- Workers are better at giving suggestions for a task than actually completing said task.
- User(requester) can accept or reject a task completed by worker
- Requesting for either of Shortn, Crowdproof or Human Macro can cost additional charges for the user.
- The likelihood of receiving inaccurate results even after expenditure is substantial.
- For using Shortn, requester is able to shortn many times iteratively according to the degree of summarisation required
Recommended Readings (Optional)
You can read about our summary of recommended readings here.
We browsed through Reddit for observations and concluded the following:
- Workers want to maximise pay with least amount of time spent as observed here.
- Workers are trying to find shortcuts to reduce time spent by using scripts like hit scraper, greasyfork, turkmaster and hitmonitor as observed here.
- New workers do not get any guide or help as to how to work on tasks like audio transcriptions as observed here.
- Requesters cannot go back on rejections/acceptance if they make a mistake as observed here.
- Workers get frusturated with the amount of time spent but earning very little as observed here.
- Workers face the negative impact of a requester's carelessness in designing tasks and when their work is rejected by the same also get affected by a low rating as observed here.
- Some requesters are diligent and nitpick work done by the workers as observed here.
Synthesize the Needs You Found
- Workers expect a fair wage for the work they complete.
- Evidence: No written contracts, non-disclosure agreements, or employee agreements are made with workers.
- Interpretation: Workers want a minimum wage protocol to be followed.
- Workers expect a good explanation to the HITs that are rejected.
- Evidence: Workers email requesters about the work.
- Interpretation: Workers expect their work to be reviewed properly.
- Workers need a complaint mechanism against requesters who take their work without paying. They wish to read the reviews of the requester before working for him.
- Evidence: The development of Turkopticon.
- Interpretation: It is not easy to find trustworthy requesters.
- Turking cannot be done in isolation.
- Evidence: It is observed that turkers often interact with one another in external forums.
- Interpretation: Workers often require help and they don't hesitate to offer help to other workers.
- Experienced workers avoid new requesters in the market due to unreliability.
- Evidence: In one of the papers, it was noted that sophisticated Turkers are aware that surveys that are completed too quickly stand a good chance of being rejected. Hence, they advise each other to hold back on survey submission.
- Interpretation: Workers are in search of effective and efficient ways to select HITs that value their efforts.
A set of bullet points summarizing the needs of requesters.
- Requesters need to keep the workers engaged.
- Evidence: It was observed that Turkers’ engagement via entertaining and realistic situations increased the probability of completing long tasks.
- Interpretation: Requesters need to make sure that workers don't get bored after a while.
- Recruiting the right type and number of participants is a common challenge.
- Evidence: In research studies, the researchers need to filter the submissions to the demographics they are concerned with.
- Interpretation: Platforms need to be more flexible to allow the requester to control the participation.
- Requesters cannot always trust the responses to their tasks.
- Evidence: Some people "game" the system which affects the response quality.
- Interpretation: Platforms need to make sustainable such that they address these type of concerns of the requester.
- Requesters need technical support for posting tasks.
- Evidence: In one of the papers it was noted that even if the researcher has access to programmers to assist in the implementation of the study, some understanding of the framework's capabilities and constraints is required to determine how to best adapt the study to it.
- Interpretation: The people who maintain the platform should help the requesters in addressing their technical posting needs.
Slack usernames of all who helped create this wiki page submission: @diksha and @amitoj