Milestone 2 cder
- Naman Gupta
- Aditi Mithal
- 1 Attend a Panel to Hear from Workers and Requesters
- 2 Reading Others' Insights
- 3 Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc
- 4 Synthesize the Needs You Found
Attend a Panel to Hear from Workers and Requesters
Report on some of the observations you gathered during the panel.
Reading Others' Insights
Worker perspective: Being a Turker
1. The workers work on AMT for extra money, or as a sole earning job. 2. The workers might favor AMT for the following; No travelling costs, No fixed working hours, No restriction on the method you chose to complete the desired task. 3. Sometimes, the workers loses hope on AMT because of the fear of being not getting the job on the first hand while being rejected /failure (but that actually happens in real life too) never to return on AMT ever. 4.Most of the times, notion arises that the workers do the job for fun rather than the money, but our understanding is that is not the case. Money is indeed a big motivating factor for a worker. 5. A very few of them actually completes tasks to gain knowledge/information or learn new things. They are just satisfied with the money they earn.
Worker perspective: Turkopticon
1. The relationship between a worker and a requester is unfair as a requester has more power to reject and discard the work/effort of a worker anytime. 2. Turkopticon is a system that was created to allow workers to publicize and evaluate their relationships with employers. 3. Workers want the requester to be based on communicability, generosity fairness and promptness. 4. Time and effort of justifying every rejection is generally more than the cost invested for the HIT's. 5. The communication between the worker and a requester is inefficient.
Requester perspective: Crowdsourcing User Studies with Mechanical Turk
1. Most of the requesters are fair/genuine in giving and assigning the tasks to the workers. But a few of them also scams people in doing the required job without paying them the money they deserve. 2. The quality standards set by the requesters are purely subjective. 3. A requester in AMT has to have a fair experience on how to utilize the platform to the fullest, ie. get the work done as efficiently as possible while ensuring the quality of work.
Requester perspective: The Need for Standardization in Crowdsourcing
In the crowd labor market scams are inevitable. The current situation of the Crowdsourcing market is chaotic and unordered. Each requester generates a work request, adds a payment unit and evaluates the HITs as per his needs. The need is to implement "best practices" for every work which includes fixing the design of the UI and the complete workflow. This is to be done to ensure uniformity and equality as a new user need not have to spend a lot of time just to familiarize with the platforms. Each requester needs to price its work unit without being aware of the market conditions. Workers need to adapt to different quality requirements of the requeters and learn the interfaces used by the requesters. Workers have a freedom to complete the tasks till any time they want to without having to think about the requester's reputation. The scope of Turker platforms helps in virtual migration since the crowdlabor market helps in outsourcing the work to remote workers. A requester can also access an inexpensive labor force due to this as well as there is scope for new ideas spread across the globe. This scenario develops more positions, demand etc and thereby ensures a level playing field.
Both perspectives: A Plea to Amazon: Fix Mechanical Turk
1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.
MTurk does not guarantee trustworthiness of requesters and they are free to reject good work and may even refuse to pay.When a new requester comes he has to maintain some reputation and earn trust of the workers. The worker analyses about any new requester by checking if the requester is a scammer or not and if the requester pays on time. This observation is done in order to avoid trust issues in future. The workers generally avoid newcomer requesters who post big batch of HITs in order to avoid mass rejection so it's better for the newcomer requester to post smaller batch HITs to prove his/her legibility.
A worker's behaviour on seeing a new requester is as follows: The worker checks the objective characteristics of a requester including : a) Speed of payment made by the requester. b) Rejection rate of the requester based on reports. c)Appeal rate for requesters. d) Disallowing ability to reject work which is not a spam : rejecting a work by a requester should be done only when it's considered a spam. e) API to make the above things accessible.
Workers need a better User Interface: To gain efficiency, minimize the cost involved in a task and reduce the transaction overhead a good UI is important indirectly.
Functionalities a worker needs on MTurk in order to make his/her search for the requester and a particular category of a job easier can be provided by adding some features(good UI). Some valuess adeed points for a good UI can be : a) Having a browsing system with tasks posted based on categories. b) Improving the search engine c) Using a system to recommend HITs. Everything depends on the requester-worker relation. One needs to trust the other to establish a conducive working environment.
2) What observations about requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.
A requester faces a lot of problems in the beginning like: a) Scaling up. b) Managing the API and execution time. c)Ensuring Quality
Requesters need a better Interface that would help them post HIT in an effective manner. The command-line used earlier proved to be non-ser friendly as it increased complexity so the UI was of a core concern. It was of utmost importance for requesters to a)build a system that ensures quality b) to ensure proper allocation of qualifications and c) ensure proper breakage of tasks into a workflow.
Requesters also need a good Worker review system and reputation system to analyse each worker. Someone quoted : "A market without a reputation mechanism turns quickly into a market for lemons." It's important for requesters to be aware of the quality of workers to assign and accept HITs accordingy. To have this system: a) Qualification tests are conducted. b) Working history of workers is recorded. c) Workers are rated on their performance. d)HITs and ratings are separated on different types/categories . e) Providing payment based on rating should be discontinued.
The question arose was that why not have a two-sided reputation/rating system? The answer given was indeed true that it's not much needed and it might also create many mutual-admiration schemes. So, this idea of having two-sided rating was rejected.
List out the observations you made while doing your fieldwork. Links to examples (posts / threads) would be extremely helpful.
Many interesting observations were made regarding the needs of the workers and requesters :
MTurk forums are really great for observation, they provide good insights into the workers and requesters need. The workers and requesters get a platform to interact on the forums which proves beneficial.
a) Workers need an MTurk functionality which would help them subscribe to preferred requesters, categorize HITs ,get notifications for the same and be able to search the site to find a specific requester.
b) Workers need to be patient and develop trust in terms of payment transactions.
c) Workers need to trust the requesters.
d)Workers need to trust the requesters and the HITs posted by them.
e) Workers need to have a a better user interface and need to learn the intricacies of the interface for each requester.
2) Requester Needs:
a) Requester needs to set the 'Time Allotted' limit for HITs to an amount of time much longer than the expected amount of time.
b) Requesters need to be clear about bonuses and be interactive with the Workers.
c) A requester needs to use a consent/intro page or paragraphs for his/her HIT request.
d) A requester needs to implement a custom qualification to grade workers and assign complexity to the HIT.
e)Requesters need to implement the “best practices” for each type of work. For example, the designing part)
f) Requesters need to understand the “Lifetime Approval Rate”
g) Requesters need to understand rejections and 'hardblocks.'
h) Every Requester needs to price its work unit without knowing the conditions of the market .
Synthesize the Needs You Found
List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.
1. Example: Workers need an MTurk functionality which would help them subscribe to preferred requesters and get notifications for the same. Interpretation: this will increase the efficient on the worker’s side (efforts) as they get to choose whom they wish to work for and It will also save time . Evidence:Many people upvoted for this suggestion on the link provided in the previous section.
2. Workers need an MTurk feature to enable searching the website to find out a specific requester or the hit count. Interpretation:This feature would save time and effort, it would help a worker choose a specific kind of task and help him keep a record of his/her HITS. Evidence:Many people upvoted for this suggestion on the link provided in the previous section.
3. Workers need to be able to separate hits based on different categories . Interpretation: Category based HITs help a worker work efficiently and maximize his output. Evidence:In many forums it was a suggestion offered by many workers.
4. Workers need to be patient and develop trust in terms of payment transactions. Interpretation: The lack of trust arises due to the payment suspension of many Mturk workers. Since MTurk is quite a big and reputed crowd-sourcing platform it would definitely deal with this payment error and rectify it ASAP. Evidence: In this thread(http://www.reddit.com/r/mturk/comments/2yj04z/the_account_has_been_unsuspended_so_that_you_may/ ) the requester is having payment issues since his account has been suspended.
5. Workers need to adapt to the requirements of each requester and handle the intricacies. Interpretation: For ease of gaining maximum HITs the worker needs to comply with the working criteria and standards or the requester. Evidence: Since we know that experienced MTurks are likely to earn better wages on Mturk than the inexperienced , it highlights the fact that those people are aware of how to interact with their each requester and are comfortable working in their environment.
6. Workers need to trust the requesters. Interpretation : Once the trust between a requester- worker develops it will be easy to work and get paid off for hard work done. Evidence: http://www.behind-the-enemy-lines.com/2010/10/plea-to-amazon-fix-mechanical-turk.html
A set of bullet points summarizing the needs of requesters.
1. Requester needs to set the 'Time Allotted' limit for HITs to an amount of time much longer than the expected amount of time. Interpretation: Time duration provided is an important factor in determining the number of workers one wants to get within a specific interval of time. It attracts more workers and the requester earns points. Evidence:http://www.mturkgrind.com/threads/lifetime-approval-rate-misunderstood-by-requesters.27458/ With MTurk support the requester s were able to resolve the problems caused to the workers due to rejection.
2. Requesters need to understand the “Lifetime Approval Rate” Interpretation:When the workers made a HIT for this requester, and it wasn’t approved for some duration , the requester claimed it to be "0% (0/0)" which was much less than the targeted HIT rate margin (90%) set by the requester. Evidence: http://www.mturkgrind.com/threads/lifetime-approval-rate-misunderstood-by-requesters.27458/ With MTurk support the requester s were able to resolve the problems caused to the workers due to rejection.
3. Requesters need to be clear about bonuses. Interpretation: If a bonus is offered,the potential amount (or range of possible amounts) and how to earn it, and how soon workers should expect it to be paid must be mentioned .
4. A requester needs to implement a custom qualification to grade workers and assign complexity to the HIT. Interpretation: This avoids time wastage as unnecessary extra HITS don’t need to be done.Plus, one can avoid mass rejections which can kill a requester’s reputation.
5. Requesters need to implement the “best practices” for each type of work. For example, the designing part) Interpretation: To attract more workers the designing part chosen by the requester should be well enough and legible.
6. Every Requester needs to price its work unit without knowing the conditions of the market . Interpretation: In order to get the work done the price limit to fetch workers needs to be set.
7. Requesters need to understand rejections and 'hardblocks.' Interpretation: Requesters reject a worker as they never intended to pay. Or even when the requester set up the hit incorrectly. The workers are even hardblocked without a reason and are sent an Amazon letter stating that another attempt may lead to jeopardy, nothing much can be done about it. Evidence:http://www.mturkgrind.com/threads/guideline-for-academic-requesters-on-mturk.26327/ (master’s reply)