Milestone 2 Opera
- 1 Observations from Workers and Requesters Panel Discussion
- 2 Reading Others' Insights
- 2.1 Worker perspective: Being a Turker
- 2.2 Worker perspective: Turkopticon
- 2.3 Requester perspective: Crowdsourcing User Studies with Mechanical Turk
- 2.4 Requester perspective: The Need for Standardization in Crowdsourcing
- 2.5 Both perspectives: A Plea to Amazon: Fix Mechanical Turk
- 3 Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc
- 4 Synthesize the Needs You Found
Observations from Workers and Requesters Panel Discussion
Following are our observation after watching panel discussion
- People start off for money and then they form social groups since there are some frustrations around working with mTurk
- Workers also help the requesters during the unpaid time
- There is a large time spent on unpaid tasks like understanding the requirement
- Hard to find work due to the interface, requires a lot of digging and working with scripts, mostly for novices
- Account selling is possible, people creating account in US and selling it to India
- People come for only for money not for other non-monetary things.
- There is no support system/forum/customer care in built in the MTurk, workers/requesters have to rely on external forums like turk-nation, reddit for support
- MTurk is created for ease of Requester and not for workers
- Difficult to verify that worker did the job seriously, so there are some open ended questions asked by requesters
- Tries to make HIT as quick as possible so people don’t get bored and give up and the quality decreases
- It costs lesser ( specially surveys) than doing a campus research and hence they come to MTurk
- Might not reject tasks which are short timed because it will take more time to reject than to design task and verify the results.
- Tasks for research purposes are restricted to US.
- Quality has increased post 2012
- Requesters don’t filter workers based on country
- Task to be done by specific worker is complicated in terms of assigning the work to a specific person
- Requesters keep timer on the page, but they want to know the average time for screening threshold
- While evaluating completed HITs, previous number of HITs done and approval rate is sufficient to approve a worker.
- Requester if give high incentives people are more likely to cheat on that HIT to get more of it easily before they exhaust it out.
Reading Others' Insights
Worker perspective: Being a Turker
- 80% of the tasks are carried out by 20% of the active turkers.
- Majority of the turkers are US based.
- Most turkers work for money, it’s a safety net for some living on the edge and also has some perks like one can work from home in any desired time, no need to get ready for office / how you appear or speak and anonymity.
- Some have to suffer unfair rejection of work
- Some have experienced slow payments
- Turkers work for low wages
- Cannot communicate sufficiently with requesters/AMT
- With current pure AMT platform Turkers cannot rate requesters, but requesters can rate turkers
- Some turkers are also employed in other firms that gives them a salary
- Internet is a basic necessity for turkers
- Most turkers don’t make even the minimum wage as per the federal minimum
- It’s not clear how many hours turkers spend on mTurk
- There is a lot of invisible work that needs to be done in order to do efficient work on mTurk
- A wage might be low in US but high in other countries like India
- Turkers can tend to rate a Requester in haste both good/bad rating
- Turkers can give conflicting opinions about the requesters
- Turkers tend to indulge in cheating by asking how to pass qualifications
- Some turkers do find fault with themselves when the task gets rejected
- Turkers do not welcome legislation or laws for forming rules for AMT, they believe they can regulate it themselves
- Turkers believe that working for fun is a misnomer as most do work for money and such a message should not be propagated since it encourages requesters to pay lesser
- Novice workers are okay with low pay as they want to get their ratings improving
- Primary concern of turkers is to find good Requesters and avoid bad ones
- Turkers are understandably offended when Requesters reject HIT submissions for reasons they do not understand.
- Substantial amount of time is spent gathering information through Turking; Information such as good and bad Requesters, pay rates per hour, how best to search, how best to monitor for good HITs being posted, etc
- These implicit observations are drawn from the explicit observation #1. Since only 20% of the turkers are active, it means that a vast majority of the work force is not satisfied with the platform or does not have sufficient time to devote. It can also mean that the ones who are active stay active, they know the tips and tricks and fast enough to pick the tasks, perhaps not allowing others to get a chance. It could also mean that 20% of turkers really do depend on mTurk for money and pursue it very seriously than the rest.
- These are drawn for explicit observation #2. Since micro tasks needs to be submitted as quickly as possible, the large number of US citizenship on AMT is due to fact that requesters and workers are on the same time-zone and have more chances of work being available and getting completed faster hence. It also means that the availability of resources such as a computer and internet is more in US (at the time of writing the paper) which makes it more feasible for larger US population to join on AMT.
- These are drawn from explicit observation #4. Assuming that the requester is fair, the rejection can happen either due to fact that the problem was not clearly explained/understood and hence the result was not as expected, or due to fact that the worker has really submitted the wrong output and has failed to see his error. This means there is lack of prosaic and quick communication channels ( also stated in explicit observation #7 ). Additionally it could mean that the requester is not available most of the times to communicate. Assuming that the worker has submitted the right output, it means that requester is exploiting the worker.
- These are drawn from explicit observation #5. The slow payment can cause potential delays in bill payment etc. So overtime, the worker will either want to move to a better platform or she might need to withdraw rewards earlier, which means more work to do in shorter time, to increase the chances of getting some money earlier.
- These are drawn from explicit observation #6. The worker normally cannot depend entirely on mTurk for a living for a long period of time, which means sooner or later she will want to find other options to earn money.
- These are drawn from explicit observation #7. Lot of Turkers time is spent in waiting for clarifications and it can impact the time he actually gets to spend on the task. This leads to frustration and loss of time and money.
- These are drawn from explicit observation #8. It gives a sense of unfairness and lack of equality. It can breed a sense of suppression and curtailment of right to express. A worker may end up feeling disrespected.
- These are drawn from explicit observation #9. A lot of time and money needed for things like commute, office clothing etc is not required. This is good in case the turker is not well off and it trying to save money. Moreover, people who are differently abled can still make money in the comforts of their home. Same goes for mothers who find it difficult to go out to work.
- These are drawn from explicit observation #10. Of all the bills, the turker might be most particular of this one. Some of them might work on mTurk in office hours to avoid cost of internet at home.
- Workers might deliberately give bad ratings to a requester so that no one else gets to work with him other than the worker himself
- From explicit observation #22: Rejection directly impact there wages/rating and its more difficult to revive since rating decide what will be assigned to you in future.
- From explicit observation #23: Lot of time spent on additional activities which is not directly related to earning money.
Worker perspective: Turkopticon
- Lost or delayed income to workers due to rejected work or slow payment
- Accidental download of Malware that damaged their computers
- Once rating goes down, it becomes even more difficult to gain them as the work becomes further scarce
- Frustration as it’s not mandatory for requesters/AMT to justify rejections
- Workers expressed need for anonymity while submitting reviews
- Workers need a feedback mechanism to give ratings to the requester
- When workers publicly show trust and great working relationship with a requester, it is possible that other new worker might be discouraged to work with the requester as they might think that the requester already has a set of favorites.
- Workers may choose to not trust or even look at the ratings in case of dire circumstances
- Worker rating is one-way feedback and non-negotiable, hence turkers are hesitant to take work from the new requesters since they dont know that whether this requester is fair one and will not do mass rejection, which will impacting their rating.
- Workers may be afraid that their negative review impact worker-requester relationship hence they need anonymity. It also implies that, if system allows identity based review system then it might not reflect genuine review since workers dont want to harm their relationship with requsters.
- Some rouge workers might register themselves as requester and spread Malware putting others workers at risk.
Requester perspective: Crowdsourcing User Studies with Mechanical Turk
Explicit requester observations
- Requesters use mTurk to rapidly collect user input at low cost
- Requesters find it difficult to evaluate submissions when the questions are subjective like opinions or surveys
- Requesters find it difficult to distinguish between a bot and human turker
- Requesters can put pre-tests or block users from working on HIT
- The survey tasks which require improvement suggestions from the workers as an optional input ,did not have good suggestions for most in the submitted task
Implicit requester observations
- A new requester will need to do a sufficient study/survey before posting on mTurk any HITS in order to follow the best practices
- Often some tasks may have invalid result, so mTurk does not guarantee a perfect labor
Explicit worker observations
- Some can do task for reputation rather than money
- Participation cost is low. However, it's not true when worker need to put extra effort to understand/find/decide the HIT - in this case the actual HIT cost might be lesser than that of the time spent
- Some tasks got finished within minutes of arrival
Implicit worker observations
- Some turkers are very alert and might always steal the show, from explicit worker observation #3
- Since time is a constraint , it is not possible for a turker to give detailed review even if they want to, from explicit requester observation #5
Requester perspective: The Need for Standardization in Crowdsourcing
Observations about requesters
- Standardization is required because some common tasks like image tagging, audio scribbing are done differently by different employers.
- Few advantages of standardization are common UI, pricing, quality expectation etc.
- Drawing analogues to mechanics like screws, levers etc which are standardized, similarly a list of tasks need to be choosen and standardize them. For example: Image tagging will have same UI, price and quality expectation across different requesters
- Having basic components standardized, more complex tasks can be created using these basic units
- Based on the pricing of basic units, the entire task can be priced more accurately.
- Time to complete a task can be predicted more accurately
- The bottleneck unit can be identified and can be priced higher to get it completed quickly
Automatic evaluation or quality assurance of a completed task can be achieved because of standardization.
Both perspectives: A Plea to Amazon: Fix Mechanical Turk
Observations about Workers
- New requesters are looked at with suspicion because of task getting rejected, so in turn good workers follow a wait and see approach or do a tasks in incremental manner
- Instead of rating system for requester, have objective characteristics of requesters like speed of payment, rate of rejection, total volume posted etc. so that workers can have a sense of trust before starting the task.
- Once a task is done and worker gets a sense of trust in requester, it is not possible for worker to look at other pending tasks of the requester. This is because of inadequate searching capabilities.
- Some solutions for better user interface are: browsing the tasks per category, better search and filtering capabilities, a recommender showing similar tasks pending in the MTurk which currently worker finished and/or browsed.
- A good API with all above abilities
- If HIT is not handled by the quality worker for the new requester then he will move away from the system.
- Qualitative feedback is baised, but objective feedback is system generated (e.g. speed of payment, rate of rejection, total volume posted) and cannot be easily fooled.
Observations about requesters
- Not able to predict the estimated completion time of a task once it is posted.
- Few big requester but many small requesters
- Requesters need to build their own mechanism/UI to track and manage the tasks that are posted. This turns up to be very very high cost when compared to tasks being posted on MTurk.
- iFrames are to be used in building custom UI which allows requesters to get workers from other systems like CrowdFlower etc.
- No reputation or rating system for requester which in turn allows spammers to become top requesters easily.
- No system to rate workers by requesters.
- Separating payment from rejection i.e., ability to reject a task allowing payment to be made.
- Workers skills should be categorized and rating on different category so that requesters can view them
- A good API with all above abilities
- Tasks remaining for a longer time and not getting picked by worker is because of lack of searching/filtering capabilities
- Allowing rating of requesters by workers and workers rating requester will lead both of them getting in mutual agreements to have better rating or dedicated workers/requesters will just rate each other.
- Separating payment from rejection will help improve requester-worker relationship and may increase quality of work. Sometimes may cause change of heart in worker who deliberately cheated but still got payment although his tasks was rejected.
Following data is collected from TurkNatio and Reddit forum.
- Minimum hourly wage, minimum per-item pay and a team that helps Requesters set pay fairly. From Turknation
- Better communication between workers, Requesters and the platform administration and tools for conflict mediation. From Turknation
- Improved platform (From Reddit):
- Built-in desktop/browser alert tools
- A fully customisable search engine
- Responsive web design: Some user has large monitors but couldn't use it efficiently since MTurk provides very small screen.
- Way to sort the work by requestor rating
- Integration of all the scripts we have on mTurk now
- International workers/Requesters allowed. From Turknation
- Payment via Paypal/bitcoin/gold bars/whatever. From Turknation
- An in-platform way to rate the requestor. This would include things like rejection rate, user rating, blocking or flagging requester etc. From Reddit
- If the type of work has been done by a previous worker I would like to know the average time it took them to complete it and using this average automatic breakdown of the HIT earning by $-per-minute or second. From Reddit
- A way to see how you are doing: how much you are earning per day, total time worked and $ per minute for you worked. From Reddit
- Consequences for requestors that have a history of not paying or mass rejecting. From Reddit
- A support team that actually investigates and acts on flagged/reported requesters in a reliable and timely manner. From Reddit
- MTurk needs to have a channel where workers can get to know the reason for suspension and request to re-new the account
- Way to adjust qualification attributes of an active HIT. For example, if you find that the quality of results is not high enough maybe you'd want to increase the minimum approval percentage. Or if you're not getting HITs completed fast enough you could decrease it. From Reddit
- Way to block block workers from working on HIT without impacting their record/rating. From Reddit
- A demographic page that could simply be incorporated into any task, so requester don't have to be constantly filling in the same old demographic page each time. From Reddit
Synthesize the Needs You Found
List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.
- Workers need to express their opinion about the requesters. Evidence: Explicit observation #7 of the paper Being a Turker.
- Worked need that the US government will not intervene on legal aspects. Evidence:Explicit observation #19 of the paper Being a Turker.
- Workers need to make good decision on selecting the jobs. Evidence:Explicit observation #13 and #23 of the paper Being a Turker.
- Workers need to establish and maintain good relationship with the requesters.Evidence: Explicit observation #5 of the paper "Turkopticon" and implicit observation #4 of paper "Turkopticon"
- Workers need to act collectively for concerns.Evidence:Explicit observation #19 of the paper Being a Turker.
- Workers need to perform fair rating of the requester, so that others don’t lose opportunity to good work. Evidence:Explicit observation #15 and #16 of the paper Being a Turker.
- Workers need to give rating to requester as to how much of the approved tasks are paid finally. Evidence: Explicit observation #4 and #6 of the paper "Turkopticon"
- Workers need to avoid wage theft.Evidence: Explicit observation #1 of the paper "Turkopticon"
- Workers need disputes to be mandatorily resolved, currently the requesters can choose to ignore the disputes raised and the platform takes no ownership of such arbitration.Evidence: Explicit observation #4 of the paper "Turkopticon", explicit observation #4 and #7 of the paper Being a Turker
- Workers need to get at least minimum wage. Evidence: Reddit
- Workers need that the payments will be realized faster. Evidence: Explicit observation #1 of the paper "Turkopticon"
- Workers need requesters to answer promptly.Evidence:Implicit observation #6 of paper Being a Turker
- Workers need to post reviews anonymously.Evidence: Explicit observation #5 of the paper "Turkopticon"
- Workers needs to work on HIT on Mobile: Evidence: Reddit
- Requester needs to be able to create HITs such that it does not attract spammers, specially if HITs are subjective. Evidence: Observation #2 from video panel of the worker observation section
- Requester needs to be able to determine fair price based on type of questions. Evidence: Observation #11 from video panel of the requester observation section
- Requester needs to increase the pay of an existing HIT, from Reddit