View source for WinterMilestone 2 vinyoshy

Template for your submission for [[Winter Milestone 2]]. '''DO NOT EDIT THIS DIRECTLY''' - instead, make a new page at [[WinterMilestone 2 YourTeamName]] or whatever your team name is, and copy this template over. You can view the source of this page by clicking the Edit button at the top-right of this page, or by clicking [http://crowdresearch.stanford.edu/w/index.php?title=WinterMilestone_2_Template&action=edit here].

== Attend a Panel to Hear from Workers and Requesters ==
'''Observations'''
* Most workers use some form of script in order to find work
* Many workers like to use forums like TurkerNation or Reddit
** These forums seem to built more towards workers than requesters
** Such forums are usually communities of very dedicated crowd laborers
* For most workers, pace and hours for working are variable
** Work and life seems to clash a lot -- hard to balance the two
** Inconsistency in the availability of good work can often ruin otherwise good work sessions
** Highly competitive nature of work means that workers are often thinking about work 24/7
** Having a daily earning goal is a common practice to try to instill order in an otherwise chaotic system
* Requesters often use crowd labor to rapidly collect data to test a hypothesis
* Both workers and requesters find that having clear instructions is very important
* As a worker, maintaining a good approval rating is very important
** Try to avoid having a bunch of hits with one requester so they don't all get rejected and lower the worker's overall rating
** Workers tend to stick to requesters with a safer/better reputation in terms of accepting hits
** Communication between workers and requesters is very important for workers.
*** Allows for clarifications on instructions, to ensure hits are being done correctly
*** Most workers find that communication is very spotty and unreliable
* Requesters find that there's a lot of pressure not to reject bad work, but also can't accept everything
** Some requesters circumvent this by temporarily removing qualifications from bad workers so they can prevent them from sending bad hits without lowering their rating
* Occasionally, language can present a barrier to proper communication (this doesn't seem too common)
** Requesters feel that they don't have enough time to respond to worker messages
** There seems to be no proper process to protest an unfair hit rejection
* Hard for requesters to be sure workers maintain attention span and give good hits
** Willing to pay more for good workers who they trust (often found on forums)
* Given approximations for how long a hit takes can be very inaccurate. Workers tend to guess just by experience instead.
** Forums can sometimes help with this but people still work at different paces.
* Wages vary a lot -- too much
* Rejecting work can scare off good workers
** Requester perceptions of workers tend to be much higher than worker perceptions of requesters.

== Reading Others' Insights ==

=== Worker perspective: Being a Turker  ===

* Turkers are largely American or Indian
* One-third of Turkers make less than $10,000 a year
** Despite this, interest for workers on AMT is almost entirely driven by money
** There is a disparity between what is considered a good wage in India and in America that might explain some of this
** Not everyone is entirely reliant on AMT for their income, though some are
* 3 problems workers often face are: identifying scammers, not getting paid, and poorly designed tasks
* No system in place for rating requesters
** Requesters have more info on workers than workers have on requesters
* Flexible hours and the prospect of being judged by the quality of their work draws people to AMT
** Poor job market can also further this
* Having work unfairly rejected hits workers very hard
** No real process for appealing this, gives workers little voice vs requesters
* Strong community desire for fairness
* $6/hour was considered to be a wage that Turkers would work for
* Novice Turkers have to spend a lot of time getting their hit count up
** Novice or experienced, all Turkers have to worry about their approval ratings
* Honest workers don't feel like they're reaching honest requesters, and honest requesters don't feel like they're reaching honest workers




=== Worker perspective: Turkopticon ===

* There is nothing workers can do about unfair hit rejectsion
* 4 primary worker concerns:
** Unfair hit rejectiosn
** Payments need to come in faster
** No minimum wage
** Lack of communication between workers and requesters
* No long-term relationships between workers and requesters, though there appears to be some desire for this
* Communicativity, generosity, fairness, and promptness are major criterion to be considered when judging requesters
* No standardized wages
* TurkerOpticon faces some opposition because it highlights poor treatment of workers, but is very well regarded among most workers

=== Requester perspective: Crowdsourcing User Studies with Mechanical Turk ===

* The system as it is is best for questions with definitive answers
** Its hard to tell whether data is good for more subjective work
* No demographic info makes it hard to conduct a proper experiment or study
* The same few people tend to be the ones trying to game the system, rather than it being a widespread phenomenon
** At the same time, this is not an isolated issue, requesters need to consider this when making hits
** Well constructed, straightforward tasks tend to produce more honest answers


=== Requester perspective: The Need for Standardization in Crowdsourcing ===

* Crowd labor as of right now is very non-standardized
** Standardization will increase efficiency
*** This would include clear templates for instructions that can be applied to many scenarios
** Standardization would mean less guess work on the side of both requesters and workers
** Would make it easier to set pricing for work as well
* Standardization allows the platform to prevent people from engaging in activities with negative side-effects without realizing it
** (EG cheating on questions without realizing it will reduce the trust between requesters and workers)


=== Both perspectives: A Plea to Amazon: Fix Mechanical Turk ===
* Requesters and workers have to work with a very poor UI
* There is no real way to differentiate good workers from bad ones
* The qualification system as it is is not extensive enough
* Rating is tied purely to acceptance rate
** Can't differentiate someone who is cheating from someone who made a mistake
** Rating is universal when people might actually be performing better at some tasks than others
* There is little to no information listed about requesters on AMT itself

== Synthesize the Needs You Found ==

=== Worker Needs ===

* Workers need to be sure that they will be able to survive off of the income they make from their labor. Almost every paper mentioned how low the wages on crowdsourcing platforms tended to be. A few users in the first essay noted that they would rather not be Turker workers if they had the choice. A minimum wage of some sort would be a good starting point.

* Workers need to know that during any given work session they will have access to reliable sources of work. Both workers during the panel noted how inconsistent good work could be to find. Workers shouldn't feel like they're wasting their time while doing crowd labor

* Workers need their voices to be heard when protesting unfair requesters. There is currently nothing that prevents requesters from rejecting work and then using it anyways. There is no process for protesting unfair hit rejection (which hurts all workers since they need to worry about their approval rating)

* Workers need a sense of community. As both workers pointed out during the panel, forums and chats are very popular with dedicated Turkers. Because the labor is so isolating, being able to feel like part of a community is very appealing to workers. Currently this need is being met my forums like TurkerNation and Reddit, but this has yet to be integrated into the platform itself

* Workers need some leverage over requesters. In the current set up requesters hold all the power in the relationship, and many of the papers noted the lack of some kind of way to rate requesters meaningfully.

* Workers need better channels to communicate to their requesters. One of the workers in the panel noted that she would never know whether or not a requesters would actually respond to her messages. Because workers risk lowering their approval rate every time they take on a task, they need to be able to talk to the person ultimately responsible for whether their work is accepted
 

=== Requester Needs ===

* Requesters, like workers, need a better UI to work with, as Peter Ipeirotis points out

* Requesters need some way to know the demographics of the Turker population. Since much of the work performed for scientific studies and data collection, knowing your population sample is important to producing good data

* Requesters need some way of distinguishing workers who make mistakes and those who deliberately try to game the system as pointed out in A Plea to Amazon. As of right now, people who produce lower quality work are considered to be as bad as those who cheat

* Requesters need to access to workers who consistently produce good results. One of the requesters mentioned in the panel that he would actively reach out to communities like TurkerNation to discuss work. Requesters seem to be willing to pay for good results but feel distrustful due to the relative anonymity of the current set up.

* Requesters need clearer standards for how a hit should be set up (in terms of layout, wages, and instructions). Peter points out the benefits of how standardization of crowd sourcing labor would mutually benefit workers and requesters.

* Requesters need better ways to identify people who consistently try to game the system. One of the papers pointed out that it was the same few people who would repeatedly try to cheat on hits. If requesters were able to do this, there would be a better dynamic of trust between workers and requesters


Author: @vkoshy