Difference between revisions of "WinterMilestone 2 stormsurfer"

From crowdresearch
Jump to: navigation, search
(Worker perspective: Turkopticon)
 
(6 intermediate revisions by the same user not shown)
Line 2: Line 2:
  
 
'''Report on some of the observations you gathered during the panel.'''
 
'''Report on some of the observations you gathered during the panel.'''
 +
 +
=== Questions asked ===
 +
* @requesters/workers: If you've used other crowdsourcing platforms (other than MTurk), what features ultimately drew you to use MTurk over these other platforms?
 +
* @requesters/workers: If you've used other crowdsourcing platforms (other than MTurk), what features would you borrow from these other platforms?
  
 
=== Worker observations ===
 
=== Worker observations ===
  
 
* '''What observations about workers can you draw from the panel? Include any that may be strongly implied but not explicit.'''
 
* '''What observations about workers can you draw from the panel? Include any that may be strongly implied but not explicit.'''
 +
** Some workers are '''completely dependent''' on MTurk financially, whereas others aren't. Regardless, money is the primary motivator for most workers.
 +
** Workers react strongly to rejections, and rightfully so. First, upon receiving a rejection, the worker wasted his/her time on that HIT (and therefore money). Second, a high rejection rate prevents a worker from accessing certain tasks (and making a higher wage). Workers try hard to flip a requester's rejection.
 +
** Work and pay is highly variable for workers.
 +
** Forums are used to connect workers and allow them to share their experiences, many of whom are frustrated.
 +
** Workers feel that they are not paid fairly for high quality results and dislike tasks that are vague (sometimes intentionally) and especially hate requesters who do not respond to communication about these vague tasks.
  
 
=== Requester observations ===
 
=== Requester observations ===
  
 
* '''What observations about requesters can you draw from the panel? Include any that may be strongly implied but not explicit.'''
 
* '''What observations about requesters can you draw from the panel? Include any that may be strongly implied but not explicit.'''
 +
** Requesters often need to filter out "garbage" and many times do not trust workers (many are inexperienced or are spammers).
 +
** MTurk does not address a lot of the tasks that requesters want to create.
 +
** Requesters sometimes create qualification tests to find and limit a task to high quality workers.
  
 
== Reading others' insights ==
 
== Reading others' insights ==
Line 39: Line 51:
  
 
* '''What observations about workers can you draw from the readings? Include any that may be strongly implied but not explicit.'''
 
* '''What observations about workers can you draw from the readings? Include any that may be strongly implied but not explicit.'''
 +
** For each worker who reported needing the money to pay for rent or groceries, there was another who did it for fun or to kill time.
 +
** MTurk does not have a minimum wage, and many workers complain about not making a fair wage for their work.
 +
** More than half of the workers in the study thought that their work was rejected arbitrarily or unfairly (35 out of the 67 workers).
 +
** Other issues that came up include faster payment (Amazon currently has a 30-day automatic payment policy), unfair wages that were often under the U.S. minimum wage, and general dissatisfaction with employers’ and Amazon’s lack of response to their concerns.
 +
** If a worker's rating drastically falls, it is hard for him/her to pick it back up.
 +
** Workers value the requester (e.g. speed of payment, effective communication, fair pay) as much as the task (e.g. how interesting it is, clearly detailed).
 +
** Qualities that a worker looks for when selecting a requester from Turkopticon include speed of payment, clearly detailed tasks, fair pay, and effective communication. However, workers are scared to post negative reviews about requesters in fear that requesters will retaliate and reject their work (creating a downward spiraling feedback loop).
  
 
* '''What observations about requesters can you draw from the readings? Include any that may be strongly implied but not explicit.'''
 
* '''What observations about requesters can you draw from the readings? Include any that may be strongly implied but not explicit.'''
 +
** Requesters often create criteria for their tasks.
 +
** As I mentioned in the previous milestone, requesters have '''a lot''' of power. They have the ability to use the output from HITs for a task while rejecting that HIT so they don't have to pay the worker (abusing the system).
 +
** Requesters are not required to respond to workers, creating an imbalance of trust; why should workers trust the requesters?
  
 
=== Requester perspective: Crowdsourcing User Studies with Mechanical Turk ===
 
=== Requester perspective: Crowdsourcing User Studies with Mechanical Turk ===
  
 
* '''What observations about workers can you draw from the readings? Include any that may be strongly implied but not explicit.'''
 
* '''What observations about workers can you draw from the readings? Include any that may be strongly implied but not explicit.'''
 +
** Workers spend more time answering verifiable, quantitative questions than on quick, qualitative fields.
 +
** Worker response time for each HIT was very fast (90 seconds) for quantitative answers but much slower (246 seconds) for verifiable, quantitative answers.
  
 
* '''What observations about requesters can you draw from the readings? Include any that may be strongly implied but not explicit.'''
 
* '''What observations about requesters can you draw from the readings? Include any that may be strongly implied but not explicit.'''
 +
** Requesters look for high quality submissions among workers.
 +
** Requesters want to make sure that workers are not cheating and/or gaming the system.
  
 
=== Requester perspective: The Need for Standardization in Crowdsourcing ===
 
=== Requester perspective: The Need for Standardization in Crowdsourcing ===
Line 69: Line 95:
 
** Trust needs to come from both sides: workers and requesters. However, MTurk does not guarantee trustworthiness of requesters at the moment; workers (noted as slaves in the article) want to learn more about their requesters (noted as masters in the article) via forums such as Turker Nation by giving feedback on good/bad requesters.
 
** Trust needs to come from both sides: workers and requesters. However, MTurk does not guarantee trustworthiness of requesters at the moment; workers (noted as slaves in the article) want to learn more about their requesters (noted as masters in the article) via forums such as Turker Nation by giving feedback on good/bad requesters.
 
** As a result, workers will only complete a few HITs for a new requester to see how he/she behaves and ensure that he/she is legitimate (e.g. pays fairly and promptly). This leads to a situation where only spammers and inexperienced workers are willing to do a large number of HITs for a task, leading to lower quality results. Requesters will be disappointed with the work and potentially leave the market or lower the wage price for each HIT. ''Workers should not be afraid to work for a requester.''
 
** As a result, workers will only complete a few HITs for a new requester to see how he/she behaves and ensure that he/she is legitimate (e.g. pays fairly and promptly). This leads to a situation where only spammers and inexperienced workers are willing to do a large number of HITs for a task, leading to lower quality results. Requesters will be disappointed with the work and potentially leave the market or lower the wage price for each HIT. ''Workers should not be afraid to work for a requester.''
** Objective ratings should be available for each requester to the worker, including speed of payment, rejection rate, appeal rate, disallowing the ability to reject work that is not spam, total volume of posted work, and making all of the above accessible via an API. However, the problem (in my view) is still not fixed: new requesters will not have this information available about them (because they just joined) and their "sample size" will not be large, so these pieces of information will not help workers in deciding which new requesters are trustworthy.
+
** Objective ratings should be available for each requester to the worker, including speed of payment, rejection rate, appeal rate, disallowing the ability to reject work that is not spam, total volume of posted work, and making all of the above accessible via an API. However, the problem (in my view) is still not fixed: new requesters will not have this information available about them (because they just joined) and their ''sample size'' will not be large, so these pieces of information will not help workers in deciding which new requesters are trustworthy.
 
** '''Better user interface:''' workers cannot search for good requesters and it is difficult for workers to find tasks of interest. Potential solutions include having a browsing system, improving the search engine, and creating a recommendation system to propose tasks for workers.
 
** '''Better user interface:''' workers cannot search for good requesters and it is difficult for workers to find tasks of interest. Potential solutions include having a browsing system, improving the search engine, and creating a recommendation system to propose tasks for workers.
  
Line 129: Line 155:
  
 
=== Worker needs ===
 
=== Worker needs ===
 +
 +
* '''Workers need to be able to trust/compare/rate requesters (especially new ones).''' As mentioned in ''A Plea to Amazon: Fix Mechanical Turk'' and ''Questions From a Soon-to-be Requester'', workers are weary of new requesters because they are unsure about how fast they pay, how fair they are with pay, and whether they fairly accept/reject HITs. As I stated earlier, workers seem to care about requesters as much (in fact, even more) as the task itself. There is a huge "bump" in the curve for new requesters to scale, which would be solved by having workers trust new requesters. Furthermore, as shown in ''Turkopticon'', there is an imbalance of power in which workers can not rate requesters and must look to forums and other platforms (Turkopticon) to gather information about different requesters. This feature should be built into MTurk itself.
 +
* '''Workers need to be respected by requesters for their time and effort, even if it is not clearly visible.''' As mentioned in ''Being a Turker'', there is a large amount of invisible work on the worker side, and requesters need to value the amount of time that a worker put into a task and not arbitrarily reject a HIT. A good example is when workers have to find results for a certain query as part of a HIT but then are forced to return the HIT (making no money) when there are no results, but the user's work/effort should be valued in some way. Furthermore, it takes time for workers to look up appropriate tasks for them, which is also invisible work (answered in my fourth bullet point).
 +
* '''Workers need to receive a livable wage.''' As shown by pretty much all of the articles (e.g. the panel, ''Being a Turker'', Reddit forums), most Turkers work for money, and for many, Turking is a full-time job; however, the wage they receive is below minimum wage! Even those not working full-time are many times struggling financially, and workers should receive a livable wage.
 +
* '''Workers need to be able to search for tasks that match their interests and abilities.''' As mentioned in my second bullet point, looking up appropriate tasks (judged on pay, whether the task is interesting, etc.) is an invisible task and takes time even though no money is earned. Furthermore, as a benefit to requesters, workers should be able to work on tasks that they are qualified for and will spend an appropriate amount of time on. Therefore, workers should be able to search for tasks that match their interests and abilities.
 +
* '''Workers need to do a task well.''' Workers need to do a task well in order to be paid well. This creates a positive feedback loop between the requester and worker.
  
 
=== Requester needs ===
 
=== Requester needs ===
 +
 +
* '''Requesters need to maintain good communication (provide feedback quickly and justify reasons for a rejection in order to improve worker-requester relationships).''' Workers clearly value their time, as mentioned in the bullet points above for worker needs. Requesters need to respect this by not providing demeaning comments in a rejection response. Furthermore, as shown in the Reddit forums, ''Being a Turker'', and ''Turkopticon'', workers many times feel that they are being unfairly rejected but will take the blame when they understand where they went wrong. Especially with new requesters, as mentioned above (and commented on ''Questions From a Soon-to-be Requester''), speed of communication is important so that workers trust them.
 +
* '''Requesters need to pay workers fairly.''' As shown by pretty much all of the articles (e.g. the panel, ''Being a Turker'', Reddit forums), most Turkers work for money, and for many, Turking is a full-time job; however, the wage they receive is below minimum wage! Even those not working full-time are many times struggling financially, and workers should receive a livable wage.
 +
* '''Requesters need to be able to lower their overhead costs, including setting up/creating a task and filtering out spam/garbage results from a task.''' As primarily mentioned in ''The Need for Standardization in Crowdsourcing'' and ''A Plea to Amazon: Fix Mechanical Turk'', there are huge overhead costs for a new requester, as well as existing ones. Command line tools are not user friendly to most requesters! Making requesting more complex creates a higher cost for requesters (e.g. hiring a full-time developer, which is at least $60k) and defeats the purpose of using MTurk. Furthermore, it is a long process for requesters to evaluate work (again defeating the purpose, many times, because they can do the work more quickly themselves at no cost): building a quality assurance system from scratch, ensuring proper allocation of qualifications, learning to break tasks properly into a workflow, stratifying workers according to quality, etc. This creates a difficulty for growth of small requesters. The purpose of a market is to reduce overhead, friction, transaction costs, and search costs, but that isn't always the case with MTurk.
  
 
== Milestone contributors ==  
 
== Milestone contributors ==  
  
 
Slack usernames of all who helped create this wiki page submission: @shreygupta98
 
Slack usernames of all who helped create this wiki page submission: @shreygupta98

Latest revision as of 10:10, 25 January 2016

Attend a panel to hear from workers and requesters

Report on some of the observations you gathered during the panel.

Questions asked

  • @requesters/workers: If you've used other crowdsourcing platforms (other than MTurk), what features ultimately drew you to use MTurk over these other platforms?
  • @requesters/workers: If you've used other crowdsourcing platforms (other than MTurk), what features would you borrow from these other platforms?

Worker observations

  • What observations about workers can you draw from the panel? Include any that may be strongly implied but not explicit.
    • Some workers are completely dependent on MTurk financially, whereas others aren't. Regardless, money is the primary motivator for most workers.
    • Workers react strongly to rejections, and rightfully so. First, upon receiving a rejection, the worker wasted his/her time on that HIT (and therefore money). Second, a high rejection rate prevents a worker from accessing certain tasks (and making a higher wage). Workers try hard to flip a requester's rejection.
    • Work and pay is highly variable for workers.
    • Forums are used to connect workers and allow them to share their experiences, many of whom are frustrated.
    • Workers feel that they are not paid fairly for high quality results and dislike tasks that are vague (sometimes intentionally) and especially hate requesters who do not respond to communication about these vague tasks.

Requester observations

  • What observations about requesters can you draw from the panel? Include any that may be strongly implied but not explicit.
    • Requesters often need to filter out "garbage" and many times do not trust workers (many are inexperienced or are spammers).
    • MTurk does not address a lot of the tasks that requesters want to create.
    • Requesters sometimes create qualification tests to find and limit a task to high quality workers.

Reading others' insights

Worker perspective: Being a Turker

  • Crowdsourcing (definition): the act of a company or institution taking a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call.
  • What observations about workers can you draw from the readings? Include any that may be strongly implied but not explicit.
    • 80% of the tasks are carried out by the 20% most active Turkers.
    • The majority of Turkers (56%) are U.S. based, but there is a growing number of Indian Turkers (36%) and other nationalities, and nearly one-third of respondents had a median annual income of <$10,000.
    • For posters on Turker Nation, the primary reason for Turking is to earn money (money comes up in the discussions the most). All of the other benefits of Turking are simply side-benefits (Turkers also enjoy doing tasks that are fun, interesting, and/or educational). Some Turkers accept a slightly lower pay if a task is more enjoyable, but most dismiss the idea.
    • Turkers seem to be responsible for promoting fair pay in the system (MTurk).
    • Highest earnings for the most experienced Turkers are approximately $15,000 per year (full-time job), and the minimum is $50, although the researchers do not know how long the Turkers worked. Some workers are on MTurk as a full-time job, whereas others only spend a few hours. Those who only spend a few hours are often not doing well financially either; they use their earnings from MTurk as a supplement to their regular salary ("hand-to-mouth"). Many Turkers are living under difficult circumstances. Not all individuals prefer MTurk and are only Turking as a consequence of the economy; they would much rather prefer to have their salary back. (However, it is important to note that a $15k salary would be considered good in several countries outside of the U.S., including India.)
    • Turkers are interested in their earning potential and set targets for themselves (e.g. $10/day).
    • Primary use of Turker Nation (as well as other forums) is to rate requesters as bad/good with respect to fair pay, approval/rejection of tasks, and comments (which can be demeaning). Consequences of low approval ratings (<90% prevents workers from accessing certain tasks) and blocking (large amounts can cause a ban from MTurk) simply prevent them from earning money, and they feel insulted. Workers value communication; they dislike when a requester rejects their work without a valid reason. They value communication, and positive worker-requester relationships can develop. However, workers generally accept a rejection on a HIT when there is a valid fault in their work. On forums, they are asked to be objective in their approvals/disapprovals of requesters and refrain from posting too quickly (flaming) when their HITs are rejected.
    • A pay rate of $4-6/hour is seen by workers as an acceptable rate and is, in fact, higher than the average pay rate for many tasks on MTurk (yet lower than the minimum wage in the U.S.). However, workers understand the tradeoffs and benefits of working on MTurk. Benefits for Turkers: set hours are not required, money does not have to be spent on transportation costs, and judgments are restricted to the work submitted rather than personal appearance.
    • Invisible work: workers must spend time searching for tasks and often accept poorer paying jobs as an interim means to the bigger goal of better paying (and often more interesting) work.
    • Interestingly, while workers want to adhere to Amazon’s Terms of Service, they do not want the U.S. government to legislate and regulate MTurk. They believe in collective individual actions and believe that if the government intervened, it would lead to a closing of MTurk, which acts as a safety net for many Turkers. Some distrust the government; furthermore, they do not want other individuals making decisions for them.
    • Problems for Turkers: employers who don’t pay, identifying scams, and the cost (to workers) of poorly designed tasks.
    • Problems with MTurk: information asymmetry and the imbalance of power (workers cannot rate/block requesters).
  • What observations about requesters can you draw from the readings? Include any that may be strongly implied but not explicit.
    • Several requesters do improve their relationship with workers, and positive worker-requester relationships can develop over time. Requesters can improve over time (e.g. paying faster, being more specific in their tasks) as they gain feedback.
    • Some requesters outright mass reject tasks and give demeaning comments, but this is not always the case; many times, there is a reason to reject a task.
    • Problems with MTurk: information asymmetry and the imbalance of power (requesters can rate/block workers).

Worker perspective: Turkopticon

  • What observations about workers can you draw from the readings? Include any that may be strongly implied but not explicit.
    • For each worker who reported needing the money to pay for rent or groceries, there was another who did it for fun or to kill time.
    • MTurk does not have a minimum wage, and many workers complain about not making a fair wage for their work.
    • More than half of the workers in the study thought that their work was rejected arbitrarily or unfairly (35 out of the 67 workers).
    • Other issues that came up include faster payment (Amazon currently has a 30-day automatic payment policy), unfair wages that were often under the U.S. minimum wage, and general dissatisfaction with employers’ and Amazon’s lack of response to their concerns.
    • If a worker's rating drastically falls, it is hard for him/her to pick it back up.
    • Workers value the requester (e.g. speed of payment, effective communication, fair pay) as much as the task (e.g. how interesting it is, clearly detailed).
    • Qualities that a worker looks for when selecting a requester from Turkopticon include speed of payment, clearly detailed tasks, fair pay, and effective communication. However, workers are scared to post negative reviews about requesters in fear that requesters will retaliate and reject their work (creating a downward spiraling feedback loop).
  • What observations about requesters can you draw from the readings? Include any that may be strongly implied but not explicit.
    • Requesters often create criteria for their tasks.
    • As I mentioned in the previous milestone, requesters have a lot of power. They have the ability to use the output from HITs for a task while rejecting that HIT so they don't have to pay the worker (abusing the system).
    • Requesters are not required to respond to workers, creating an imbalance of trust; why should workers trust the requesters?

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

  • What observations about workers can you draw from the readings? Include any that may be strongly implied but not explicit.
    • Workers spend more time answering verifiable, quantitative questions than on quick, qualitative fields.
    • Worker response time for each HIT was very fast (90 seconds) for quantitative answers but much slower (246 seconds) for verifiable, quantitative answers.
  • What observations about requesters can you draw from the readings? Include any that may be strongly implied but not explicit.
    • Requesters look for high quality submissions among workers.
    • Requesters want to make sure that workers are not cheating and/or gaming the system.

Requester perspective: The Need for Standardization in Crowdsourcing

  • What observations about workers can you draw from the readings? Include any that may be strongly implied but not explicit.
    • Workers in offline settings are normally screened in order to ensure quality, and the challenge to do this arises in crowdsourcing.
    • Task standardization was introduced for workers offline by Henry Ford with the assembly line.
    • What do workers value more, reputation or flexibility?
    • Workers must learn the intricacies of each requester, including the different interfaces for their tasks and the various quality requirements for each requester.
  • What observations about requesters can you draw from the readings? Include any that may be strongly implied but not explicit.
    • There is a need for standardization in order to reduce overhead costs.
    • Requesters need to learn the "best practices" of the system and learn how to effectively price their tasks. Creating standard work units can allow requesters to know how to price a task.
    • Feedback is costly but creates better workers and allows scalability for workers.
    • Benefits of standardization: reusability, trading commodities, true market pricing.
    • Requesters value work flows; estimating a price for work flows would be valuable.
    • Requesters value high quality work and want to enforce standards for workers to make quality assurance easier.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

  • What observations about workers can you draw from the readings? Include any that may be strongly implied but not explicit.
    • Trust needs to come from both sides: workers and requesters. However, MTurk does not guarantee trustworthiness of requesters at the moment; workers (noted as slaves in the article) want to learn more about their requesters (noted as masters in the article) via forums such as Turker Nation by giving feedback on good/bad requesters.
    • As a result, workers will only complete a few HITs for a new requester to see how he/she behaves and ensure that he/she is legitimate (e.g. pays fairly and promptly). This leads to a situation where only spammers and inexperienced workers are willing to do a large number of HITs for a task, leading to lower quality results. Requesters will be disappointed with the work and potentially leave the market or lower the wage price for each HIT. Workers should not be afraid to work for a requester.
    • Objective ratings should be available for each requester to the worker, including speed of payment, rejection rate, appeal rate, disallowing the ability to reject work that is not spam, total volume of posted work, and making all of the above accessible via an API. However, the problem (in my view) is still not fixed: new requesters will not have this information available about them (because they just joined) and their sample size will not be large, so these pieces of information will not help workers in deciding which new requesters are trustworthy.
    • Better user interface: workers cannot search for good requesters and it is difficult for workers to find tasks of interest. Potential solutions include having a browsing system, improving the search engine, and creating a recommendation system to propose tasks for workers.
  • What observations about requesters can you draw from the readings? Include any that may be strongly implied but not explicit.
    • Problems for requesters: scaling up, managing the complex API, managing execution time, ensuring quality.
    • A marketplace is used to reduce overhead, friction, transaction costs, and search costs.
    • Last major change on MTurk was adding a UI to create batch tasks; command line tools are not user friendly to most requesters! Making requesting more complex creates a higher cost for requesters (e.g. hiring a full-time developer, which is at least $60k) and defeats the purpose of using MTurk. Furthermore, it is a long process for requesters to evaluate work (again defeating the purpose, many times, because they can do the work more quickly themselves at no cost): building a quality assurance system from scratch, ensuring proper allocation of qualifications, learning to break tasks properly into a workflow, stratifying workers according to quality, etc. This creates a difficulty for growth of small requesters.
    • Requesters want to easily create workflows (somewhat related to the idea behind Foundry).
    • Current reputation system is easy to manipulate, and it is frustrating for requesters to ensure quality HITs among workers. This leads to lower wages, resulting in only bad workers staying on MTurk. Recommended fixes: public qualification tests (e.g. literacy tests, writing samples), working history, rating workers, disconnecting payment from rating, separating HITs/ratings by type, making all of the above accessible via an API.
    • Even requesters want a two-way reputation system!

Needfinding by browsing MTurk-related forums, blogs, Reddit, etc.

List out the observations you made while doing your fieldwork. Links to examples (posts/threads) would be extremely helpful.

Reddit Forum Winter Milestone 2 stormsurfer.png

I browsed Reddit to gather perspectives from workers and requesters on Amazon Mechanical Turk.

Worker perspective: Mechanical Turk Pet Peeves

Reddit Pet Peeves Winter Milestone 2 stormsurfer.png

  • What observations about workers can you draw from the readings? Include any that may be strongly implied but not explicit.
    • Asking workers to return the HIT if they can't find results. Workers do not seem to like requesters who do not value their time (and do not pay workers for the time they spent on a task, even if there are no hard results).
    • Workers value fair pay, no matter who the requester is (even if it is a poor graduate student doing research); workers themselves need to pay basic commodities/needs, including their rent.
    • Workers need enough time to complete a HIT for a task.
    • Workers should not be blocked in order to prevent survey retakes (as mentioned before, this could potentially ban them from MTurk). There should be a better way to do this!
    • Problems arise in the task. For example, the task itself is not proofread and has several errors or the description/directions are not clear enough.
    • Requesters do not give a correct estimate for task completion time, leading workers to provide bad data as they quickly try to complete the task (e.g. a 20-minute survey that was promised to only take 5 minutes to complete; the worker then rushes through the last 15 minutes of the survey).
    • Surveys screen workers 5 minutes into a survey, wasting a worker's time (which is money).
  • What observations about requesters can you draw from the readings? Include any that may be strongly implied but not explicit.
    • Requesters are annoyed when workers are not qualified for a job and do not complete the task properly.
    • Workers often rush through a survey and provide bad data, which is worthless.
    • Requesters do not often put system requirements in the task, which is important from a worker's point of view.
    • Requesters reject work even though it is their fault (e.g. they can't use the data, but the worker did their party correctly). Currently, there is no feature in MTurk to prevent this from happening.

Requester perspective: Questions From a Soon-to-be Requester

Reddit Requester Questions Winter Milestone 2 stormsurfer.png

  • What observations about workers can you draw from the readings? Include any that may be strongly implied but not explicit.
    • Completion rate for a batch of HITs is directly proportionate to the compensation. If a requester's HITs are not being completed fast enough, he/she should increase the reward. This suggests that workers are keen about earning money.
    • Highly qualified workers have access to higher-paying tasks; requesters need to offer a competitive rate in order to attact these workers.
    • Many workers will not complete writings tasks.
    • Workers will be deterred from completing a task when they discover they can only complete 1 HIT from a HIT group. The time spent reading the instructions often make it not worth it for a single task. Workers will gravitate towards tasks where they can do multiple HITs in a single session.
    • Some workers are weary of new requesters.
  • What observations about requesters can you draw from the readings? Include any that may be strongly implied but not explicit.
    • Requesters are more familiar with a task than workers are, and they could probably do the task in a smaller amount of time (law of worth).
    • New requesters need to learn how to properly create tasks, including figuring out things like load time, reading and making sense of instructions, proofreading, etc.
    • Requesters want to eliminate as much garbage as possible and try to create inclusion/exclusion criteria.
    • New requesters need to get a good TO rating so that workers trust them. Advice from a commenter: start off with a batch of 20 HITs with 10 assignments each. Make sure the HIT instructions are extremely clear. Make the HIT as easy to complete as possible. Make the HIT fun. Include a way for workers to easily let you know of any suggestions for improving your HIT. Your goal is to get a couple of workers to rate you positively on TurkOpticon (or other forums).

Synthesis

List out your most salient and interesting needs for workers and requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

Worker needs

  • Workers need to be able to trust/compare/rate requesters (especially new ones). As mentioned in A Plea to Amazon: Fix Mechanical Turk and Questions From a Soon-to-be Requester, workers are weary of new requesters because they are unsure about how fast they pay, how fair they are with pay, and whether they fairly accept/reject HITs. As I stated earlier, workers seem to care about requesters as much (in fact, even more) as the task itself. There is a huge "bump" in the curve for new requesters to scale, which would be solved by having workers trust new requesters. Furthermore, as shown in Turkopticon, there is an imbalance of power in which workers can not rate requesters and must look to forums and other platforms (Turkopticon) to gather information about different requesters. This feature should be built into MTurk itself.
  • Workers need to be respected by requesters for their time and effort, even if it is not clearly visible. As mentioned in Being a Turker, there is a large amount of invisible work on the worker side, and requesters need to value the amount of time that a worker put into a task and not arbitrarily reject a HIT. A good example is when workers have to find results for a certain query as part of a HIT but then are forced to return the HIT (making no money) when there are no results, but the user's work/effort should be valued in some way. Furthermore, it takes time for workers to look up appropriate tasks for them, which is also invisible work (answered in my fourth bullet point).
  • Workers need to receive a livable wage. As shown by pretty much all of the articles (e.g. the panel, Being a Turker, Reddit forums), most Turkers work for money, and for many, Turking is a full-time job; however, the wage they receive is below minimum wage! Even those not working full-time are many times struggling financially, and workers should receive a livable wage.
  • Workers need to be able to search for tasks that match their interests and abilities. As mentioned in my second bullet point, looking up appropriate tasks (judged on pay, whether the task is interesting, etc.) is an invisible task and takes time even though no money is earned. Furthermore, as a benefit to requesters, workers should be able to work on tasks that they are qualified for and will spend an appropriate amount of time on. Therefore, workers should be able to search for tasks that match their interests and abilities.
  • Workers need to do a task well. Workers need to do a task well in order to be paid well. This creates a positive feedback loop between the requester and worker.

Requester needs

  • Requesters need to maintain good communication (provide feedback quickly and justify reasons for a rejection in order to improve worker-requester relationships). Workers clearly value their time, as mentioned in the bullet points above for worker needs. Requesters need to respect this by not providing demeaning comments in a rejection response. Furthermore, as shown in the Reddit forums, Being a Turker, and Turkopticon, workers many times feel that they are being unfairly rejected but will take the blame when they understand where they went wrong. Especially with new requesters, as mentioned above (and commented on Questions From a Soon-to-be Requester), speed of communication is important so that workers trust them.
  • Requesters need to pay workers fairly. As shown by pretty much all of the articles (e.g. the panel, Being a Turker, Reddit forums), most Turkers work for money, and for many, Turking is a full-time job; however, the wage they receive is below minimum wage! Even those not working full-time are many times struggling financially, and workers should receive a livable wage.
  • Requesters need to be able to lower their overhead costs, including setting up/creating a task and filtering out spam/garbage results from a task. As primarily mentioned in The Need for Standardization in Crowdsourcing and A Plea to Amazon: Fix Mechanical Turk, there are huge overhead costs for a new requester, as well as existing ones. Command line tools are not user friendly to most requesters! Making requesting more complex creates a higher cost for requesters (e.g. hiring a full-time developer, which is at least $60k) and defeats the purpose of using MTurk. Furthermore, it is a long process for requesters to evaluate work (again defeating the purpose, many times, because they can do the work more quickly themselves at no cost): building a quality assurance system from scratch, ensuring proper allocation of qualifications, learning to break tasks properly into a workflow, stratifying workers according to quality, etc. This creates a difficulty for growth of small requesters. The purpose of a market is to reduce overhead, friction, transaction costs, and search costs, but that isn't always the case with MTurk.

Milestone contributors

Slack usernames of all who helped create this wiki page submission: @shreygupta98