Milestone 2 supercowpowers

From crowdresearch
Jump to: navigation, search

Attend a Monday Panel to Talk with Expert Workers and Requesters


The deliverable for the panel subsection: report on some of the observations you gathered during the panel. You can hold back on interpretations and needs until you finish the rest of the observation-gathering in the next steps.


During the panel:

  • One worker, David, mentioned occasionally using oDesk and other non-AMT sites, but was put off by the much-greater intensity of competition which made finding work difficult.
  • Serge, a requester, had a battle plan for ensuring quality results of: put most of the effort into post-hoc cleaning, ignore "gold standard" questions for weeding out low quality answers, and wherever possible trying to elicit open ended responses.
  • R. Gordon, a requester mostly running surveys on the site, mentioned a preference for rewarding Turkers with high bonuses instead of announcing up front a high base salary, because a high salary may draw too much (of the wrong kind of?) traffic
  • The single biggest initial hurdle workers had to clear when starting their tenures on AMT was finding work; this was doubly true for non-US workers.
  • Though many of the workers on the panel encouraged the use of third-party forums, both Serge and R. Gordon felt the forums weren't a good match for their workflow as a requester to consult the forums for advice or vetting of tasks. The feedback tools provided by ATM -- comment fields in HITs, emails from Turkers -- was sufficient.

A panel participant linked, during the associated Slack chat, to the paper Monopsony and the Crowd: Labor for Lemons?. Some observations from this reading:

  • Firms including Google, Unilever, and Netflix are running large scale crowdsourcing operations. Their SEC reports are reported to contain details of many private crowdsourcing platforms.
  • Workers tend to concentrate on particular specialties, at least to the extent that the authors designed a methodology to be robust against this specialization: "Early ethnographic evidence suggested that workers focus on finding and doing specific types of tasks on AMT to optimize their time on the platform, specializing in certain task types that fit their availability and skill sets. Merely posting the survey on AMT, as is commonly done by those conducting surveys on crowdsourcing platforms, may over-sample workers who typically do surveys as tasks for work."
  • A non-trivial number of Turkers need the money: "Again, because wages are predeterminately set by requesters, crowdworkers have no alternative to accepting the wages requesters offer, except for unemployment or earning no crowdsourcing income from the platform. While crowdsourcing income may seem an insignificant amount to some, our survey data indicates that at least 14% of crowdworkers in the United States are living under the federal poverty line, and a majority report that they rely on crowdsourcing money to supplement their income, if they even have other source income (many do not)."
  • Requesters may practice wage discrimination: "In fact, it is common practice among requesters to restrict their HITs to U.S.-based workers because U.S.-based workers are perceived to do higher quality work on average (Chandler and Kapelner, 2013). Thus high quality India-based workers face wage discrimination simply because they live in India."
  • Many Turkers find HITs through ancillary websites; it's possible that most work is done by Turkers who use such third party networks: "[W]e asked respondents to identify their current geographic location and tell us how they found our mapping HIT. As Figure 3 demonstrates, the majority of traffic to the HIT came from online forums opposed to general searches conducted on the AMT platform itself. If the AMT Marketplace provided sufficient information, we would expect the opposite of what we found through our mapping HIT."
  • Request authorship is concentrated: "[T]he top 1 percent of requesters on AMT post approximately 60 percent of the total rewards available on the AMT marketplace (Ipeirotis, 2010), and 10 percent of all requesters post 90 percent of total rewards (wages) available."
  • The platform itself may have to take certain precautions to keep from being labelled, legally, an employer of the Turkers: "Today, platform providers are incentivized to minimize the risk of being deemed an employer under the law. Most platform providers will not integrate technical fixes to their APIs that support workers through training, collaboration, and information-sharing, as such enhancements may suggest that the platform curates a workforce."

Reading Others' Insights

1) What observations about workers can you draw from the readings? Include any that may be are strongly implied but not explicit.

2) What observations bout requesters can you draw from the readings? Include any that may be are strongly implied but not explicit.

Worker perspective: Being a Turker

Martin D, Hanrahan B V, O'Neill J, et al. Being a turker. Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 2014: 224-235.


Probably the main point the paper is making is that, no really, most turkers are primarily turking to earn money, and it's not fair to pretend that they aren't. Other observations from the paper:

  • It's hard to even get to US minimum wage by working on Turk.
  • Workers value the unregulated nature of the marketplace. They trust in their ability to jointly navigate its pitfalls.
  • Turkers have a strong sense of fairness. E.g., this means that they might feel less bad doing shoddy work for a requester who's paying crap wages.
  • Communication is important for workers. And requesters who reach out can get valuable advice.

Worker perspective: Turkopticon


On workers:

  • A 2008 survey found around 47% of Turkers were American.
  • Workers can receive payment on a delay of as many as 30 days

Requester perspective: Crowdsourcing User Studies with Mechanical Turk


On workers:

  • In these experiments, most workers provided good faith answers to the survey questions: "Only 8 users gave 5 or more responses flagged as potentially invalid based on either comments or time; yet these same users accounted for 73% (90 responses) of all flagged responses. Thus it appeared that, rather than widespread gaming, a small group of users were trying to take advantage of the system multiple times."

On requesters:

  • Requesters have ways of eliciting higher quality responses from Turkers: responses were more meaningfully correlated with expert opinion when more in depth questions were required, ones that could be verified, as in Experiment II.
  • Requesters bear a cost for bad faith responses on the part of Turkers: "Even though these users were not [monetarily] rewarded (invalid responses were rejected), they consumed experimenter resources in finding, removing, and rejecting their responses.”


A new and interesting observation for me was that a newbie requester who posts a large number of HITs will actually have a worse experience that one who posts a small number of HITs, because good/experienced Turkers will be afraid to do more than a few of them.

Requester perspective: The Need for Standardization in Crowdsourcing


On requesters:

  • Some requesters leave job postings in bad faith as well: "Even more troubling, there is some evidence that at least some markets are becoming inundated with spammers."
  • It may make sense to subsidize some requesters, for the good of the platform: "The process would then operate like this: The workers identify the price for which they are willing to work. Then, the automated market maker takes into consideration the 'ask' (the worker quote) and the 'bid' (the price of the task), and can perform the trade by 'bridging' the difference. Essentially, such automated market makers provide a subsidy in order for the transactions to happen. We should note that a market owner can typically benefit even in scenarios, where they need to subsidize the market through an automated market maker: the fee from a transaction that happens can cover the necessary subsidy which is consumed by the automated market maker."


Amazon has clearly not wanted to invest a lot of resources into MTurk. So I really don't expect Amazon to take up anything like this agenda... Some requester needs pointed out:

  • Having it be easy to get standard tasks done, including:
    • Not having to come up with best practices for interfaces
    • Not having to guess at pricing
    • Not having to identify spammers
  • Having it be easy to string together workflows

Both perspectives: A Plea to Amazon: Fix Mechanical Turk


On requesters:

  • In the author's experience consulting with large-scale businesses trying to use the AMT platform: "Every single requester has the same problems: [s]caling up[, m]anaging the complex API[, m]anaging execution tim[, and e]nsuring quality”
  • Requesters may be able to route workers from many platforms into the same tasks: "For many many HITs, the only way to have a decent interface is to build it yourself in an iframe. What is the problem with iframe? Doing that, MTurk makes it extremely easy for the requester to switch labor channels."

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc


List out the observations you made while doing your fieldwork. Links to examples (posts / threads) would be extremely helpful.

Synthesize the Needs You Found


List out your most salient and interesting needs for workers, and for requesters. Please back up each one with evidence: at least one observation, and ideally an interpretation as well.

A set of bullet points summarizing the needs of workers.

  • Example: Workers need to be respected by their employers. Evidence: Sanjay said in the worker panel that he wrote an angry email to a requester who mass-rejected his work. Interpretation: this wasn't actually about the money; it was about the disregard for Sanjay's work ethic.
  • Brian: Workers need to feel they have options when it comes to what tasks they pick. Evidence: panelists all roundly agreed that finding that initial amount of work was the strongest first hurdle; the Monopsony paper found most working Turkers were finding work through "where are the good HITs" message boards; workers tend to specialize in certain tasks; most Turkers give good faith responses to the HITs they do find. Interpretation: Turkers act as if -- and I think this belief is reasonable -- that there are many more workers interested in working, than there are requesters interested in paying. They're quite nervous that this means they'll wind up exploited (mass rejection of HITs to avoid payment), so they form support groups and use third-party rating tools and guides. They don't react to this competition by resenting each other: the panelists seemed to agree with Spamgirl that it's the opposite case, that they depend on each other for that particular kind of "I know what you're going through" kind of support. Instead, the problem is they worry that the work they're currently finding will start underbidding them so they specialize to try and command a premium as (preemptive?) compensation. It's a huge source of anxiety, and a lot of would-be Turkers drop out way too early from it.

A set of bullet points summarizing the needs of requesters.

  • Example: requesters need to trust the results they get from workers. Evidence: In this thread on Reddit (linked), a requester is struggling to know which results to use and which ones to reject or re-post for more data. Interpretation: it's actually quite difficult for requesters to know whether 1) a worker tried hard but the question was unclear or very difficult or an edge case, or 2) a worker wasn't really putting in a best effort.
  • Brian: Requesters need to be guided by the platform much more than is currently the case. Evidence: calls for greater standardization, patterns of requesters seeking advice from other requesters, the NPR hosts confusion about how to post and price their transcription tasks; all the effort to avoid or ameliorate bogus replies (and all the effort that other, more senior requesters say is actually counterproductive, like "gold standard" items). Interpretation: Requesters helping requesters is nice, but may be one-sided advice. The platform wants the platform as a whole to thrive, and has visibility into both sides of the market, so it makes more sense for the platform to be the source of advice. Everyone's naturally risk averse, and the requesters are the ones taking the greater initial risk. (It is worth considering, though, if there are legal limits to how hands-on a platform can be before triggering labor law standards which complicate the project.)