Milestone 2 NotMathMajors

From crowdresearch
Jump to: navigation, search

Attend a Panel to Hear from Workers and Requesters

Observations

These sessions primarily focused on Mechanical Turk as a platform and the difficulties each user had with it, it also touched on oDesk and it's competitive nature.

During the panel, it was immediately apparent that both the workers and the requestors shared an altruistic view towards their community. This is showcased through many of the interactions including, Spamgirl's TurkerNation forum, Dahn's concern for a supplied method of notifying other requestors about dishonest workers, or Sergie Maurice only submitting tasks that offer ethical wages. The emotional response from each user to their respective platform was also observable. The worker Spamgirl mentioned the improbability that she or other workers would stick around if not for the community while the requestor Dahn became excitable at the mention of an easy method of making tasks granular.

Interpretation

After observing the panel, I felt a common root of each topic was that their platform did not supply a large amount of their necessities and lacked usability or transparency. This principle along with a love for the community, seems to drive the users aside from just the money or data gained by using the service half-heartedly provided by Amazon.

Needs

To begin, it is apparent by the number of helpful community forums in use, both workers and requestors need better direction supplied by the platform. Mechanical Turk is virtually devoid of any information or tutorials for the worker side of the platform and the requestor side is not much better.

In the same vein, each worker in the panel brought up an external tool that provided utility for Mechanical Turk. These utilities aren't only an additive to the platform, but also a necessity for any worker who wishes to make a non-negligible amount of income. These workers need assistance in completing and locating tasks provided by the platform and, although the altruistic community creates these tools, not all workers can or will be exposed to them.

In addition, both workers and requestors need to be able to communicate to both improve service and job quality. While Mechanical Turk supplies small scale direct communication between the two, it lacks a larger community portal.

In relation to requestor's specific needs, each requestor needs to be able to weed through the dishonest and illegitimate workers that pop up on Mechanical Turk without each requestor having to create their own method.

Reading Others' Insights

Worker perspective: Being a Turker

Worker and Requester Observations

This paper concentrates little on the actual activity of workers and requesters on MTurk, and more of their "invisible work" that takes place outside of MTurk, primarily on the TurkerNation forum. As such, the work details the activities of planning, tooling, communication, and review that turkers do in order to maximize their efficiency in completing HITs.

Activities

The following activities can be observed:

  • Workers communicate with each other as well as requesters in order to improve their efficiency together.
  • Workers will accept blame for rejected work when the reason why it is rejected is explained thoroughly.
  • Workers develop tools in order to improve upon the platform they use for crowdsourcing (but only because the platform does not fulfill their requirements).
  • Workers value quality Requesters and HIT design over higher pay.
  • Workers negatively respond to any research or activism that has a chance of impeding their "power to control" the labor market.
  • Workers control the market through community.
  • Workers are happy to tell requesters how to construct a well-designed HIT.
  • Requesters design HITs to provide efficient, fast, and cheap results.
  • "Good" requesters are also having to do some "invisible work" to maintain a good relationship with workers and provide well-designed HITs.
Environments

The environments observed are as follows:

  • TurkNation, a forum that fosters communication between turkers.
  • MTurk, the primary crowd-sourcing platform used by workers on TurkNation.
  • MTurkForum.com, a forum that the workers on TurkNation view as a problem as it is known to promote "cheating" or "hacking" the HITs listed on MTurk.
Interactions

The primary interactions between users:

  • Workers foster great relationships between each other.
  • Workers have an emotional attachment to the work they complete on MTurk.
    • This means that the workers are emotionally hurt whenever a requester rejects their work with little reason, or treats the workers like objects rather than people.
  • Workers highly respect requesters that communicate with them on their own terms.
  • Requesters expect quality results from workers.
  • Interactions between requesters and workers often foster better results for both parties involved.
    • The requester is taught by workers how to properly design HITs to avoid scamming and provide quality results.
    • The workers are thus treated like human beings and have another well-designed HIT to complete.
Objects

There are several objects that encompass the whole of MTurk, without all being on the MTurk platform:

  • Tools and scripts that workers develop to improve efficiency.
  • Forums that workers use to communicate effectively.
  • Mechanical Turk itself
  • The websites, images, and audio that make up the content of the HITs.
Users

The primary users of this platform are:

  • Requesters
    • They post HITs expecting fast, quality results.
    • Come in "good" and "bad" flavors in the eyes of the workers.
  • Workers
    • They complete HITs to supplement or make their income.
    • They extend their crowd-work to forums where they believe their discussions can influence and control the crowd-labor market.
    • Love communication with other workers and requesters.
    • The driving force of MTurk, while they are treated more poorly than requesters on MTurk.
      • Without the labor there can be no labor-market.
      • The platform that appeals most to workers actual desires (whether stated or not) will be the one that succeeds.
    • Also come in "good" and "bad" flavors in the eyes of requesters and workers.
  • Researchers
    • Studied the flow of information between workers on TurkerNation forums.
    • Their desire was to complete an ethnographic study of the workers without intervening in their workflow.

Worker perspective: Turkopticon

Worker and Requester Observations

This paper concentrates on the researcher's perspective that the workers on AMT deserved visibility and action in response to requesters. The initial aim was to provide disruption to the standard requester-based crowd-labor procedures, but it also lead to the development of Turkopticon.

Activities

The following activities can be observed:

  • Intervention into the established workflow was successfully completed.
  • Workers were paid for their opinions.
  • Requesters deny work without retribution.
  • Requesters deny payment while still owning any work completed.
  • Requesters automate computation approval.
  • Workers complete tasks without assurance that they will be paid.
Environments

The environments observed are as follows:

  • Turkopticon, a technology that promotes activism by workers for other workers, but is exploitable by requesters.
  • Work environments of workers, while there are some workers that use AMT to pay bills, not all do.
  • Requester workflows, where requesters can simply tap into "human computation" by a method call in a program.
Interactions

The primary interactions between users observed in the paper:

  • Researchers interacted with workers by paying them to complete a survey asking for their opinions.
  • Workers interact with each other by submitting reviews on Turkopticon.
  • Requesters interact with third parties to establish methods of human computation.
Objects

The objects observed in the paper:

  • Turkopticon, a tool for activism by workers.
  • Third-party companies that develop toolkits for employers who wish to tap into AMT's human computation.
Users

The primary users described in the paper:

  • Researchers
    • Directly intervened in the workflow of turkers with little backlash.
    • Did so by injecting themselves in the market that the turkers were already using.
  • Workers
    • Were frustrated with the inability to react to mass rejection by requesters.
    • Some live on the pay from AMT while others do it for fun.
  • Requesters
    • Rated by Turkopticon users
    • Can react to it by flagging reviews they deem untrue.
    • Moderators were enacted to prevent abuse by requesters.
  • Third-party companies
    • Can be used by requesters to develop toolkits for improving requester success with AMT.
    • Make money by aiding requesters.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

Worker and Requester Observations

The focus of this paper was a study conducted in hopes of determining whether crowd-sourced labor could be used to yield valid subjective responses that align with those of experts.

Activities

The following activities can be observed:

  • Several workers "cheated" the first experiment by attempting to submit useless information for pay.
    • These workers were caught and not compensated.
  • Catching the cheaters required much effort on the part of the researchers to ensure they did not deny pay to legitimate respondents.
  • Researchers exerted much effort in designing the second experiment to avoid illegitimate responses.
    • They attempted to make it as easy, if not easier, to submit valid work than to cheat.
Environments

The environments observed are as follows:

  • Wikipedia was used as the subject for their research studies. This allowed the researchers to compare the subjective opinion of the crowd to that of Wikipedia admins.
Interactions

The primary interactions between users observed in the paper:

  • Researchers inadvertently interacted with turkers by submitting two HITs for completion and analyzing the results of the study.
  • Workers inadvertently interacted with Wikipedia admins, and vice versa, because their analyses of Wikipedia pages were compared without their knowledge.
Objects

The objects observed in the paper:

  • Wikipedia pages provided a subject for experimentation by the researchers. Because the pages were topic-specific and included references, the researchers were also able to ask objective validation questions in the second experiment that ensured the workers were actually completing the task.
  • Validation required considerably more programming than the HIT of the first experiment.
Users

The primary users described in the paper:

  • Turkers
    • Divided into two groups by their responses to the experiments.
    • Cheaters
      • These turkers attempted to receive compensation without actually completing the survey that was requested by submitting random responses.
      • Upon the release of the second experiment, many of these turkers were warded off by the realization that the responses would be legitimately analyzed for validity.
    • Valid Workers
      • These turkers were much more scarce during the first experiment, but were the primary respondents to the second.
      • Their valid responses correlated closely to those of the Wikipedia admins.
  • Wikipedia admins
    • Follow a strict procedure for evaluating pages on Wikipedia.
    • Are selected for the role, rather than it being available to everyone.
  • Researchers
    • Hoped to determine whether crowd-sourced results are comparable in quality to that of experts.

Requester perspective: The Need for Standardization in Crowdsourcing

Worker and Requester Observations

This paper concentrates on the negative impact of chaos on the crowd-labor markets, and how standardization would provide more scalability and opportunity for crowd-sourcing as a whole.

Activities

The following activities can be observed:

  • Workers complete very similar HITs.
  • Requesters request very similar work.
  • "Complex" tasks are composed of "building block" tasks.
  • Requesters pay a wide difference in wages for completing similar tasks.
  • Quality Assurance is difficult without a standard for what "quality" work is.
  • Fraud presents the negative externality of distrust to both requesters and workers.
Environments

The environments observed are as follows:

  • Current crowd-labor market which is chaotic due to a lack of standardization or proper handling of negative externalities.
Interactions

The primary interactions between users observed in the paper:

  • Workers desire properly designed HITs from requesters.
  • Requesters desire proper responses from workers.
  • Standards would provide a better experience for both users.
Objects

The objects observed in the paper:

  • "Building-block" tasks are tasks that are extremely simple and are commonly found throughout crowd-labor markets.
  • "Complex" tasks are tasks that can be constructed out of "building-block" tasks.
  • Prices - prices would be more easily decided with standards as it would allow for an easily implemented supply-demand market for work.
Users

The primary users described in the paper:

  • Society
    • Society is the primary user described in the paper aside from the obvious requesters and workers.
    • The author poses the idea that society as a whole would benefit from standardization of crowdsourcing much like society benefited from the standardization of SMS messaging once standards were adopted by companies.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

Worker and Requester Observations

This paper focuses on the shortcomings of AMT, and how they can be improved upon or replaced.

Activities

The following activities can be observed:

  • Workers complete HITs without certainty of pay.
    • Less likely to complete tasks for a new requester.
  • Requesters submit tasks without certainty of quality results.
  • Requesters act as the buyer of labor.
  • Workers wade through a poorly-designed interface and HIT search in order to find work.
  • Requesters devote large amounts of resources in order to develop HITs within iframes.
Environments

The environments observed are as follows:

  • MTurk, which is compared to AltaVista search engine.
Interactions

The primary interactions between users observed in the paper:

  • Amazon does little to interact with their users and ensure trustworthiness.
  • Requesters can mark worker responses as spam with little response.
  • Requesters have little incentive to provide feedback to workers.
  • Workers "rate" requesters through TurkOpticon and forums, despite the fact that the requesters are the "buyers" in this market.
    • The idea of the author is that the workers should not have to do this, Amazon should ensure that all requesters are trustworthy.
Objects

The objects observed in the paper:

  • MTurk Interface
    • Clunky for both workers and requester
Users

The primary users described in the paper:

  • Workers
    • The "sellers" of labor in the market.
    • Thus, the author suggests that they should be reviewed like sellers on Amazon.
    • The workers should have objective data about requesters to determine whether they want to complete their HITs.
    • The workers should be able to find the HITs they want more easily.
    • Workers should not doubt the trustworthiness of requesters.
  • Requesters
    • The "buyers" of labor.
    • Should not be reviewed, but objective data about them should be available.
    • The requesters should not have to hire a developer to post an HIT.
    • Should not doubt the quality of the results they receive.
  • Amazon
    • Creator of AMT that is not regularly responding to its users' requests.
  • Other developers
    • The author presents much opportunity and optimism to developers that are capable of improving upon AMT's shortcomings.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

Browsing through the various communities, there is a visible relation to the success of workers and requestors to their activity in forums, utilizing resources and learning from the other side of their role in Mechanical Turk. It appears as if the community itself dictates the rules that Amazon is unwilling to provide; examples of this being the Turkopticon rating and even some workers trying to form labor union type agreements in their secular community. These forums are integral to being productive on the platform and provide information that isn't remotely provided by Amazon and would be unlikely learned by a single unexposed user.

Synthesize the Needs You Found

Worker Needs

A set of bullet points summarizing the needs of workers.

  • Workers need to confidently complete work.
    • Evidence: The entire development of Turkopticon was in response to this. It is reiterated throughout all of the papers. Workers should not have to worry that the people buying their service aren't trustworthy. Being paid should not be a risk and there should be repercussions for mass-rejecting work.
    • Interpretation: It is hard to find requesters that are trustworthy.
  • Workers need to easily submit quality work.
    • Evidence: In "Crowdsourcing User Studies with Mechanical Turk," the researchers found that providing validation questions and making it just as hard to cheat as to provide real work, actual workers were motivated to complete the HIT.
    • Interpretation: If it is easier to cheat, than to actually complete the task, then cheating is incentivized. By eliminating the incentive to cheat, cheating will be diminished.
  • Workers need to communicate.
    • Evidence: Spamgirl mentioned it in a panel, and it was also referenced in the majority of the readings.
    • Interpretation: As many workers stated, a large part of why they use AMT is because of the community they have created. This is what also gives them a sense of having the power to control the market.
  • Workers need to trust and understand their rejections.
    • Evidence: New users such as this redditor who are unable to determine why their work was rejected or incorrectly completed.
    • Interpretation: It is difficult to improve on AMT or continue using the platform when work is rejected without supplying an explanation for why.

Requester Needs

A set of bullet points summarizing the needs of requesters.

  • Requestors need to communicate.
    • Evidence: Requestors that communicate with workers received better TurkOpticon reviews and, therefore, better results.
    • Interpretation: Requestors have trouble determining what is a good HIT for a worker to complete honestly.
  • Requestors need to easily formulate tasks.
    • Evidence: In all of the requestor-based papers, the difficulty required to create well-designed tasks were mentioned. In "Crowdsourcing User Studies with Mechanical Turk," it was mentioned that it required a great deal of programming in order to verify the data. It is posed in "The Need for Standardization in Crowdsourcing," that standardization would make HIT development easier as well as Quality Assurance.
    • Interpretation: Designing and developing an HIT requires a lot of work. The idea of HITs is to quickly, and efficiently provide answers to normally difficult-to-compute tasks.
  • Requestors need to confidently and quickly collect valid data.
    • Evidence: This is showcased by Gordon's transparent need for reliable data for his lab as well as Dahn's repeatedly mentioned survey involving men pretending to be women with children.
    • Interpretation: It is difficult for requestors to 1) filter workers who are dishonest, or 2) ensure the attention of workers to the instructions.