Difference between revisions of "Milestone 2 taskforce"

From crowdresearch
Jump to: navigation, search
(A Plea to Amazon: Fix Mechanical Turk)
(A Plea to Amazon: Fix Mechanical Turk)
Line 158: Line 158:
 
'''For requesters''':
 
'''For requesters''':
 
"A Better Interface To Post Tasks"
 
"A Better Interface To Post Tasks"
    * Most requesters have crowdsourced workflows and few have one task projects.
+
* Most requesters have crowdsourced workflows and few have one task projects.
    * Requesters needs easier ways to create tasks, and not spend money on developers to do that for them.
+
* Requesters needs easier ways to create tasks, and not spend money on developers to do that for them.
 +
 
 
"A Worker Reputation System"
 
"A Worker Reputation System"
    * Define more public qualifications tests.
+
* Define more public qualifications tests.
    * Create public resumes for workers.
+
* Create public resumes for workers.
    * Create worker ratings.
+
* Create worker ratings.
  
 
'''For workers''':
 
'''For workers''':

Revision as of 17:54, 11 March 2015

Template for your submission for Milestone 2. Do not edit this directly - instead, make a new page at Milestone 2 YourTeamName or whatever your team name is, and copy this template over. You can view the source of this page by clicking the Edit button at the top-right of this page, or by clicking here.

Attend a Panel to Hear from Workers and Requesters

Deliverable

What were the most salient insights you got from hearing the workers and requesters on the panel? What did you learn about their needs?

Workers

  • Workers need an easy and effective way to select work which might take into account
    • requesters’ information
    • offered payment
    • time requested
    • topic
  • Workers need people to talk to and to feel understood by others (within community and in the offline social environment).
  • Workers need to collaborate with other workers.
  • Workers need to tell requesters their opinion and how to improve things.
  • Workers need work to be as granular as possible.
  • Workers need to be able to compare requesters and tasks.
  • Workers need to book the work they will do - they open several windows at the same time.
  • Workers need to communicate easily with requesters and to get responses from them.
  • Workers need explanations on why they get rejected.
  • Workers need to be allowed to work no matter the place they work from (country exclusion etc.)
  • Workers need to go over the cold-start problem (regarding skills).
  • Workers need to face a low learning curve, otherwise they won´t join the platform.
  • Workers need to get accepted to work from time to time
  • Workers need to know where their ratings come from
  • Workers need better to have a better management of different international times (for communication with requesters, for finding work, for working).
  • Workers need to see clearly how to translate their real world skills to microtask crowdsourcing (how to communicate them, how to realize if they are useful).
  • Workers need better instructions on the task and what its purpose is

Requesters

  • Requesters need to have a way to get back to their workers to offer them more work and continue the cycle.
  • Requesters need to design tasks in a way that they are not too boring. Specially after a while, workers may get extremely bored. Introducing bonus helps.
  • Requesters need to get summaries of what is discussed / evaluated (they cannot check all the emails or forums).
  • Requesters need to imagine themselves on the other side (to calculate time estimations etc.).
  • Requesters would love to have a two-step (iteration) process
  • Requesters have to be aware of the selection bias w.r.t. the amount of the reward.
  • Requesters have to provide appropriate training materials.
  • Requesters need to know who their workers are.
  • Requesters need better support from the platform (no reinventing the wheel at a microtask level in for example selection / screening, and improve platform workflows instead).
  • Requesters need to have quality assurance mechanisms that do not require a big overhead but are at the same time working effectively (e.g. gold standard selection etc.)
  • Requesters need to find methods to get the right people for their tasks.
  • Requesters and workers need to have a way to communicate and reach “a mutual understanding of the task”.
  • Requesters need to identify what the right people for their task looks like.
  • Requesters need ratings on different workers skills.
  • Requesters need to take into account that workers are aware that flat rates make them work (consciously) less effectively.
  • Workers would like to have a guarantee on the money they can do per week.
  • Nice idea of an oDesk worker: freelancers motivated by working on work freely, to learn - like internships.
  • Requesters need to still publish during academic breaks! the tasks are 50% from industry and 50% from academia.
  • Requesters and workers need to find a solution together for a stable amount and quality of work available.

Readings

Turkopticon

What worker needs are discussed or implied by the reading? What requester needs are discussed or implied by the reading?

The authors observed the way AMT works, asked crowd workers how they would define their “Bill of Rights” and implemented a disruptive plugin for AMT that offers the possibility to rate the relationship workers had with requesters.

Workers

  • need to know which requester to trust: when workers browse available work in online labor marketplaces they would like to know more information about the requesters. They would like to know whether they should trust them or not.
  • need to speak loud and they need to be heard: workers have the need of sharing their experience with requesters to the community of crowd workers (i.e. advertise and evaluate their relationships with requesters). They need to be visible and express their opinion freely - without getting penalties for doing it.
  • need to feel protected and supported by the community.
  • need to feel respected and fairly treated by the requesters
    • fair acceptance / rejection
    • explained acceptance / rejection
    • fast payment
  • need (and like) to help each other.
  • need an effective mechanism to communicate with requesters.
  • need to have a similar infrastructure to what it exists in the traditional workplace. That´s why they think of minimum wage
  • need a balance between anonymity and reputation.
  • need to get responses from the platform and the requesters to questions and complaints.
  • need to see the platform managing injustices.
  • need to know about the 1) communicativity 2) generosity 3) fairness and 4) promptness of requesters

Turkopticon allows people to give a numeric rating on these qualities and they can also write in a free-form text further reviews of the requesters

(indirect observations)

  • workers need to receive specific instructions on how they should work on tasks

Requesters

  • need to find an evaluation system that filters accepted and rejected work easily as it is not feasible to go through each single response they get.
  • need to listen to the workers’ feedback in order to improve their microtasks and get more interested workforce, but they need an efficient method to process the workers’ comments and feedback, since they cannot spend hours on the email.
  • requesters need to be fairly evaluated.

(indirectly mentioned)

  • requesters need to define what is good and bad work for their microtasks before they assess workers’ answers.

Platforms

  • felt the need to support employers,because the platform works after all thanks to the employers who publish the work.

(the reading motivated these thoughts, but they don’t say this explicitly)

  • need to define a legal framework on which requesters and contributors agree.
  • need to take compromises in order to take into account a global scenario (workers and requesters may come from any country in the world).
  • need to be competitive.
  • need to be able to offer a sustainable amount of work and be able to recruit a sustainable workforce.
  • need to offer a (technologically) reliable service.

Reviews

  • need to be adapted if design changes occur in the crowdsouring process.
  • need to be objective (tactical quantification), simple and collective.
  • reviews need to be globally moderated.

Being a Turker

What worker needs are discussed or implied by the reading? What requester needs are discussed or implied by the reading?

In this paper, the authors study the mechanics of Turker Nation, an online community for Turkers. Through this study, the authors aim to answer questions regarding the relationships and dynamics between the ‘turkers’ and requesters. This work presents a different perspective, one from the counterpoint of a Turker.

Workers

  • Workers perform micro-tasks with a primary aim to earn money. While there may be workers who enjoy the tasks/HITs and perform tasks for lower prices since they are fun, the larger majority agrees that monetary incentive is a driving motivation.
  • Turkers like to learn about ways to become more efficient and earn more money, they see Turker Nation as an ideal place to achieve this.
  • They set monetary targets to achieve on a regular basis.
  • Some crowd workers are full-time Turkers, while some others are part-time workers. These circumstances, along with a Turker’s experience influence the amount of money one can earn.
  • Turking is an important source of income for some turkers, making AMT a ‘safety net’ for them. Turker Nation contains discussion threads where people share their problems and advice.
  • Turkers discuss about requesters on the forum to warn other workers of ‘bad requesters’ or share news about ‘good requesters’.
  • Turkers value polite communication channels with requesters. While a requester that provides clear instructions and approves the work is lauded, the way requesters communicate with the Turkers also influence how their mutual relationship evolves.
  • Genuine Turkers take responsibility when their work is rejected due to fair reasons.
  • Turkers often discuss about acting collectively in different situations.

Requesters

  • Requesters need workers to contribute, so they are willing to keep the workers satisfied.
  • Requesters need workers that provide useful and diligent responses.
  • Requesters are wary of scammers and malicious Turkers.

Crowdsourcing User Studies with Mechanical Turk

What worker needs are discussed or implied by the reading? What requester needs are discussed or implied by the reading?

In this paper the authors ventured into investigating the suitability of microtask markets for collecting user measurements and carrying out user studies. They reflect on the feasibility of doing so, yet caution about the special care that is required while designing such studies. The authors experiment with AMT and present their findings, that confirmed the promise and shortcomings of using MTurk for user studies.

Workers

  • Workers are compensated for the work they contribute to and successfully complete through monetary rewards.
  • Workers use their valuable ‘human intelligence’ to complete simple tasks.
  • The diversity in the workers can be a boon or a bane depending on the microtask at hand.
  • The lack of constraints with regards to ‘expertise’ required to work on tasks, gives (seemingly*) equal opportunity to different workers to consume a task. *Workers still need to rely on their experience in order to get most out of their ‘Turking’.


Requesters

  • Requesters often need to trade-off between the length of the task, the time allocated for task completion and the monetary costs.
  • Requesters can gather large amounts of *representative* data at relatively low costs.
  • Any requester can post a task on AMT and is free to fix the monetary award.
  • Requesters get access to easily available ‘human-intelligence’.
  • Requesters can gather user measurements at rapid speeds.
  • Requesters have to contend with workers that attempt to ‘game’ the system without providing useful responses.
  • A requester can leverage the crowd to approximate expert-like judgements.

A Plea to Amazon: Fix Mechanical Turk

What worker needs are discussed or implied by the reading? What requester needs are discussed or implied by the reading?

In his blog post “ A Plea to Amazon: Fix Mechanical Turk! “ [1] Panos Ipeirotis discusses ways to improve mturk both for requesters and workers. Here are the most interesting points For requesters: "A Better Interface To Post Tasks"

  • Most requesters have crowdsourced workflows and few have one task projects.
  • Requesters needs easier ways to create tasks, and not spend money on developers to do that for them.

"A Worker Reputation System"

  • Define more public qualifications tests.
  • Create public resumes for workers.
  • Create worker ratings.

For workers: "A Requester Trustworthiness Guarantee"

  • How fast does the requester pay.
  • Requester rejection rate.
  • Requester rate.
  • How much work did the requester posted.

"A Better Task Search Interface"

  • The ability to browse tasks per category.
  • Create a better search engine.
  • Propose HITs to workers.

[1]. http://www.behind-the-enemy-lines.com/2010/10/plea-to-amazon-fix-mechanical-turk.html

The Need for Standardization in Crowdsourcing

TODO: What worker needs are discussed or implied by the reading? What requester needs are discussed or implied by the reading?

Synthesize the Needs You Found

Worker Needs

TODO: A set of bullet points of the needs of workers.

  • Example: workers need to be respected

Requester Needs

TODO: A set of bullet points of the needs of requesters.

  • Example: requesters need to trust the results they get from workers