Difference between revisions of "Milestone 2 taskforce"

From crowdresearch
Jump to: navigation, search
(Deliverable)
(Turkopticon)
Line 57: Line 57:
 
=== Turkopticon ===
 
=== Turkopticon ===
  
TODO: What worker needs are discussed or implied by the reading? What requester needs are discussed or implied by the reading?
+
''What worker needs are discussed or implied by the reading? What requester needs are discussed or implied by the reading?''
 +
 
 +
The authors observed the way AMT works, asked crowd workers how they would define their “Bill of Rights” and implemented a disruptive plugin for AMT that offers the possibility to rate the relationship workers had with requesters.
 +
 
 +
'''Workers'''
 +
* need to know which requester to trust: when workers browse available work in online labor marketplaces they would like to know more information about the requesters. They would like to know whether they should trust them or not.
 +
* need to speak loud and they need to be heard: workers have the need of sharing their experience with requesters to the community of crowd workers (i.e. advertise and evaluate their relationships with requesters). They need to be visible and express their opinion freely - without getting penalties for doing it.
 +
* need to feel protected and supported by the community.
 +
* need to feel respected and fairly treated by the requesters
 +
** fair acceptance / rejection
 +
** explained acceptance / rejection
 +
** fast payment
 +
* need (and like) to help each other.
 +
* need an effective mechanism to communicate with requesters.
 +
* need to have a similar infrastructure to what it exists in the traditional workplace. That´s why they think of minimum wage
 +
* need a balance between anonymity and reputation.
 +
* need to get responses from the platform and the requesters to questions and complaints.
 +
* need to see the platform managing injustices.
 +
* need to know about the 1) communicativity 2) generosity 3) fairness and 4) promptness of requesters
 +
Turkopticon allows people to give a numeric rating on these qualities and they can also write in a free-form text further reviews of the requesters
 +
 
 +
(indirect observations)
 +
* workers need to receive specific instructions on how they should work on tasks
 +
 
 +
'''Requesters'''
 +
* need to find an evaluation system that filters accepted and rejected work easily as it is not feasible to go through each single response they get.
 +
* need to listen to the workers’ feedback in order to improve their microtasks and get more interested workforce, but they need an efficient method to process the workers’ comments and feedback, since they cannot spend hours on the email.
 +
* requesters need to be fairly evaluated.
 +
 
 +
(indirectly mentioned)
 +
* requesters need to define what is good and bad work for their microtasks before they assess workers’ answers.
 +
 
 +
'''Platforms'''
 +
* felt the need to support employers,because the platform works after all thanks to the employers who publish the work.
 +
(the reading motivated these thoughts, but they don’t say this explicitly)
 +
* need to define a legal framework on which requesters and contributors agree.
 +
* need to take compromises in order to take into account a global scenario (workers and requesters may come from any country in the world).
 +
* need to be competitive.
 +
* need to be able to offer a sustainable amount of work and be able to recruit a sustainable workforce.
 +
* need to offer a (technologically) reliable service.
 +
 
 +
Reviews
 +
* need to be adapted if design changes occur in the crowdsouring process.
 +
* need to be objective (tactical quantification), simple and collective.
 +
* reviews need to be globally moderated.
  
 
=== Being a Turker  ===
 
=== Being a Turker  ===

Revision as of 17:49, 11 March 2015

Template for your submission for Milestone 2. Do not edit this directly - instead, make a new page at Milestone 2 YourTeamName or whatever your team name is, and copy this template over. You can view the source of this page by clicking the Edit button at the top-right of this page, or by clicking here.

Attend a Panel to Hear from Workers and Requesters

Deliverable

What were the most salient insights you got from hearing the workers and requesters on the panel? What did you learn about their needs?

Workers

  • Workers need an easy and effective way to select work which might take into account
    • requesters’ information
    • offered payment
    • time requested
    • topic
  • Workers need people to talk to and to feel understood by others (within community and in the offline social environment).
  • Workers need to collaborate with other workers.
  • Workers need to tell requesters their opinion and how to improve things.
  • Workers need work to be as granular as possible.
  • Workers need to be able to compare requesters and tasks.
  • Workers need to book the work they will do - they open several windows at the same time.
  • Workers need to communicate easily with requesters and to get responses from them.
  • Workers need explanations on why they get rejected.
  • Workers need to be allowed to work no matter the place they work from (country exclusion etc.)
  • Workers need to go over the cold-start problem (regarding skills).
  • Workers need to face a low learning curve, otherwise they won´t join the platform.
  • Workers need to get accepted to work from time to time
  • Workers need to know where their ratings come from
  • Workers need better to have a better management of different international times (for communication with requesters, for finding work, for working).
  • Workers need to see clearly how to translate their real world skills to microtask crowdsourcing (how to communicate them, how to realize if they are useful).
  • Workers need better instructions on the task and what its purpose is

Requesters

  • Requesters need to have a way to get back to their workers to offer them more work and continue the cycle.
  • Requesters need to design tasks in a way that they are not too boring. Specially after a while, workers may get extremely bored. Introducing bonus helps.
  • Requesters need to get summaries of what is discussed / evaluated (they cannot check all the emails or forums).
  • Requesters need to imagine themselves on the other side (to calculate time estimations etc.).
  • Requesters would love to have a two-step (iteration) process
  • Requesters have to be aware of the selection bias w.r.t. the amount of the reward.
  • Requesters have to provide appropriate training materials.
  • Requesters need to know who their workers are.
  • Requesters need better support from the platform (no reinventing the wheel at a microtask level in for example selection / screening, and improve platform workflows instead).
  • Requesters need to have quality assurance mechanisms that do not require a big overhead but are at the same time working effectively (e.g. gold standard selection etc.)
  • Requesters need to find methods to get the right people for their tasks.
  • Requesters and workers need to have a way to communicate and reach “a mutual understanding of the task”.
  • Requesters need to identify what the right people for their task looks like.
  • Requesters need ratings on different workers skills.
  • Requesters need to take into account that workers are aware that flat rates make them work (consciously) less effectively.
  • Workers would like to have a guarantee on the money they can do per week.
  • Nice idea of an oDesk worker: freelancers motivated by working on work freely, to learn - like internships.
  • Requesters need to still publish during academic breaks! the tasks are 50% from industry and 50% from academia.
  • Requesters and workers need to find a solution together for a stable amount and quality of work available.

Readings

Turkopticon

What worker needs are discussed or implied by the reading? What requester needs are discussed or implied by the reading?

The authors observed the way AMT works, asked crowd workers how they would define their “Bill of Rights” and implemented a disruptive plugin for AMT that offers the possibility to rate the relationship workers had with requesters.

Workers

  • need to know which requester to trust: when workers browse available work in online labor marketplaces they would like to know more information about the requesters. They would like to know whether they should trust them or not.
  • need to speak loud and they need to be heard: workers have the need of sharing their experience with requesters to the community of crowd workers (i.e. advertise and evaluate their relationships with requesters). They need to be visible and express their opinion freely - without getting penalties for doing it.
  • need to feel protected and supported by the community.
  • need to feel respected and fairly treated by the requesters
    • fair acceptance / rejection
    • explained acceptance / rejection
    • fast payment
  • need (and like) to help each other.
  • need an effective mechanism to communicate with requesters.
  • need to have a similar infrastructure to what it exists in the traditional workplace. That´s why they think of minimum wage
  • need a balance between anonymity and reputation.
  • need to get responses from the platform and the requesters to questions and complaints.
  • need to see the platform managing injustices.
  • need to know about the 1) communicativity 2) generosity 3) fairness and 4) promptness of requesters

Turkopticon allows people to give a numeric rating on these qualities and they can also write in a free-form text further reviews of the requesters

(indirect observations)

  • workers need to receive specific instructions on how they should work on tasks

Requesters

  • need to find an evaluation system that filters accepted and rejected work easily as it is not feasible to go through each single response they get.
  • need to listen to the workers’ feedback in order to improve their microtasks and get more interested workforce, but they need an efficient method to process the workers’ comments and feedback, since they cannot spend hours on the email.
  • requesters need to be fairly evaluated.

(indirectly mentioned)

  • requesters need to define what is good and bad work for their microtasks before they assess workers’ answers.

Platforms

  • felt the need to support employers,because the platform works after all thanks to the employers who publish the work.

(the reading motivated these thoughts, but they don’t say this explicitly)

  • need to define a legal framework on which requesters and contributors agree.
  • need to take compromises in order to take into account a global scenario (workers and requesters may come from any country in the world).
  • need to be competitive.
  • need to be able to offer a sustainable amount of work and be able to recruit a sustainable workforce.
  • need to offer a (technologically) reliable service.

Reviews

  • need to be adapted if design changes occur in the crowdsouring process.
  • need to be objective (tactical quantification), simple and collective.
  • reviews need to be globally moderated.

Being a Turker

TODO:What worker needs are discussed or implied by the reading? What requester needs are discussed or implied by the reading?

Crowdsourcing User Studies with Mechanical Turk

TODO: What worker needs are discussed or implied by the reading? What requester needs are discussed or implied by the reading?

A Plea to Amazon: Fix Mechanical Turk

TODO: What worker needs are discussed or implied by the reading? What requester needs are discussed or implied by the reading?

The Need for Standardization in Crowdsourcing

TODO: What worker needs are discussed or implied by the reading? What requester needs are discussed or implied by the reading?

Synthesize the Needs You Found

Worker Needs

TODO: A set of bullet points of the needs of workers.

  • Example: workers need to be respected

Requester Needs

TODO: A set of bullet points of the needs of requesters.

  • Example: requesters need to trust the results they get from workers