Difference between revisions of "Milestone 3 taskforce"

From crowdresearch
Jump to: navigation, search
 
(One intermediate revision by the same user not shown)
Line 121: Line 121:
 
=== Trust-related Ideas ===
 
=== Trust-related Ideas ===
  
* [[Milestone 3 YourTeamName TrustIdea 1: Description of some trust-related idea]] - http://crowdresearch.meteor.com/posts/TaNbEkmeoSSiv5vDB
+
* [[Milestone_3_taskforce_trustidea1_norequesterspam]] - http://crowdresearch.meteor.com/out?url=http%3A%2F%2Fcrowdresearch.stanford.edu%2Fw%2Findex.php%3Ftitle%3DMilestone_3_taskforce_trustidea1_norequesterspam
* [[Milestone 3 YourTeamName TrustIdea 2: Description of another trust-related idea]] - http://crowdresearch.meteor.com/posts/ShEsGS3tx2Gdq9noJ
+
* [[Milestone_3_taskforce_trustidea2_transparency]] - http://crowdresearch.meteor.com/out?url=http%3A%2F%2Fcrowdresearch.stanford.edu%2Fw%2Findex.php%3Ftitle%3DMilestone_3_taskforce_trustidea2_transparency
  
 
=== Power-related Ideas ===
 
=== Power-related Ideas ===
  
* [[Milestone 3 YourTeamName PowerIdea 1: Description of some power-related idea]] - http://crowdresearch.meteor.com/posts/HEiG2GyJWLFPzThYc
+
* [[Milestone_3_taskforce_poweridea1_qualitycontrolmanagers]] - http://crowdresearch.meteor.com/out?url=http%3A%2F%2Fcrowdresearch.stanford.edu%2Fw%2Findex.php%3Ftitle%3DMilestone_3_taskforce_poweridea1_qualitycontrolmanagers
* [[Milestone 3 YourTeamName PowerIdea 2: Description of another power-related idea]] - http://crowdresearch.meteor.com/posts/qsv2bcRNjSL3PC8sC
+
  
== Dark Horse idea ==
+
* [[Milestone_3_taskforce_poweridea2_crowdactions]] - http://crowdresearch.meteor.com/out?url=http%3A%2F%2Fcrowdresearch.stanford.edu%2Fw%2Findex.php%3Ftitle%3DMilestone_3_taskforce_poweridea2_crowdactions
  
Describe your dark horse idea (using diagrams, sketches, storyboards, text, or some combination).
+
== Dark Horse idea ==
  
Please <strong>create a separate wiki page for your dark horse idea</strong> so we can link to it individually. Post the link on http://crowdresearch.meteor.com/category/milestone-3-dark-horse-ideas when done
 
  
* [[Milestone 3 YourTeamName DarkHorseIdea: Description of some dark horse idea]] - http://crowdresearch.meteor.com/posts/3xiwnx7ir35hgW3PS
+
* [[Milestone_3_taskforce_darkehorseidea_crowdbeers]] - http://crowdresearch.meteor.com/out?url=http%3A%2F%2Fcrowdresearch.stanford.edu%2Fw%2Findex.php%3Ftitle%3DMilestone_3_taskforce_darkehorseidea_crowdbeers

Latest revision as of 18:17, 18 March 2015

Template for your submission for Milestone 3. Do not edit this directly - instead, make a new page at Milestone 3 YourTeamName or whatever your team name is, and copy this template over. You can view the source of this page by clicking the Edit button at the top-right of this page, or by clicking here.

Initial Brainstorm

Trust-related Ideas

  • Creating a method that automatically identifies dishonest tasks posted by worker
    • it is not only crowd workers who may not behave honestly, there are also requesters posting tasks to for example, make crowd workers write fake good reviews.
    • there is a growing number of publications in research looking at this issue: http://goo.gl/hwBnpF
  • Letting requesters explicitly declare what the purpose of their task is (e.g. why they are collecting this information, what will happen with the information)
    • the requester needs to explain what the purpose of the task is.
    • the use of the crowdsourced data is limited to what has been declared a priori. The collected data must not be used for other purposes.
  • Instead of enabling trust based on evidence (i.e. I trust you because you showed before that you are reliable / good worker), building trust by direct communication
    • offer the opportunity to chat directly. Workers and requesters should be able to discuss via VoIP / Video chat services
    • create crowd job fairs (i.e. events in which requesters advertise the kind of work they offer and requesters go and see what kind of jobs they can get).There could be virtual stands for requesters, but also for crowd workers. In these fairs there could also be tutorials, idea walls, and contact information exchange. The main idea is that they all get to talk, solve their doubts, learn more and discover what kind of work is waiting out there.
  • Ensuring transparency. In a fair environment, in which information does not lead to any kind of conscious or unconscious bias, people would clearly show * who they are and be responsible for their actions. There is related work that found out that transparency has a positive impact in the results: http://goo.gl/sWaUzN
    • as a worker show your identity to requesters, other workers and marketplace
    • as a requester show your identity to workers, marketplace, (other requesters)
    • check veracity of identities with standard mechanisms like sending confirmation email
  • Enabling quick and transparent mechanisms to pay workers.
    • explain beforehand when they will pay and when not
    • after the work is done, allow workers to ask what they were not payed and explain them the reasons
  • Facilitating requester reimbursements in case of malicious activity or poor responses due to workers’ negligence.
    • Payment Window : A few hours or days during which workers work can be evaluated as honest/dishonest.
    • Such a facility will motivate workers to ensure their work is of high quality.
    • Will drastically reduce malicious activity.
  • Defining collective trust
    • it might be related to collective intelligence (Malone et al.)
    • trust groups of people working on particular types of tasks
  • Involving both workers and requesters in the task consumption phase.
    • Workers can bid for tasks they are interested in.
    • Requesters can hand-pick the workers they want.
    • This fosters a healthier trust setting.
  • Testing first, and then deciding
    • let the worker have a test on the task and predict whether he/she will be good in subsequent tasks (i.e. whether he/she is suitable for the task or not).
    • if someones shows to be suitable for the task, works on it and does a good job then pay.
    • if someone shows to be suitable for the task, works on it and does a bad job, then do not pay
  • Let the people work on the tasks and evaluate the task itself
    • simple smiley-based evaluation in different parts of the task
    • tasks and consequently requesters will be trusted only when they collect a high number of positive smileys


Power-related Ideas

Power to the requester

  • An invite system.
    • Workers get allocated batches of tasks.
    • They can work on all them, or break them into smaller pieces and delegate sub-batches to trusted workers.
    • We can consider for example a pyramidal structure:
      • Every “crowd trustee” is responsible of forwarding the final answers to the intermediate “crowd requester”; Up to the external requester.
  • Crowd workflow (turkomatic).
    • Let the crowd break up a complex task into smaller microtasks.
    • make the “workflow creator” be responsible of forwarding the end result.
  • Power Testing (Quiz Like Task Execution.. Frequent Feedback).
    • Workers deal with a significant amount of tests tasks.
      • Workers will be more careful when submitting.
    • Need for a reputation score.
    • Challenge: how to create such an amount? and who pays for it?

Power to the crowd

  • Reputation System (Stackoverflow like)
    • Reputation should be earned, rather than lost, like with mturk.
    • Reputation should be recoverable.
    • Make it harder for a requester to affect a worker’s score.
    • Merge this activity with the Crowd Supervisors (see below).
  • Undo
    • It happens that a worker answers by rushing.
    • Offer the possibility to revisit the last worker’s answers.
  • Crowd Feedback.
    • Crowd can (voluntarily) inspect previous answers.
      • 1 to make them better
      • Offer feedback.
  • VETO
    • In case of an a disagreement, forward to the crowd supervisor.
    • Rank the cases with most disagreement first.
    • Send responses directly otherwise.
  • Open Discussion (see ideas on truth serum from Boi Faltings’ work)
    • Be open about the answers of a given task repetition.
    • Make the workers discuss and reach an agreement.


Power to the community

  • Crowd Supervisor (inspired from Wikipedia organization)
    • A group of crowd supervisors “vet” regular workers.
    • Must be done for all the workers.
    • This group is vetted by requesters/platform/crowd-representatives.
  • Establish social roles and hierarchies
    • local president like politics
    • emerging social hierarchies
    • establish networks and price accordingly
      • by communication channels, electing people
      • based on results, every year elections for roles
      • collaboration, emerging structure, coalitions like wikipedia


Dive Deeper into Specific Ideas

For each of the 4 ideas (2 for trust, 2 for power), describe (using diagrams, sketches, storyboards, text, or some combination) the ideas in further detail.

Please create a separate wiki page for each of your ideas, so we can link to them individually. The title of the wiki page should be Milestone 3 followed by your team name and a description of the idea itself (ex: Milestone 3 YourTeamName TrustIdea 1: Automatic Pricing for Tasks based on Average Completion Time). Once done, post a link to each of your trust-related ideas to http://crowdresearch.meteor.com/category/milestone-3-trust-ideas and a link to each of your power-related ideas to http://crowdresearch.meteor.com/category/milestone-3-power-ideas when done, following the instructions at http://crowdresearch.meteor.com/posts/bXSNbqihjajASBQEL

Trust-related Ideas

Power-related Ideas

Dark Horse idea