Difference between revisions of "Crowdresearch:Winter Milestone 4 - Design (Test Flight) Alipta"

From crowdresearch
Jump to: navigation, search
 
(55 intermediate revisions by 4 users not shown)
Line 1: Line 1:
'''Note: The idea submissions for Milestone 3 have been archived at http://crowdresearch.meteor.com/ '''
+
'''Note''': The idea submissions for Milestone 2 can still be accessed http://crowdresearch.meteor.com/
  
 
'''Due date (PST): 8:00 pm 31st Jan 2016 for submission, 12 pm 1st Feb 2016 for peer-evaluation.'''
 
'''Due date (PST): 8:00 pm 31st Jan 2016 for submission, 12 pm 1st Feb 2016 for peer-evaluation.'''
Line 6: Line 6:
  
 
* Youtube link of the meeting today: [http://www.youtube.com/watch?v=BbzBAm73Slg watch]  
 
* Youtube link of the meeting today: [http://www.youtube.com/watch?v=BbzBAm73Slg watch]  
* Winter Meeting 3 slideshow: coming soon...
+
* Winter Meeting 3 slideshow: [[:Media:01-25-brainstorming.pdf| slides pdf]]
 +
 
 +
* Archive of Milestone 2 [http://crowdresearchw16-m2.meteor.com submissions]
  
 
== Needs from Milestone 2 ==
 
== Needs from Milestone 2 ==
  
We synthesized the main needs groups identified in Milestone 2 in the following table:
+
We synthesized the main needs identified in Milestone 2 below:
  
=== Worker Needs ===
+
=== Worker and Requester Needs ===
  
{| class="wikitable"
+
[[:Media:01-25-brainstorming.pdf| Please see this week's slides]]
|-
+
! rowspan = "2" | Needs
+
! colspan = "2" | Evidence
+
|-
+
! Observations
+
! Interpretations
+
|-
+
|Workers need to be able to quickly find tasks they'd want to work on
+
|Monday evening panel workers cite the challenge in identifying or distinguishing tasks because of poor tagging, Reddit discussion also cited the exorbitant amount of time that they spend trying to find tasks and do the mental calculations to find the opportunities that match them best (e.g. time to complete the task on average, average $ per minute on task, requirements to complete the task).
+
|Finding a task that matches well with the worker's skillset and pays well takes a significant amount of (unpaid) time.
+
|-
+
|Workers need to feel they are being fairly compensated for their work.
+
|Reddit discussion cites that the payment system for HITs is not adaptive and does not take into account changing marketplace conditions (supply/demand) and pricing of tasks based on those changes.
+
|Monetary compensation is the primary motivator for many crowd workers.
+
|-
+
|Workers need to feel like they are treated fairly and respectfully, and have a voice in the platform
+
|Comment on Turkopticon: "Got a mass rejection from some hits I did for them! Talked to other turkers that I know in real life and the same thing happened to them. There rejection comments are also really demeaning. Definitely avoid!"
+
|Unreasonable rejections and low payments lead workers to feel disrespected. The implicit assumption on MTurk is that workers are unskilled and replaceable. They can do little if their work is rejected.
+
|-
+
|Workers need to be able to expose their skills so they can get work they are qualified for and advance their skills
+
|Monday evening panel workers from oDesk cite that most employers will not work with them until they have enough feedback or past work on the platform
+
|If users cannot get new work without feedback, this makes it difficult for new users to establish their reputation and get jobs that will help develop their skillsets.
+
|-
+
|-
+
|Workers need to be confident that they understand the goal of the task, and quickly
+
|Workers avoid tasks that have ambiguous goals or might result in errors that get them rejected
+
|Workers become risk-averse when a task might be confusing enough to threaten their reputation or payment
+
|-
+
|}
+
  
=== Requester Needs ===
+
=== Our synthesis ===
 +
These needs boil down to two main issues: 1) task ranking and reputation, and 2) worker and requester representation (open governance).
  
{| class="wikitable"
+
'''Reputation''':
|-
+
* How to improve reputation system beyond Boomerang? ([[:Media:Daemo white paper (private).pdf|read this paper to learn more about Boomerang]]).  
! rowspan = "2" | Needs
+
* How would reputation work for new comers - workers or requesters?
! colspan = "2" | Evidence
+
* How do workers find relevant work, and requesters find ideal worker? 
|-
+
! Observations
+
! Interpretations
+
|-
+
|Requesters need to get their HITs completed (quickly / correctly)
+
|Requester asking on forum why nobody is doing his HITs (7-minute, 25-cent surveys - a very low wage)
+
|Requesters want their HITs done, and when nobody's doing them, they do not know the reason why (e.g. it is because he is underpaying workers)
+
|-
+
|Requesters need to be able to trust the results they get
+
|Requesters will often rely on previous workers whose results they can trust, and add mechanisms to detect spammers, or manually verify some results.
+
|If spammers are not caught, this brings the correctness of results into question. If requesters are not sure the results are correct, they may need to discard the data.
+
|-
+
|Requesters need to have workers who have the appropriate skills and demographics do their tasks
+
|Requesters worry that they are not able to verify self-reported demographics for surveys.
+
|Workers' self-reported skills and demographics are often not viewed as trustworthy. This is a problem for surveys, which need to have correct demographic data to be useful.
+
|-
+
|Requesters need to be able to easily generate good tasks
+
|Companies hire full-time developers to deal with the complexities of posting microtasks on MTurk. Requesters often develop their own tools and workflow systems on top of Amazon's.
+
|The process of authoring HITs is currently difficult and makes crowd-work inaccessible to potential requesters
+
|-
+
|Requesters need to price their tasks appropriately
+
|Requesters asking on forums about the appropriate amount they should pay for their HITs
+
|Requesters often don't have a good intuition of what the appropriate wage for their task would be in terms of price per HIT.
+
|-
+
|Requesters need workers to trust them
+
|Requesters say they are reluctant to reject work, because they fear they might get bad reviews.
+
|Workers are more likely to do HITs if the requester seems trustworthy. Requesters do not want bad reviews, because they may result in workers ignoring the requester's HITs
+
|-
+
|}
+
  
=== Michael Bernstein's synthesis ===
+
'''Representation''':
These needs boil down to two main issues: 1) trust, and 2) power.
+
* How to make your voice heard?
 +
* How to represent concerns/rights as workers and requesters? Who has the power to post work?
 +
* What would open governance look like on Daemo?
  
Trust:
+
As we brainstorm, we should be thinking about solutions that holistically address these issues of reputation and open governance, not just surface fixes that get at micro-elements of the system.
* How do I trust who you say you are?
+
* How do I trust that the results I get are results that will be good?
+
* How do I trust that you’ll respect me as a worker, and pay me accordingly?
+
  
Power:
+
== Dive Deeper into Specific Ideas (Mandatory) ==
* Who has the power to post work?
+
* To edit other peoples’ posted work?
+
* To return results to the requester? Can I, as a worker, send it back myself, or does someone else need to vet it?
+
  
As we brainstorm, we should be thinking about solutions that holistically address these issues of power and trust, not just surface fixes that get at micro-elements of the system.
+
For the two topics - reputation and representation, pitch an idea for each, that you would like to pursue, and expand on them a bit further. In addition to describing the idea itself, make sure you also tell us:
  
== Recommended Readings ==
+
* What are the goals of the design? For example, Google's Android design goals are: delight me in surprising ways, simplify my life, and make me amazing (e.g., grant me special powers).
 +
* Which aspects of your design reflect each goal? How does your design solution address the users' needs?
  
Coming up with good, novel visions and ideas is a crucial part of doing successful research, and reading other researchers' visions and ideas can help you come up with better ideas yourself. These readings discuss visions that crowdsourcing researchers have thought of related to future crowd marketplaces. This week's readings are optional and don't have a deliverable, but are highly recommended.
+
=== Deliverable ===
  
=== Design notes for a future crowd work market ===
+
For each of the 2 ideas (1 for reputation, 1 for representation), describe (using diagrams, sketches, storyboards, text, or some combination) the ideas in further detail.
  
[https://medium.com/@silberman/design-notes-for-a-future-crowd-work-market-2d7557105805 Design notes for a future crowd work market] - This is a Medium post written by researchers involved with Turkopticon in response to hearing about this research project. It discusses their vision for a future crowd marketplace, where workers are more involved in the management of the marketplace.
+
Please <strong>create a separate wiki page for each of your ideas</strong>, so we can link to them individually. The title of the wiki page should be WinterMilestone 3 followed by your team name and a description of the idea itself (ex: [[WinterMilestone 3 YourTeamName ReputationIdea: Automatic Suggestion for Tasks based on Average Completion Time]]). Post a link to each of your reputation-related ideas to http://crowdresearch.meteor.com/category/wintermilestone-3-reputation-ideas and a link to each of your representation-related ideas to http://crowdresearch.meteor.com/category/wintermilestone-3-representation-ideas when done.
  
=== The future of crowd work ===
+
== Dark Horse idea (Mandatory) ==
  
[[:Media:The_future_of_crowd_work_(private).pdf | Kittur A, Nickerson J V, Bernstein M, et al. The future of crowd work. Proceedings of the 2013 conference on Computer supported cooperative work. ACM, 2013: 1301-1318.]] - This paper envisions a future crowd marketplace that emphasizes on workers' long-term development, and where people can be proud to be workers. It is a long paper. Feel free to focus on just the parts that particularly interest you.
+
Now that you've identified some design directions you like, it's time to change tack and toss in a dark horse idea. A dark horse, in horse racing, is a contender who most people don't think will win, but may turn in an unexpectedly strong performance and produce a huge payoff. Dark horse ideas are intended to be something far out there or nearly impossible. In the best case, your dark horse ideas might end up winning the race. However, even in the worst case, they can give us tremendous design insight and prevent design fixation, where the design space shrinks too rapidly.
  
== Initial Brainstorm ==
+
There are three requirements for dark horse ideas. '''First''', they must be "dark": they must explore a space that is risky, radical, infeasible, and/or in a direction orthogonal to previously explored solutions. They should feel slightly uncomfortable. '''Second''', they must be brainstormed after the more traditional ideas — you can't have a dark path without a traditional "light" path to contrast it against. '''Third''', they must be refined enough that they could be prototyped and objectively tested. That is, it cannot be infeasible: it needs to be something that we could put in front of real people to see whether it would work.
  
Now it's time for your team to brainstorm some ideas based on these needs.
+
If you have dark horse ideas that came up in your initial brainstorm, you can use that. If you're not satisfied, brainstorm some! Try using [https://dschool.stanford.edu/groups/k12/wiki/faf1d/Powers_of_Ten.html Powers of Ten] and other techniques to push further and generate even more. After you brainstorm and sketch out dark horse ideas, choose one that you'd like to include among your set of two top candidates from before. Expand on your dark horse idea like you did in the previous section.
  
Work with your team to brainstorm as many ideas as you can under two headings: '''trust''' and '''power'''. Here are some examples of "[[:Media:How_might_we.pdf | How might we]]" questions (a technique which can inspire specific brainstorms) which can drive the generation of your ideas:
+
=== Deliverable ===
  
* “[[:Media:How_might_we.pdf | How might we]]” enable workers to trust the requester’s intention to pay?”
+
Describe your dark horse idea (using diagrams, sketches, storyboards, text, or some combination).
* “How might we” enable requesters to trust the results they get back?
+
  
Use whatever tools you need - if you can get together in-person, whiteboards and sketchbooks are great tools, while if you're a remote team, services like [http://docs.google.com Google Docs] and [http://sketchboard.io sketchboard.io] should help you with the brainstorming process.
+
Please create a separate wiki page for your dark horse idea so we can link to it individually. Post the link on http://crowdresearch.meteor.com/category/wintermilestone-3-dark-horse-ideas when done
  
Sketch out enough ideas until you find a set that you’re inspired to explore further. It should be at least 20 ideas total - 10 ideas for trust, and 10 ideas for power. This brainstorm should be wild and broad. Focus not on usability patches, but deeper design innovations.
+
== Research Engineering (All) ==
 +
Great job on the issues from last week.
 +
For those of you who have been doing tutorials and reading documentation now it's time to put that into practice.
 +
This week we will improve and finalize some of the issues we started last week, and we will also complete new ones.
  
=== Deliverable ===
+
- issue-forms (done), issue-profile (almost there, finalizing), issue-csv (cont'd this week) and issue-task (to be finalized)
  
The ideas you brainstormed, (at least 10 ideas for trust, and at least 10 ideas for power). Provide them in whatever format you want - diagrams, sketches, descriptions, or a combination (the wiki supports images, [http://www.mediawiki.org/wiki/Help:Images see here] for instructions on uploading them).
+
- new issues: #645 (timer), #646 (advanced project properties), #647 (template item data_source), #648 (bug, task creation)
  
== Dive Deeper into Specific Ideas ==
 
  
Select 2 ideas per heading (trust, power) that you would like to pursue, and expand on them a bit further. In addition to describing the idea itself, make sure you also tell us:
+
announce in #research-engineering that you are working on a particular issue and please let the others know about the progress of the issues you are working on (so that we don't do duplicate work). You are encouraged to work together.
  
* What are the goals of the design? For example, Google's Android design goals are: delight me in surprising ways, simplify my life, and make me amazing (e.g., grant me special powers).
+
For any questions ping @aginzberg, @dmorina, and @shirish.goyal on Slack #research-engineering
* Which aspects of your design reflect each goal? How does your design solution addresses the users' needs?
+
  
=== Deliverable ===
+
== Usability Experience (Test Flight) ==
  
For each of the 4 ideas (2 for trust, 2 for power), describe (using diagrams, sketches, storyboards, text, or some combination) the ideas in further detail.
+
You may have played with [https://daemo-playground.herokuapp.com Daemo's live version], but never got a chance to reflect on your experiences. That's your first test flight. Please login to the website, and try to work on it. Now that you have basic experience working on a crowd platform - MTurk, Microworkers etc; you may be able to add to your experience further.
  
Please <strong>create a separate wiki page for each of your ideas</strong>, so we can link to them individually. The title of the wiki page should be Milestone 3 followed by your team name and a description of the idea itself (ex: [[Milestone 3 YourTeamName TrustIdea 1: Automatic Pricing for Tasks based on Average Completion Time]]). Post a link to each of your trust-related ideas to http://crowdresearch.meteor.com/category/milestone-3-trust-ideas and a link to each of your power-related ideas to http://crowdresearch.meteor.com/category/milestone-3-power-ideas when done.
+
=== Deliverable ===
  
== Dark Horse idea ==
+
Reach out to people within Crowd Research community, who are not participating in this test flight and get them to use a specific feature of Daemo and get their feedback on it. You are free to reach out to people outside Daemo too, but they must have minimal crowd platform experience and don't share it publicly. We're not ready for huge traffic at this point. Report a synthesis of your findings along with your own experience. Your report should include '''1) actionable item''', one that can be implemented into Daemo quickly; '''2) good-to-have long term vision item''', one that you'd like to see at some point of time. There is no limit to how many people you work with, it can be 1, it can be 10, it can be none (means you reflect your own findings). '''Note''': be respectful of others time, and don't push people for it - you can always report your own experience only.
  
Now that you've identified some design directions you like, it's time to change tack and toss in a dark horse idea. A dark horse, in horse racing, is a contender who most people don't think will win, but may turn in an unexpectedly strong performance and produce a huge payoff. Dark horse ideas are intended to be something far out there or nearly impossible. In the best case, your dark horse ideas might end up winning the race. However, even in the worst case, they can give us tremendous design insight and prevent design fixation, where the design space shrinks too rapidly.
+
For questions/discussions, go to #usability-testflight. Your DRI for this milestone is @catherine.mullings
  
There are three requirements for dark horse ideas. First, they must be "dark": they must explore a space that is risky, radical, infeasible, and/or in a direction orthogonal to previously explored solutions. They should feel slightly uncomfortable. Second, they must be brainstormed after the more traditional ideas — you can't have a dark path without a traditional "light" path to contrast it against. Third, they must be refined enough that they could be prototyped and objectively tested. That is, it cannot be infeasible: it needs to be something that we could put in front of real people to see whether it would work.
+
== Recommended Readings (Optional) ==
  
If you have dark horse ideas that came up in your initial brainstorm, you can use that. If you're not satisfied, brainstorm some! Try using [https://dschool.stanford.edu/groups/k12/wiki/faf1d/Powers_of_Ten.html Powers of Ten] and other techniques to push further and generate even more. After you brainstorm and sketch out dark horse ideas, choose one that you'd like to include among your set of two top candidates from before. Expand on your dark horse idea like you did in the previous section.
+
Coming up with good, novel visions and ideas is a crucial part of doing successful research, and reading other researchers' visions and ideas can help you come up with better ideas yourself. These readings discuss visions that crowdsourcing researchers have thought of related to future crowd marketplaces. This week's readings are optional and don't have a deliverable, but are highly recommended.
  
=== Deliverable ===
+
=== Design notes for a future crowd work market ===
  
Describe your dark horse idea (using diagrams, sketches, storyboards, text, or some combination).
+
[https://medium.com/@silberman/design-notes-for-a-future-crowd-work-market-2d7557105805 Design notes for a future crowd work market] - This is a Medium post written by researchers involved with Turkopticon in response to hearing about this research project. It discusses their vision for a future crowd marketplace, where workers are more involved in the management of the marketplace.
  
Please create a separate wiki page for your dark horse idea so we can link to it individually. Post the link on http://crowdresearch.meteor.com/category/milestone-3-dark-horse-ideas when done
+
=== The future of crowd work ===
 +
 
 +
[[:Media:The_future_of_crowd_work_(private).pdf | Kittur A, Nickerson J V, Bernstein M, et al. The future of crowd work. Proceedings of the 2013 conference on Computer supported cooperative work. ACM, 2013: 1301-1318.]] - This paper envisions a future crowd marketplace that emphasizes on workers' long-term development, and where people can be proud to be workers. It is a long paper. Feel free to focus on just the parts that particularly interest you.
  
 
== Submitting ==
 
== Submitting ==
Line 156: Line 100:
 
=== Create a Wiki Page for your Team's Submission ===
 
=== Create a Wiki Page for your Team's Submission ===
  
Please create a page for your team's submission at http://crowdresearch.stanford.edu/w/index.php?title=Milestone_3_YourTeamName&action=edit (substituting in YourTeamName with the team name), copy over the template at <strong>[[Milestone 3 Template]] </strong>. If you have never created a wiki page before, please see [http://www.mediawiki.org/wiki/Help:Starting_a_new_page this] or watch [https://www.youtube.com/watch?v=83-lCpAnaFw this].
+
Please create 5 wiki pages for each topic - 1) reputation, 2) representation, 3) dark horse, 4) actionable and 5) long-term. See the template here for more details on "how to title and link different pages" <strong>[[WinterMilestone 3 Template]] </strong>. If you have never created a wiki page before, please see [http://www.mediawiki.org/wiki/Help:Starting_a_new_page this] or watch [https://www.youtube.com/watch?v=83-lCpAnaFw this].
 
+
=== [Team Leaders] Submission or Post the links to your ideas until 18th March 11:59 pm===
+
 
+
We have a service on which you can post the links to the wiki-pages for the individual ideas you generated, explore them, and upvote them.
+
  
Instructions for posting are at http://crowdresearch.meteor.com/posts/bXSNbqihjajASBQEL
+
=== [Team Representative] Submission or Post the links to your ideas until 8:00 pm 31st Jan 2016===
  
<strong>Sign-up Instructions: </strong> Log in with either Twitter or Facebook. When it asks you to pick your username, pick the ''same username as your Slack'', this will help us identify and track your contributions better.  
+
We have a [[http://crowdresearchw16-m3.meteor.com/ Reddit like service]] on which you can post the links to the wiki-pages for the submissions, explore them, and upvote them.
  
There are 3 submission categories:
+
<strong>NEW INSTRUCTIONS - Create 5 wiki pages</strong>
  
1- http://crowdresearch.meteor.com/category/milestone-3-trust-ideas where you can post links to the wiki pages for each of the 2 trust-related ideas you generated in the "Dive Deeper into Specific Ideas" stage
+
Please use the same login avenue (Facebook, Twitter, or email address) as you’ve done in the past with Meteor​. This will help us identify and track your contributions better.
  
2- http://crowdresearch.meteor.com/category/milestone-3-power-ideas where you can post links to the wiki pages for each of the 2 power-related ideas you generated in the "Dive Deeper into Specific Ideas" stage
+
For newcomers joining Crowd Research, when it asks you to pick your username, pick the ''same username as your Slack''.
  
3- http://crowdresearch.meteor.com/category/milestone-3-dark-horse-ideas where you can post a link to the wiki page for your dark horse idea
+
On Meteor, there are 5 submission categories, for each, create a separate wiki page - see [[WinterMilestone 3 Template]] for tips on how to set the title of your submission:
  
Post links to your ideas <strong>only once they're finished</strong>. Give your posts titles which summarize your idea. Viewers should be able to <strong>get the main point by skimming the title</strong> ("Automatic Pricing for Tasks based on Average Completion Time" is a good title. "YourTeam TrustIdea 1" is a bad title).
+
1- [Mandatory] http://crowdresearchw16-m3.meteor.com/?cat=reputation where you can post link to the wiki page for 1 reputation related idea you generated in the "Dive Deeper into Specific Ideas" stage
  
-<strong>Please submit your finished ideas by 11:59 pm 18th March 2015, and DO NOT vote/comment until 19th March 12:05 am</strong>
+
2- [Mandatory] http://crowdresearchw16-m3.meteor.com/?cat=representation where you can post link to the wiki page for 1 representation related idea you generated in the "Dive Deeper into Specific Ideas" stage
  
=== [Everyone] Peer-evaluation (upvote ones you like, comment on them) from 12:05 am 19th March until 9 am 20th March ===
+
3- [Mandatory] http://crowdresearchw16-m3.meteor.com/?cat=dark-horse where you can post a link to the wiki page for your dark horse idea
  
Post submission phase, you are welcome to browse through, upvote, and comment on others' ideas. We encourage you especially to look at and comment on ideas that haven't yet gotten feedback, to make sure everybody's ideas gets feedback. You can use http://crowdresearch.meteor.com/needcomments to find ideas that haven't yet gotten feedback, and http://crowdresearch.meteor.com/needclicks to find ideas that haven't been yet been viewed many times.
+
4- [Test flight] http://crowdresearchw16-m3.meteor.com/?cat=design-test-flight-actionable-ideas where you can post actionable ideas based on your usability study/experience.  
  
<strong>COMMENT BEST-PRACTICES</strong>: As on Crowdgrader, <strong>everybody</strong> reviews at least 3 ideas, supported by a comment. The comment has to justify your reason for upvote. The comment should be constructive, and should mention positive aspect of the idea worth sharing. Negative comments are discouraged, rather make your comment in the form of a suggestion - such as, if you disliked an idea, try to suggest improvements (do not criticize an idea, no idea is bad, every idea has a scope of improvement).
+
5- [Test flight] http://crowdresearchw16-m3.meteor.com/?cat=design-test-flight-long-term-idea where you can post long term ideas based on your usability study/experience.
  
=== [Team Leaders] Milestone 3 Submissions ===
+
'''Give your posts titles which summarize your idea. Viewers should be able to <strong>get the main point by skimming the title</strong> ("Automatic Suggestion for Tasks based on Average Completion Time" is a good title. "YourTeam TrustIdea 1" is a bad title).'''
  
To help us track all submissions and browsing through them, once you have finished your Milestone 3, go to the link below and post the link:
+
-<strong>Please submit your finished ideas by 8:00 pm 31st January 2016, and DO NOT vote/comment until then</strong>
  
[[Milestone 3 Submissions]]
+
=== [Everyone] Peer-evaluation (upvote ones you like, comment on them) from 8:05 pm 31st Jan until 12 pm 1st Feb 2016 ===
  
== Fill out this week's survey ==
+
Post submission phase, you are welcome to browse through, upvote, and comment on others' ideas. We encourage you especially to look at and comment on ideas that haven't yet gotten feedback, to make sure everybody's ideas gets feedback.
  
Please provide your feedback on this week's meeting and milestone so we can improve it, by filling out this [https://docs.google.com/forms/d/1ERZ_fg5N3Pfzg8WC63-jj8GFOej7K-zPTA4a21WtLwo/viewform?usp=send_form survey]
+
<strong>COMMENT BEST-PRACTICES</strong>: Everybody in the team reviews at least 3 ideas, supported by a comment. The comment has to justify your reason for upvote. The comment should be constructive, and should mention positive aspect of the idea worth sharing. Negative comments are discouraged, rather make your comment in the form of a suggestion - such as, if you disliked an idea, try to suggest improvements (do not criticize an idea, no idea is bad, every idea has a scope of improvement).

Latest revision as of 10:46, 7 February 2016

Note: The idea submissions for Milestone 2 can still be accessed http://crowdresearch.meteor.com/

Due date (PST): 8:00 pm 31st Jan 2016 for submission, 12 pm 1st Feb 2016 for peer-evaluation.

This week, we will take the set of needs that we collectively identified in the previous milestone and use those insights to generate design ideas.

  • Youtube link of the meeting today: watch
  • Winter Meeting 3 slideshow: slides pdf

Needs from Milestone 2

We synthesized the main needs identified in Milestone 2 below:

Worker and Requester Needs

Please see this week's slides

Our synthesis

These needs boil down to two main issues: 1) task ranking and reputation, and 2) worker and requester representation (open governance).

Reputation:

  • How to improve reputation system beyond Boomerang? (read this paper to learn more about Boomerang).
  • How would reputation work for new comers - workers or requesters?
  • How do workers find relevant work, and requesters find ideal worker?

Representation:

  • How to make your voice heard?
  • How to represent concerns/rights as workers and requesters? Who has the power to post work?
  • What would open governance look like on Daemo?

As we brainstorm, we should be thinking about solutions that holistically address these issues of reputation and open governance, not just surface fixes that get at micro-elements of the system.

Dive Deeper into Specific Ideas (Mandatory)

For the two topics - reputation and representation, pitch an idea for each, that you would like to pursue, and expand on them a bit further. In addition to describing the idea itself, make sure you also tell us:

  • What are the goals of the design? For example, Google's Android design goals are: delight me in surprising ways, simplify my life, and make me amazing (e.g., grant me special powers).
  • Which aspects of your design reflect each goal? How does your design solution address the users' needs?

Deliverable

For each of the 2 ideas (1 for reputation, 1 for representation), describe (using diagrams, sketches, storyboards, text, or some combination) the ideas in further detail.

Please create a separate wiki page for each of your ideas, so we can link to them individually. The title of the wiki page should be WinterMilestone 3 followed by your team name and a description of the idea itself (ex: WinterMilestone 3 YourTeamName ReputationIdea: Automatic Suggestion for Tasks based on Average Completion Time). Post a link to each of your reputation-related ideas to http://crowdresearch.meteor.com/category/wintermilestone-3-reputation-ideas and a link to each of your representation-related ideas to http://crowdresearch.meteor.com/category/wintermilestone-3-representation-ideas when done.

Dark Horse idea (Mandatory)

Now that you've identified some design directions you like, it's time to change tack and toss in a dark horse idea. A dark horse, in horse racing, is a contender who most people don't think will win, but may turn in an unexpectedly strong performance and produce a huge payoff. Dark horse ideas are intended to be something far out there or nearly impossible. In the best case, your dark horse ideas might end up winning the race. However, even in the worst case, they can give us tremendous design insight and prevent design fixation, where the design space shrinks too rapidly.

There are three requirements for dark horse ideas. First, they must be "dark": they must explore a space that is risky, radical, infeasible, and/or in a direction orthogonal to previously explored solutions. They should feel slightly uncomfortable. Second, they must be brainstormed after the more traditional ideas — you can't have a dark path without a traditional "light" path to contrast it against. Third, they must be refined enough that they could be prototyped and objectively tested. That is, it cannot be infeasible: it needs to be something that we could put in front of real people to see whether it would work.

If you have dark horse ideas that came up in your initial brainstorm, you can use that. If you're not satisfied, brainstorm some! Try using Powers of Ten and other techniques to push further and generate even more. After you brainstorm and sketch out dark horse ideas, choose one that you'd like to include among your set of two top candidates from before. Expand on your dark horse idea like you did in the previous section.

Deliverable

Describe your dark horse idea (using diagrams, sketches, storyboards, text, or some combination).

Please create a separate wiki page for your dark horse idea so we can link to it individually. Post the link on http://crowdresearch.meteor.com/category/wintermilestone-3-dark-horse-ideas when done

Research Engineering (All)

Great job on the issues from last week. For those of you who have been doing tutorials and reading documentation now it's time to put that into practice. This week we will improve and finalize some of the issues we started last week, and we will also complete new ones.

- issue-forms (done), issue-profile (almost there, finalizing), issue-csv (cont'd this week) and issue-task (to be finalized)

- new issues: #645 (timer), #646 (advanced project properties), #647 (template item data_source), #648 (bug, task creation)


announce in #research-engineering that you are working on a particular issue and please let the others know about the progress of the issues you are working on (so that we don't do duplicate work). You are encouraged to work together.

For any questions ping @aginzberg, @dmorina, and @shirish.goyal on Slack #research-engineering

Usability Experience (Test Flight)

You may have played with Daemo's live version, but never got a chance to reflect on your experiences. That's your first test flight. Please login to the website, and try to work on it. Now that you have basic experience working on a crowd platform - MTurk, Microworkers etc; you may be able to add to your experience further.

Deliverable

Reach out to people within Crowd Research community, who are not participating in this test flight and get them to use a specific feature of Daemo and get their feedback on it. You are free to reach out to people outside Daemo too, but they must have minimal crowd platform experience and don't share it publicly. We're not ready for huge traffic at this point. Report a synthesis of your findings along with your own experience. Your report should include 1) actionable item, one that can be implemented into Daemo quickly; 2) good-to-have long term vision item, one that you'd like to see at some point of time. There is no limit to how many people you work with, it can be 1, it can be 10, it can be none (means you reflect your own findings). Note: be respectful of others time, and don't push people for it - you can always report your own experience only.

For questions/discussions, go to #usability-testflight. Your DRI for this milestone is @catherine.mullings

Recommended Readings (Optional)

Coming up with good, novel visions and ideas is a crucial part of doing successful research, and reading other researchers' visions and ideas can help you come up with better ideas yourself. These readings discuss visions that crowdsourcing researchers have thought of related to future crowd marketplaces. This week's readings are optional and don't have a deliverable, but are highly recommended.

Design notes for a future crowd work market

Design notes for a future crowd work market - This is a Medium post written by researchers involved with Turkopticon in response to hearing about this research project. It discusses their vision for a future crowd marketplace, where workers are more involved in the management of the marketplace.

The future of crowd work

Kittur A, Nickerson J V, Bernstein M, et al. The future of crowd work. Proceedings of the 2013 conference on Computer supported cooperative work. ACM, 2013: 1301-1318. - This paper envisions a future crowd marketplace that emphasizes on workers' long-term development, and where people can be proud to be workers. It is a long paper. Feel free to focus on just the parts that particularly interest you.

Submitting

Create a Wiki Page for your Team's Submission

Please create 5 wiki pages for each topic - 1) reputation, 2) representation, 3) dark horse, 4) actionable and 5) long-term. See the template here for more details on "how to title and link different pages" WinterMilestone 3 Template . If you have never created a wiki page before, please see this or watch this.

[Team Representative] Submission or Post the links to your ideas until 8:00 pm 31st Jan 2016

We have a [Reddit like service] on which you can post the links to the wiki-pages for the submissions, explore them, and upvote them.

NEW INSTRUCTIONS - Create 5 wiki pages

Please use the same login avenue (Facebook, Twitter, or email address) as you’ve done in the past with Meteor​. This will help us identify and track your contributions better.

For newcomers joining Crowd Research, when it asks you to pick your username, pick the same username as your Slack.

On Meteor, there are 5 submission categories, for each, create a separate wiki page - see WinterMilestone 3 Template for tips on how to set the title of your submission:

1- [Mandatory] http://crowdresearchw16-m3.meteor.com/?cat=reputation where you can post link to the wiki page for 1 reputation related idea you generated in the "Dive Deeper into Specific Ideas" stage

2- [Mandatory] http://crowdresearchw16-m3.meteor.com/?cat=representation where you can post link to the wiki page for 1 representation related idea you generated in the "Dive Deeper into Specific Ideas" stage

3- [Mandatory] http://crowdresearchw16-m3.meteor.com/?cat=dark-horse where you can post a link to the wiki page for your dark horse idea

4- [Test flight] http://crowdresearchw16-m3.meteor.com/?cat=design-test-flight-actionable-ideas where you can post actionable ideas based on your usability study/experience.

5- [Test flight] http://crowdresearchw16-m3.meteor.com/?cat=design-test-flight-long-term-idea where you can post long term ideas based on your usability study/experience.

Give your posts titles which summarize your idea. Viewers should be able to get the main point by skimming the title ("Automatic Suggestion for Tasks based on Average Completion Time" is a good title. "YourTeam TrustIdea 1" is a bad title).

-Please submit your finished ideas by 8:00 pm 31st January 2016, and DO NOT vote/comment until then

[Everyone] Peer-evaluation (upvote ones you like, comment on them) from 8:05 pm 31st Jan until 12 pm 1st Feb 2016

Post submission phase, you are welcome to browse through, upvote, and comment on others' ideas. We encourage you especially to look at and comment on ideas that haven't yet gotten feedback, to make sure everybody's ideas gets feedback.

COMMENT BEST-PRACTICES: Everybody in the team reviews at least 3 ideas, supported by a comment. The comment has to justify your reason for upvote. The comment should be constructive, and should mention positive aspect of the idea worth sharing. Negative comments are discouraged, rather make your comment in the form of a suggestion - such as, if you disliked an idea, try to suggest improvements (do not criticize an idea, no idea is bad, every idea has a scope of improvement).