Milestone 4

From crowdresearch
Revision as of 08:47, 21 March 2015 by Rajan (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Flare and focus! Our goal this week will be to take the flare phase from last week, focus it around a few themes, and then use each others' ideas to flare out once more to identify the points of view we like. Due date (PST): 11:59 pm 25th March 2015 for submission, 9 am 27th March 2015 for voting and commenting on others' ideas.

  • Youtube link of the meeting today: watch
  • Meeting 4 slideshow: pdf

Themes from Milestone 3

Based on your submissions for Milestone 3, we synthesized the following themes. Take a look at the high level patterns, since these will be the themes we'll cluster our brainstorm around.

Task clarity: How might workers+requesters work together help produce higher-quality task descriptions?


Results: How might workers+requesters work together to produce higher-quality results?


Resolution: How might we enable a fairer dispute resolution process and who's fault is it?


Empathy: How might we build empathy between workers and requesters?


Transparency: How might we make payment clear and transparent?


Reputation: How might we design better reputation systems?


Non-theme with interesting ideas


Recommended Readings

Coming up with good, novel visions and ideas is a crucial part of doing successful research, and reading other researchers' visions and ideas can help you come up with better ideas yourself. These readings discuss visions that crowdsourcing researchers have thought of related to future crowd marketplaces. This week's readings are optional and don't have a deliverable, but are highly recommended.

Mason W, Watts D J. Financial incentives and the performance of crowds. ACM SigKDD Explorations Newsletter, 2010, 11(2): 100-108.

Dow S, Kulkarni A, Klemmer S, et al. Shepherding the crowd yields better work. Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. ACM, 2012: 1013-1022.

Deliverables for Milestone 4

Take inspiration from ideas in Milestone 3

Rather than brainstorming broadly as with last week, now we'll and use these ideas we've generated to inspire a focused brainstorm around these themes.

As a team, choose three of the six theme areas that you are particularly interested in. Read through the ideas listed in the category.

One of the best ways to have insights is to explicitly go through analogical reasoning, where you compare and contrast related cases. So, for each of the three themes you've chosen, talk with your team about:

  • What do these ideas tend to have in common?
  • How do these ideas tend to differ?

Use your insights from this conversation to generate design axes like we discussed in last week's meeting (see slide notes if you'd like). Write up a short paragraph (2-4 sentences) for each theme summarizing the similarities, differences, and axes inspired.

Flare and focus: generate ideas for the themes

Now, for each of the three themes you've chosen, and armed with the knowledge from your colleagues' ideas and the "How might we" question that starts each theme, brainstorm with your team to generate more ideas related to that theme. Don't forget to do some dark horse brainstorming --- those ideas were often the most highly rated ones this week! Once you've generated a number of ideas for each theme, choose one idea per theme that you'd like to share with everyone. Ideally, your team will be inspired by the ideas listed above and generate new (and even better!) ideas. You may also want to take someone else's ideas from this past week and riff on them for your submission. Or, you may take one of your own previous ideas and iterate on it based on what you've learned.

As with last week, expand on the idea so others can understand it:

  • Create a succinct description of the problem that the theme is trying to address
  • In a sentence or two, credit the ideas that inspired or influenced your new idea.
  • Explain your idea and how it can be used to solve the problem you described in your problem statement. Consider how your idea will be used in context. Mention its limitations. Use sketches, charts, diagrams, tables and anything you think will help articulate your idea. Clarity and brevity are prized over detail.


Create a Wiki Page for your Team's Submission

Please create a page for your team's submission at (substituting in YourTeamName with the team name), copy over the template at Milestone 4 Template . If you have never created a wiki page before, please see this or watch this.

[Team Leaders] Submission or Post the links to your ideas until 25th March 11:59 pm

We have a service on which you can post the links to the wiki-pages for the individual ideas you generated, explore them, and upvote them.

Sign-up Instructions: Log in with either Twitter or Facebook. When it asks you to pick your username, pick the same username as your Slack, this will help us identify and track your contributions better.

There are 7 possible submission categories:

1- where you can post links to the wiki pages for any description-related ideas you generated.

2- where you can post links to the wiki pages for any 2 results-related ideas you generated.

3- where you can post links to the wiki pages for any empathy-related ideas you generated.

4- where you can post links to the wiki pages for any resolution-related ideas you generated.

5- where you can post links to the wiki pages for any transparency-related ideas you generated.

6- where you can post links to the wiki pages for any systems-related ideas you generated.

7- where you can post links to the wiki pages for any other ideas you generated.

Post links to your ideas only once they're finished. Give your posts titles which summarize your idea. Viewers should be able to get the main point by skimming the title ("Leveling System for Workers Leading to Bettter Wages" is a good title. "YourTeam Idea 1" is a bad title).

-Please submit your finished ideas by 11:59 pm 25th March 2015, and DO NOT vote/comment until 26th March 12:05 am

[Everyone] Peer-evaluation (upvote ones you like, comment on them) from 12:05 am 26th March until 9 am 27th March

Post submission phase, you are welcome to browse through, upvote, and comment on others' ideas. We encourage you especially to look at and comment on ideas that haven't yet gotten feedback, to make sure everybody's ideas gets feedback.

Step 1: Please use to find ideas that haven't yet gotten feedback, and to find ideas that haven't been yet been viewed many times.

Step 2: Once you find an idea of interest or less attended, please vote and comment upon it. Please perform this action from 3 to 5 ideas - this will help us balance the comments and votes. Please do not vote your team's idea. Once again, everyone is supposed to vote+comment, whether you're the team leader or not.

COMMENT BEST-PRACTICES: As on Crowdgrader, everybody reviews at least 3 ideas, supported by a comment. The comment has to justify your reason for upvote. The comment should be constructive, and should mention positive aspect of the idea worth sharing. Negative comments are discouraged, rather make your comment in the form of a suggestion - such as, if you disliked an idea, try to suggest improvements (do not criticize an idea, no idea is bad, every idea has a scope of improvement).

[Team Leaders] Milestone 4 Submissions

To help us track all submissions and browsing through them, once you have finished your Milestone 4, go to the link below and post the link:

Milestone 4 Submissions