Flare and focus! Our goal this week will be to take the flare phase from last week, focus it around a few themes, and then use each others' ideas to flare out once more to identify the points of view we like. Due date (PST): 11:59 pm 25th March 2015 for submission, 9 am 27th March 2015 for voting and commenting on others' ideas.
- 1 Themes from Milestone 3
- 1.1 Task clarity: How might workers+requesters work together help produce higher-quality task descriptions?
- 1.2 Results: How might workers+requesters work together to produce higher-quality results?
- 1.3 Resolution: How might we enable a fairer dispute resolution process?
- 1.4 Empathy: How might we build empathy between workers and requesters?
- 1.5 Transparency: How might we make payment clear and transparent?
- 1.6 Reputation: How might we design better reputation systems?
- 1.7 Non-theme with interesting ideas
- 2 Recommended Readings
- 3 Deliverables for Milestone 4
- 4 Submitting
Themes from Milestone 3
Based on your submissions for Milestone 3, we synthesized the following themes. Take a look at the high level patterns, since these will be the themes we'll cluster our brainstorm around.
Task clarity: How might workers+requesters work together help produce higher-quality task descriptions?
- New tasks go to workers first who improve the task before it goes live.
- Voting on hit design
- New tasks go into a holding pattern where they need to be "voted in” by workers
- Task templates (can we do better than AMT’s templates?)
- Stronger categorization of work
- Artificial turker that has to understand tasks before they go live
Results: How might workers+requesters work together to produce higher-quality results?
- Workers review other workers’ work (like on MobileWorks)
- Could we create a crowd contractor who is in charge of each submission getting good results?
Resolution: How might we enable a fairer dispute resolution process?
- Moderators can review if rejected work should be disputed (similar idea here)
- Dispute resolution process
- Bad actors get hellbanned to a separate circle for a time being, or their tasks done more slowly
- Force requesters to mediate previous problems before they can post new tasks moving forward
Empathy: How might we build empathy between workers and requesters?
- Meetings outside of the system to build trust
- Make workers act as requesters, and visa versa
- Requesters can send gifts to workers
- A certain amount of requester costs are reserved to pay bonuses to workers
- Humanizing worker profiles
Transparency: How might we make payment clear and transparent?
- Standardize task pricing
- Required minimum wage? if you stay above the 15th percentile of “good work throughput”
- Checkpoints where you get reviewed and paid after every N tasks
- Offer increased compensation to the first few people who take the task
Reputation: How might we design better reputation systems?
- Leveling up as a worker and requester: gives you better wages (Michael's thought there), first availability of tasks for workers, etc. Also worker levels
- Ranking top workers and making them most available
- Workers rate requesters as they do tasks, used to learn matching for other workers.
- Tasks are first available to workers who match according to skill and performance
Non-theme with interesting ideas
Coming up with good, novel visions and ideas is a crucial part of doing successful research, and reading other researchers' visions and ideas can help you come up with better ideas yourself. These readings discuss visions that crowdsourcing researchers have thought of related to future crowd marketplaces. This week's readings are optional and don't have a deliverable, but are highly recommended.
Deliverables for Milestone 4
Take inspiration from ideas in Milestone 3
Rather than brainstorming broadly as with last week, now we'll and use these ideas we've generated to inspire a focused brainstorm around these themes.
As a team, choose three of the six theme areas that you are particularly interested in. Read through the ideas listed in the category.
One of the best ways to have insights is to explicitly go through analogical reasoning, where you compare and contrast related cases. So, for each of the three themes you've chosen, talk with your team about:
- What do these ideas tend to have in common?
- How do these ideas tend to differ?
Use your insights from this conversation to generate design axes like we discussed in last week's meeting (see slide notes if you'd like). Write up a short paragraph (2-4 sentences) for each theme summarizing the similarities, differences, and axes inspired.
Flare and focus: generate ideas for the themes
Now, for each of the three themes you've chosen, and armed with the knowledge from your colleagues' ideas and the "How might we" question that starts each theme, brainstorm with your team to generate more ideas related to that theme. Don't forget to do some dark horse brainstorming --- those ideas were often the most highly rated ones this week! Once you've generated a number of ideas for each theme, choose one idea per theme that you'd like to share with everyone. Ideally, your team will be inspired by the ideas listed above and generate new (and even better!) ideas. You may also want to take someone else's ideas from this past week and riff on them for your submission. Or, you may take one of your own previous ideas and iterate on it based on what you've learned.
As with last week, expand on the idea so others can understand it:
- Create a succinct description of the problem that the theme is trying to address
- In a sentence or two, credit the ideas that inspired or influenced your new idea.
- Explain your idea and how it can be used to solve the problem you described in your problem statement. Consider how your idea will be used in context. Mention its limitations. Use sketches, charts, diagrams, tables and anything you think will help articulate your idea. Clarity and brevity are prized over detail.
Create a Wiki Page for your Team's Submission
Please create a page for your team's submission at http://crowdresearch.stanford.edu/w/index.php?title=Milestone_4_YourTeamName&action=edit (substituting in YourTeamName with the team name), copy over the template at Milestone 4 Template . If you have never created a wiki page before, please see this or watch this.
We have a service on which you can post the links to the wiki-pages for the individual ideas you generated, explore them, and upvote them.
Sign-up Instructions: Log in with either Twitter or Facebook. When it asks you to pick your username, pick the same username as your Slack, this will help us identify and track your contributions better.
There are 7 possible submission categories:
1- http://crowdresearch.meteor.com/category/milestone-4-descriptions where you can post links to the wiki pages for any description-related ideas you generated.
2- http://crowdresearch.meteor.com/category/milestone-4-results where you can post links to the wiki pages for any 2 results-related ideas you generated.
3- http://crowdresearch.meteor.com/category/milestone-4-empathy where you can post links to the wiki pages for any empathy-related ideas you generated.
4- http://crowdresearch.meteor.com/category/milestone-4-resolution where you can post links to the wiki pages for any resolution-related ideas you generated.
5- http://crowdresearch.meteor.com/category/milestone-4-transparency where you can post links to the wiki pages for any transparency-related ideas you generated.
6- http://crowdresearch.meteor.com/category/milestone-4-systems where you can post links to the wiki pages for any systems-related ideas you generated.
7- http://crowdresearch.meteor.com/category/milestone-4-other where you can post links to the wiki pages for any other ideas you generated.
Post links to your ideas only once they're finished. Give your posts titles which summarize your idea. Viewers should be able to get the main point by skimming the title ("Leveling System for Workers Leading to Bettter Wages" is a good title. "YourTeam Idea 1" is a bad title).
-Please submit your finished ideas by 11:59 pm 25th March 2015, and DO NOT vote/comment until 26th March 12:05 am
[Everyone] Peer-evaluation (upvote ones you like, comment on them) from 12:05 am 26th March until 9 am 27th March
Post submission phase, you are welcome to browse through, upvote, and comment on others' ideas. We encourage you especially to look at and comment on ideas that haven't yet gotten feedback, to make sure everybody's ideas gets feedback.
Step 1: Please use http://crowdresearch.meteor.com/needcomments to find ideas that haven't yet gotten feedback, and http://crowdresearch.meteor.com/needclicks to find ideas that haven't been yet been viewed many times.
Step 2: Once you find an idea of interest or less attended, please vote and comment upon it. Please perform this action from 3 to 5 ideas - this will help us balance the comments and votes. Please do not vote your team's idea. Once again, everyone is supposed to vote+comment, whether you're the team leader or not.
COMMENT BEST-PRACTICES: As on Crowdgrader, everybody reviews at least 3 ideas, supported by a comment. The comment has to justify your reason for upvote. The comment should be constructive, and should mention positive aspect of the idea worth sharing. Negative comments are discouraged, rather make your comment in the form of a suggestion - such as, if you disliked an idea, try to suggest improvements (do not criticize an idea, no idea is bad, every idea has a scope of improvement).
[Team Leaders] Milestone 4 Submissions
To help us track all submissions and browsing through them, once you have finished your Milestone 4, go to the link below and post the link: