Difference between revisions of "Winter Milestone 6"

From crowdresearch
Jump to: navigation, search
(Task authoring)
Line 48: Line 48:
  
 
== Open Gov and design ==
 
== Open Gov and design ==
 +
 +
== Submission ==
 +
 +
=== Create a Wiki Page for your Team's Submission ===
 +
 +
Create a wiki page with either methods or systems section, diving deep into one the three themes ('''look at the template here: [[Winter Milestone 5 Templates]]'''). If you're participating in design test run, create another wiki and paste screenshots or mockup/wireframe files. If you have never created a wiki page before, please see [http://www.mediawiki.org/wiki/Help:Starting_a_new_page this] or watch [https://www.youtube.com/watch?v=83-lCpAnaFw this].
 +
 +
=== [Team Representative] Submission or Post the links to your ideas until 8:00 pm 14th Feb 2016===
 +
 +
We have a [[http://crowdresearch.meteor.com/ Reddit like service]] on which you can post the links to the wiki-pages for the submissions, explore them, and upvote them.
 +
 +
Please use the same login avenue (Facebook, Twitter, or email address) as you’ve done in the past with Meteor​. This will help us identify and track your contributions better.
 +
 +
For newcomers joining Crowd Research, when it asks you to pick your username, pick the ''same username as your Slack''. '''Please DO NOT forget to mention the milestone contributors' slackid below each wiki page.'''
 +
 +
On Meteor, there are 4 submission categories, 3 for research theme, and 1 for design test run.
 +
 +
1- [One of three mandatory] http://crowdresearch.meteor.com/category/task-rank where you can post a link to the wiki page for your task ranking proposal
 +
 +
2- [One of three mandatory] http://crowdresearch.meteor.com/category/task-author where you can post a link to the wiki page for your task author proposal
 +
 +
3- [One of three mandatory] http://crowdresearch.meteor.com/category/open-gov where you can post a link to the wiki page for your open gov proposal
 +
 +
4- [Test flight] http://crowdresearch.meteor.com/category/design-messenger where you can post mockup/wireframe of the Daemo messenger.
 +
 +
'''Give your posts titles which summarize your idea. Viewers should be able to <strong>get the main point by skimming the title</strong> ("Automatic Suggestion for Tasks based on Average Completion Time" is a good title. "YourTeam TrustIdea 1" is a bad title).'''
 +
 +
-<strong>Please submit your finished ideas by 8:00 pm 14th Feb 2016, and DO NOT vote/comment until then</strong>
 +
 +
=== [Design Test Flight] ===
 +
 +
For your Messenger Design Wiki submissions, please create Google Slides showing the flow of your designs and link it in your Wiki page. For example, see this [https://docs.google.com/presentation/d/1XWRQtkML90zSOf0g74DoKvZ6hDa5ayBMVDm7vz2IsTY/edit?usp=sharing slide show]
 +
 +
When you create the "Share" link, make sure to click “Anyone with the link can view"
 +
 +
=== [Everyone] Peer-evaluation (upvote ones you like, comment on them) from 8:05 pm 14th Feb until 12 pm 15th Feb 2016 ===
 +
 +
Post submission phase, you are welcome to browse through, upvote, and comment on others' ideas. We encourage you especially to look at and comment on ideas that haven't yet gotten feedback, to make sure everybody's ideas gets feedback. You can use http://crowdresearch.meteor.com/needcomments to find ideas that haven't yet gotten feedback, and http://crowdresearch.meteor.com/needclicks to find ideas that haven't been yet been viewed many times.
 +
 +
<strong>COMMENT BEST-PRACTICES</strong>: Everybody in the team reviews at least 3 ideas, supported by a comment. The comment has to justify your reason for upvote. The comment should be constructive, and should mention positive aspect of the idea worth sharing. Negative comments are discouraged, rather make your comment in the form of a suggestion - such as, if you disliked an idea, try to suggest improvements (do not criticize an idea, no idea is bad, every idea has a scope of improvement).

Revision as of 17:42, 15 February 2016

Due date (PST): 8:00 pm 21st Feb 2016 for submission, 12 pm 22nd Feb 2016 for peer-evaluation.

This week, we will refine methods and systems proposed last week further:

  • Youtube link of the meeting today: watch
  • Winter Meeting 6 slideshow: coming soon...
  • Youtube link to the Task feed meeting, 9am to 11am 17th Feb 2016: watch
  • Youtube link to the Task ranking meeting, 8pm to 10pm 17th Feb 2016: watch


Task Feed

Our goal for this week is to converge on a single well-defined systems section. By the end of the week, we want to produce a very specific proposal detailing exactly what needs to be built/done in order to embark on our study. You all wrote a bunch of great proposals last week, and we want to extract the best ideas so that we can synthesize a single system design.

Please be sure to have read this paper on the state of the art in personalized task recommendation in crowdsourcing systems. It is really important that our design and framing of our contributions are novel, and this paper succinctly describes what work has already been done in this domain. In particular, the Findings and Discussion sections describe previous research and what the author believes are viable future directions and Table 2 provides links to other relevant papers.

By Wednesday morning 9 am PST, read each of last week’s submissions on Meteor and write comments in an “I like / I wishstyle. For example, “I like how your proposal strives to extend Boomerang by enforcing an incentive compatible task feed that accurately estimates hourly wages” and “I wish your design accounted for workers who did not provide accurate time estimates even though they did produce good results and the effect this would have on their feed.” Please leave your comments on the task-feed ideas submitted, last week; via comments section on the meteor website.

On Wednesday from 9am-11am PST, @michaelbernstein will lead a hangout to synthesize these ideas and outline a single proposal. We will then take this outline and fill in any missing details during the rest of the week and create a single well-written systems section. RSVP here or watch here.

Topic background

Boomerang: Incentivizing Information Disclosure in Paid Crowdsourcing Platforms

There is a massive amount of information necessary for a healthy crowdsourcing marketplace — for example accurate reputation ratings, skill tags on tasks, and hourly wage estimates for tasks — that is privately held by individuals, but rarely shared. We introduce Boomerang, an interactive task feed for a crowdsourcing marketplace, that incentivizes accurate sharing of this information by making the information directly impact their future tasks or workers. Requesters' ratings of workers, and their skill classifications of tasks, are used to give early access to workers who that requester rates highly and who are experts in that skill, so giving a high rating to a mediocre worker dooms the requester to more mediocre work from that worker. Workers' ratings of requesters are used to rank their high-rated requesters at the top of the task feed, and their estimates of active work time are used to estimate their hourly wage on other tasks on the platform.

The task feed hangouts from last week:

  • Youtube link of the task feed meeting 1: watch
  • Youtube link of the task feed meeting 2: watch

Michael's synthesized needs:

  • to find new tasks that will maximize income (reduce uncertainty in payment, rejection, maximize certainty in what will be asked of me and how quickly I can do it)
  • to find new tasks that fit my expertise profile
  • to refind old requesters' new tasks, since I know I like them

In addition...

  • to identify tasks I can do on my own time
  • to learn new skills

Task authoring

Our goal for this week is to converge on a single well-defined systems section. By the end of the week, we want to produce a very specific proposal detailing exactly what needs to be built/done in order to embark on our study. You all wrote a bunch of great proposals last week, and we want to extract the best ideas so that we can synthesize a single system design.

By Wednesday morning 9 am PST, read each of last week’s submissions on Meteor and write comments in an “I like / I wishstyle. For example, “I like how your proposal explores at the motivation of workers to help requesters author their tasks” and “I wish your design accounted for workers with different level of experience.” Please leave your comments on the task-author ideas submitted, last week; via comments section on the meteor website.

On Wednesday from 8pm-10pm PST, @michaelbernstein will lead a hangout to synthesize these ideas and outline a single proposal. We will then take this outline and fill in any missing details during the rest of the week and create a single well-written systems section. RSVP here or watch here.

Open Gov and design

Submission

Create a Wiki Page for your Team's Submission

Create a wiki page with either methods or systems section, diving deep into one the three themes (look at the template here: Winter Milestone 5 Templates). If you're participating in design test run, create another wiki and paste screenshots or mockup/wireframe files. If you have never created a wiki page before, please see this or watch this.

[Team Representative] Submission or Post the links to your ideas until 8:00 pm 14th Feb 2016

We have a [Reddit like service] on which you can post the links to the wiki-pages for the submissions, explore them, and upvote them.

Please use the same login avenue (Facebook, Twitter, or email address) as you’ve done in the past with Meteor​. This will help us identify and track your contributions better.

For newcomers joining Crowd Research, when it asks you to pick your username, pick the same username as your Slack. Please DO NOT forget to mention the milestone contributors' slackid below each wiki page.

On Meteor, there are 4 submission categories, 3 for research theme, and 1 for design test run.

1- [One of three mandatory] http://crowdresearch.meteor.com/category/task-rank where you can post a link to the wiki page for your task ranking proposal

2- [One of three mandatory] http://crowdresearch.meteor.com/category/task-author where you can post a link to the wiki page for your task author proposal

3- [One of three mandatory] http://crowdresearch.meteor.com/category/open-gov where you can post a link to the wiki page for your open gov proposal

4- [Test flight] http://crowdresearch.meteor.com/category/design-messenger where you can post mockup/wireframe of the Daemo messenger.

Give your posts titles which summarize your idea. Viewers should be able to get the main point by skimming the title ("Automatic Suggestion for Tasks based on Average Completion Time" is a good title. "YourTeam TrustIdea 1" is a bad title).

-Please submit your finished ideas by 8:00 pm 14th Feb 2016, and DO NOT vote/comment until then

[Design Test Flight]

For your Messenger Design Wiki submissions, please create Google Slides showing the flow of your designs and link it in your Wiki page. For example, see this slide show

When you create the "Share" link, make sure to click “Anyone with the link can view"

[Everyone] Peer-evaluation (upvote ones you like, comment on them) from 8:05 pm 14th Feb until 12 pm 15th Feb 2016

Post submission phase, you are welcome to browse through, upvote, and comment on others' ideas. We encourage you especially to look at and comment on ideas that haven't yet gotten feedback, to make sure everybody's ideas gets feedback. You can use http://crowdresearch.meteor.com/needcomments to find ideas that haven't yet gotten feedback, and http://crowdresearch.meteor.com/needclicks to find ideas that haven't been yet been viewed many times.

COMMENT BEST-PRACTICES: Everybody in the team reviews at least 3 ideas, supported by a comment. The comment has to justify your reason for upvote. The comment should be constructive, and should mention positive aspect of the idea worth sharing. Negative comments are discouraged, rather make your comment in the form of a suggestion - such as, if you disliked an idea, try to suggest improvements (do not criticize an idea, no idea is bad, every idea has a scope of improvement).