Difference between revisions of "Winter Milestone 6"

From crowdresearch
Jump to: navigation, search
(Task Authoring)
 
(36 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 
'''Due date (PST): 8:00 pm 21st Feb 2016 for submission, 12 pm 22nd Feb 2016 for peer-evaluation.'''
 
'''Due date (PST): 8:00 pm 21st Feb 2016 for submission, 12 pm 22nd Feb 2016 for peer-evaluation.'''
  
This week, we will accept proposals to pursue different aspects of the project, and start a design test run.
+
This week, we will refine methods and systems proposed last week further:
  
* Youtube link of the meeting today: [http://www.youtube.com/watch?v=uOf3bTWbN1o watch]  
+
* Youtube link of the meeting today: [https://www.youtube.com/watch?v=0wB7C8ZCuXo watch]  
* Winter Meeting 5 slideshow: [[:Media:02-08-focusing.pdf| slides pdf]]
+
* Winter Meeting 6 slideshow: [[:Media:02-15-papers.pdf| slides pdf]]
 +
* Youtube link to the Task feed meeting, 9am to 11am 17th Feb 2016: [http://www.youtube.com/watch?v=Yxl97jMx9V8 watch]
 +
* Youtube link to the Task ranking meeting, 8pm to 10pm 17th Feb 2016: [http://www.youtube.com/watch?v=ppmXmaBh7I0 watch]
  
  
Line 13: Line 15:
 
Please be sure to have read this [http://www.sciencedirect.com/science/article/pii/S016792361400147X paper] on the state of the art in personalized task recommendation in crowdsourcing systems. It is really important that our design and framing of our contributions are '''novel''', and this paper succinctly describes what work has already been done in this domain. In particular, the Findings and Discussion sections describe previous research and what the author believes are viable future directions and Table 2 provides links to other relevant papers.
 
Please be sure to have read this [http://www.sciencedirect.com/science/article/pii/S016792361400147X paper] on the state of the art in personalized task recommendation in crowdsourcing systems. It is really important that our design and framing of our contributions are '''novel''', and this paper succinctly describes what work has already been done in this domain. In particular, the Findings and Discussion sections describe previous research and what the author believes are viable future directions and Table 2 provides links to other relevant papers.
  
By '''Tuesday night / Wednesday morning''', read each of last week’s submissions on Meteor and write comments in an “I like / I wish” [https://dschool.stanford.edu/wp-content/themes/dschool/method-cards/i-like-i-wish-what-if.pdf style]. For example, “I like how your proposal strives to extend Boomerang by enforcing an incentive compatible task feed that accurately estimates hourly wages” and “I wish your design accounted for workers who did not provide accurate time estimates even though they did produce good results and the effect this would have on their feed.”
+
By '''Wednesday morning 9 am PST''', read each of last week’s submissions on Meteor and write comments in an “'''I like / I wish'''” [https://dschool.stanford.edu/wp-content/themes/dschool/method-cards/i-like-i-wish-what-if.pdf style]. For example, “I like how your proposal strives to extend Boomerang by enforcing an incentive compatible task feed that accurately estimates hourly wages” and “I wish your design accounted for workers who did not provide accurate time estimates even though they did produce good results and the effect this would have on their feed.” '''Please leave your comments on the task-feed ideas submitted, last week; [http://crowdresearch.meteor.com/?cat%5B0%5D=task-rank via comments section on the meteor website].'''
  
On '''Wednesday from 9am-11am PST''', '''@michaelbernstein''' will lead a hangout to synthesize these ideas and outline a single proposal. We will then take this outline and fill in any missing details during the rest of the week and create a single well-written systems section.
+
On '''Wednesday from 9am-11am PST''', '''@michaelbernstein''' will lead a hangout to synthesize these ideas and outline a single proposal. We will then take this outline and fill in any missing details during the rest of the week and create a single well-written systems section. '''[https://plus.google.com/events/c5tg3dtp0c6v0aah2udbs9b2c2k RSVP here] or [http://www.youtube.com/watch?v=Yxl97jMx9V8 watch here].'''
  
=== Previously...===
+
----
  
'''Michael's summary abstract after the brainstorms:'''
+
'''Michael's summary from the hangout:'''
 +
 
 +
We have aligned on a specific vision for a smarter, more informative task feed by incentivizing workers and requesters to share information they might not otherwise share, or not share accurately. There are three main components to this:
 +
 
 +
*Reputation. It shows more accurate reputation information, by influencing which workers get your future tasks (if you're a requester), and which requesters show up at the top of your feed (if you're a worker). The individual incentive: you get better workers, or better requesters, by reporting honestly. The global win: the reputation scores more directly reflect individual incentives. This is Boomerang as previously described.
 +
 
 +
*Hourly rate. The goal is to for the task feed to show an estimate of how much you'd make with each task (e.g., $9/hr). To do so, it asks workers who just completed the task to estimate how long it took them to do it. The individual incentive: it uses the worker's responses to build a model to estimate their effective hourly rate for all the other tasks in the marketplace. The global win: those workers' responses are used to produce estimates shown to all other workers.
 +
 
 +
*Rejection information. It shows the % of tasks for workers like you that get rejected, by influencing which workers get the requesters' future tasks based on rejection information. The individual incentive: the more of a worker's tasks a requester accepts, the earlier they get access to their future tasks. (This is a smaller effect than the reputation feedback above, but does have an impact.) This prevents "accept all submissions" degenerate behavior. The global win: workers can now see the % of tasks accepted for workers like them.
 +
 
 +
'''[https://docs.google.com/document/d/1ada3U8fUZKp9emoFpOPRubFI0C5ryZSxdw1d4JU0rCQ/edit?usp=sharing Google Doc] for collaborative creation of the system proposal.'''
 +
 
 +
=== Topic background ===
 +
 
 +
''' Boomerang: Incentivizing Information Disclosure in Paid Crowdsourcing Platforms'''
  
'''Boomerang: Incentivizing Information Disclosure in Paid Crowdsourcing Platforms'''
 
 
There is a massive amount of information necessary for a healthy crowdsourcing marketplace — for example accurate reputation ratings, skill tags on tasks, and hourly wage estimates for tasks — that is privately held by individuals, but rarely shared. We introduce Boomerang, an interactive task feed for a crowdsourcing marketplace, that incentivizes accurate sharing of this information by making the information directly impact their future tasks or workers. Requesters' ratings of workers, and their skill classifications of tasks, are used to give early access to workers who that requester rates highly and who are experts in that skill, so giving a high rating to a mediocre worker dooms the requester to more mediocre work from that worker. Workers' ratings of requesters are used to rank their high-rated requesters at the top of the task feed, and their estimates of active work time are used to estimate their hourly wage on other tasks on the platform.
 
There is a massive amount of information necessary for a healthy crowdsourcing marketplace — for example accurate reputation ratings, skill tags on tasks, and hourly wage estimates for tasks — that is privately held by individuals, but rarely shared. We introduce Boomerang, an interactive task feed for a crowdsourcing marketplace, that incentivizes accurate sharing of this information by making the information directly impact their future tasks or workers. Requesters' ratings of workers, and their skill classifications of tasks, are used to give early access to workers who that requester rates highly and who are experts in that skill, so giving a high rating to a mediocre worker dooms the requester to more mediocre work from that worker. Workers' ratings of requesters are used to rank their high-rated requesters at the top of the task feed, and their estimates of active work time are used to estimate their hourly wage on other tasks on the platform.
  
Line 36: Line 51:
 
* to identify tasks I can do on my own time
 
* to identify tasks I can do on my own time
 
* to learn new skills
 
* to learn new skills
 +
 +
== Task Authoring ==
 +
 +
Our goal for this week is to converge on a single well-defined methods section. By the end of the week, we want to produce a very '''specific''' proposal detailing exactly what needs to be built/done in order to embark on our study. You all wrote a bunch of great proposals last week, and we want to extract the best ideas so that we can synthesize a single system design.
 +
 +
By '''Wednesday morning 9 am PST''', read each of last week’s submissions on Meteor and write comments in an “'''I like / I wish'''” [https://dschool.stanford.edu/wp-content/themes/dschool/method-cards/i-like-i-wish-what-if.pdf style]. For example, “I like how your proposal explores at the motivation of workers to help requesters author their tasks” and “I wish your design accounted for workers with different level of experience.” '''Please leave your comments on the task-author ideas submitted, last week; [http://crowdresearch.meteor.com/?cat%5B0%5D=task-author via comments section on the meteor website].'''
 +
 +
On '''Wednesday from 8pm-10pm PST''', '''@michaelbernstein''' will lead a hangout to synthesize these ideas and outline a single proposal. We will then take this outline and fill in any missing details during the rest of the week and create a single well-written systems section. '''[https://plus.google.com/events/csrv8tvkomh2qa52komf5somf44 RSVP here] or [http://www.youtube.com/watch?v=ppmXmaBh7I0 watch here].
 +
 +
=== Proposal ===
 +
We will now be writing up a single Task Authoring Methods Proposal together [https://docs.google.com/document/d/1B-gSbAn3atDvV8qZtsRWrJ0CdKhbMDO7B3dgFK_yZyo/edit?usp=sharing here] for the conference submission.
 +
Please pick a section (or create your own) in the doc below and start adding content!
 +
 +
===Volunteer to be a requester===
 +
The taskauthoring folks are looking for a couple volunteers to create some tasks for three datasets we have.
 +
We give you the task and some example input/outputs, and you try to write a task interface in Mechanical Turk that will get workers to produce the right answers.
 +
 +
If you are interested, please ping @catherine.mullings
 +
 +
== Open Gov and Design ==
 +
 +
[https://www.youtube.com/watch?v=0wB7C8ZCuXo Check out this week's meeting], and based on the open gov discussions here - create a mock, minimal design within Daemo. Think about questions like: how does it work as a system? how it would fit in Daemo? Like, walk us through. I’m a new worker on Daemo. What do I do? Am I already part of a guild? How do I get into one? How do I get work once I’m in one? What if the requester doesn’t like what I do? And how does all this solve the reputation problem?
 +
 +
You can use [https://balsamiq.com/ balsamic] to give shape to your ideas. Design folks, come join and help move this effort forward.
 +
 +
== Research Engineering ==
 +
 +
This weeks main issues: #77 (this is a pretty big one), #660, #509
 +
 +
announce in #research-engineering that you are working on a particular issue and please let the others know about the progress of the issues you are working on (so that we don't do duplicate work). You are encouraged to work together.
 +
 +
For any questions ping @aginzberg, @dmorina, and @shirish.goyal on Slack #research-engineering
 +
 +
== Submission ==
 +
 +
=== Create a Wiki Page for your Team's Submission (Open Gov and Design) ===
 +
 +
Create a wiki page with Balsamic designs and descriptions, for the Open Gov and Design part. If you have never created a wiki page before, please see [http://www.mediawiki.org/wiki/Help:Starting_a_new_page this] or watch [https://www.youtube.com/watch?v=83-lCpAnaFw this]. Once you're done, post here: http://crowdresearch.meteor.com/category/open-gov
 +
 +
=== Comment I like/I wish for your Team's Submission (Task Authoring and Task Feed) ===
 +
 +
By Wednesday morning 9 am, finish commenting for task feed; by Wednesday evening 8 pm, finish commenting for task authoring. Make these comments on the [http://crowdresearch.meteor.com/ meteor website].

Latest revision as of 12:55, 21 February 2016

Due date (PST): 8:00 pm 21st Feb 2016 for submission, 12 pm 22nd Feb 2016 for peer-evaluation.

This week, we will refine methods and systems proposed last week further:

  • Youtube link of the meeting today: watch
  • Winter Meeting 6 slideshow: slides pdf
  • Youtube link to the Task feed meeting, 9am to 11am 17th Feb 2016: watch
  • Youtube link to the Task ranking meeting, 8pm to 10pm 17th Feb 2016: watch


Task Feed

Our goal for this week is to converge on a single well-defined systems section. By the end of the week, we want to produce a very specific proposal detailing exactly what needs to be built/done in order to embark on our study. You all wrote a bunch of great proposals last week, and we want to extract the best ideas so that we can synthesize a single system design.

Please be sure to have read this paper on the state of the art in personalized task recommendation in crowdsourcing systems. It is really important that our design and framing of our contributions are novel, and this paper succinctly describes what work has already been done in this domain. In particular, the Findings and Discussion sections describe previous research and what the author believes are viable future directions and Table 2 provides links to other relevant papers.

By Wednesday morning 9 am PST, read each of last week’s submissions on Meteor and write comments in an “I like / I wishstyle. For example, “I like how your proposal strives to extend Boomerang by enforcing an incentive compatible task feed that accurately estimates hourly wages” and “I wish your design accounted for workers who did not provide accurate time estimates even though they did produce good results and the effect this would have on their feed.” Please leave your comments on the task-feed ideas submitted, last week; via comments section on the meteor website.

On Wednesday from 9am-11am PST, @michaelbernstein will lead a hangout to synthesize these ideas and outline a single proposal. We will then take this outline and fill in any missing details during the rest of the week and create a single well-written systems section. RSVP here or watch here.


Michael's summary from the hangout:

We have aligned on a specific vision for a smarter, more informative task feed by incentivizing workers and requesters to share information they might not otherwise share, or not share accurately. There are three main components to this:

  • Reputation. It shows more accurate reputation information, by influencing which workers get your future tasks (if you're a requester), and which requesters show up at the top of your feed (if you're a worker). The individual incentive: you get better workers, or better requesters, by reporting honestly. The global win: the reputation scores more directly reflect individual incentives. This is Boomerang as previously described.
  • Hourly rate. The goal is to for the task feed to show an estimate of how much you'd make with each task (e.g., $9/hr). To do so, it asks workers who just completed the task to estimate how long it took them to do it. The individual incentive: it uses the worker's responses to build a model to estimate their effective hourly rate for all the other tasks in the marketplace. The global win: those workers' responses are used to produce estimates shown to all other workers.
  • Rejection information. It shows the % of tasks for workers like you that get rejected, by influencing which workers get the requesters' future tasks based on rejection information. The individual incentive: the more of a worker's tasks a requester accepts, the earlier they get access to their future tasks. (This is a smaller effect than the reputation feedback above, but does have an impact.) This prevents "accept all submissions" degenerate behavior. The global win: workers can now see the % of tasks accepted for workers like them.

Google Doc for collaborative creation of the system proposal.

Topic background

Boomerang: Incentivizing Information Disclosure in Paid Crowdsourcing Platforms

There is a massive amount of information necessary for a healthy crowdsourcing marketplace — for example accurate reputation ratings, skill tags on tasks, and hourly wage estimates for tasks — that is privately held by individuals, but rarely shared. We introduce Boomerang, an interactive task feed for a crowdsourcing marketplace, that incentivizes accurate sharing of this information by making the information directly impact their future tasks or workers. Requesters' ratings of workers, and their skill classifications of tasks, are used to give early access to workers who that requester rates highly and who are experts in that skill, so giving a high rating to a mediocre worker dooms the requester to more mediocre work from that worker. Workers' ratings of requesters are used to rank their high-rated requesters at the top of the task feed, and their estimates of active work time are used to estimate their hourly wage on other tasks on the platform.

The task feed hangouts from last week:

  • Youtube link of the task feed meeting 1: watch
  • Youtube link of the task feed meeting 2: watch

Michael's synthesized needs:

  • to find new tasks that will maximize income (reduce uncertainty in payment, rejection, maximize certainty in what will be asked of me and how quickly I can do it)
  • to find new tasks that fit my expertise profile
  • to refind old requesters' new tasks, since I know I like them

In addition...

  • to identify tasks I can do on my own time
  • to learn new skills

Task Authoring

Our goal for this week is to converge on a single well-defined methods section. By the end of the week, we want to produce a very specific proposal detailing exactly what needs to be built/done in order to embark on our study. You all wrote a bunch of great proposals last week, and we want to extract the best ideas so that we can synthesize a single system design.

By Wednesday morning 9 am PST, read each of last week’s submissions on Meteor and write comments in an “I like / I wishstyle. For example, “I like how your proposal explores at the motivation of workers to help requesters author their tasks” and “I wish your design accounted for workers with different level of experience.” Please leave your comments on the task-author ideas submitted, last week; via comments section on the meteor website.

On Wednesday from 8pm-10pm PST, @michaelbernstein will lead a hangout to synthesize these ideas and outline a single proposal. We will then take this outline and fill in any missing details during the rest of the week and create a single well-written systems section. RSVP here or watch here.

Proposal

We will now be writing up a single Task Authoring Methods Proposal together here for the conference submission. Please pick a section (or create your own) in the doc below and start adding content!

Volunteer to be a requester

The taskauthoring folks are looking for a couple volunteers to create some tasks for three datasets we have. We give you the task and some example input/outputs, and you try to write a task interface in Mechanical Turk that will get workers to produce the right answers.

If you are interested, please ping @catherine.mullings

Open Gov and Design

Check out this week's meeting, and based on the open gov discussions here - create a mock, minimal design within Daemo. Think about questions like: how does it work as a system? how it would fit in Daemo? Like, walk us through. I’m a new worker on Daemo. What do I do? Am I already part of a guild? How do I get into one? How do I get work once I’m in one? What if the requester doesn’t like what I do? And how does all this solve the reputation problem?

You can use balsamic to give shape to your ideas. Design folks, come join and help move this effort forward.

Research Engineering

This weeks main issues: #77 (this is a pretty big one), #660, #509

announce in #research-engineering that you are working on a particular issue and please let the others know about the progress of the issues you are working on (so that we don't do duplicate work). You are encouraged to work together.

For any questions ping @aginzberg, @dmorina, and @shirish.goyal on Slack #research-engineering

Submission

Create a Wiki Page for your Team's Submission (Open Gov and Design)

Create a wiki page with Balsamic designs and descriptions, for the Open Gov and Design part. If you have never created a wiki page before, please see this or watch this. Once you're done, post here: http://crowdresearch.meteor.com/category/open-gov

Comment I like/I wish for your Team's Submission (Task Authoring and Task Feed)

By Wednesday morning 9 am, finish commenting for task feed; by Wednesday evening 8 pm, finish commenting for task authoring. Make these comments on the meteor website.