Difference between revisions of "Winter Milestone 8"

From crowdresearch
Jump to: navigation, search
(Created page with "'''Due date (PST): 8:00 pm 6th March 2016 for submission, 12 pm 7th March 2016 for peer-evaluation.''' This week, we will refine methods and systems proposed last week furthe...")
 
(Task Authoring)
 
(24 intermediate revisions by 3 users not shown)
Line 3: Line 3:
 
This week, we will refine methods and systems proposed last week further:  
 
This week, we will refine methods and systems proposed last week further:  
  
* Youtube link of the meeting today: [https://www.youtube.com/watch?v=pxvJzdT2FvI watch]  
+
* Youtube link of the meeting today: [http://www.youtube.com/watch?v=pxvJzdT2FvI watch]  
* Winter Meeting 8 slideshow: coming soon...
+
* Winter Meeting 8 slideshow: [[:Media:02-29-1month.pdf| slides pdf]]
  
 
== Task Feed ==
 
== Task Feed ==
  
We have 3 goals for this week:
+
Last week we achieved near convergence on our taskfeed system proposal and generated some mocks for our new taskfeed!
*Refining our '''[https://docs.google.com/document/d/1ada3U8fUZKp9emoFpOPRubFI0C5ryZSxdw1d4JU0rCQ/edit?usp=sharing system proposal]'''. Please make comments directly on the google doc. We want to address any holes in our incentive compatible structure and make any concrete decisions regarding things like the # of workers that report time spent on a task before we start displaying it on the taskfeed.
+
 
*Design tweaks in order to capture and display the new information we need. This includes a task element to record the time a worker spent and small modifications to the taskfeed to display the rejection rate and effective wage. Please post low-fi or high-fi mocks to the '''#taskfeed''' channel.
+
This week we are beginning the push for our UIST submission. Our goals for this week are the following:
*Engineering plan of action and division of work. Led by '''@dmorina'''.
+
*Design mocks for the data collection part of the taskfeed (i.e. how are we going to change the actual task interface for workers to report time spent working).
 +
*Frontend implementation of our taskfeed mocks from last week and this week (make basic HTML and CSS changes on the taskfeed to display and collect the necessary data: rejection rate and time spent).
 +
*Backend implementation of rejection rate (write the SQL needed to compute the rejection rates of each task for a given worker and return to the frontend).
 +
*Continued refinement and writeup of our system and methodologies - particularly the task time system. If enough cycles, begin outlining the paper or working on related work.
 +
 
 +
With the internal deadline on '''April 3rd,''' we are going to need all hands on deck for the next month. Whether you are a developer, designer, writer, thinker, etc. there will always be something for you to work on. Please pick up anything and give a yell in the channel so everyone knows who is working on what and so that we can help and guide each other. As always, feel free to reach out to '''@aginzberg''' or '''@dmorina''' individually with any questions.
  
 
----
 
----
  
 
=== Topic background ===
 
=== Topic background ===
'''Michael's summary from last week's hangout:'''
+
Check out our '''[https://docs.google.com/document/d/1ada3U8fUZKp9emoFpOPRubFI0C5ryZSxdw1d4JU0rCQ/edit?usp=sharing system proposal]''', mainly first few pages and comments to catch up on what we've been working on and thinking about.
  
We have aligned on a specific vision for a smarter, more informative task feed by incentivizing workers and requesters to share information they might not otherwise share, or not share accurately. There are three main components to this:
+
'''The task feed hangouts from previous weeks:'''
 
+
*Reputation. It shows more accurate reputation information, by influencing which workers get your future tasks (if you're a requester), and which requesters show up at the top of your feed (if you're a worker). The individual incentive: you get better workers, or better requesters, by reporting honestly. The global win: the reputation scores more directly reflect individual incentives. This is Boomerang as previously described.
+
 
+
*Hourly rate. The goal is to for the task feed to show an estimate of how much you'd make with each task (e.g., $9/hr). To do so, it asks workers who just completed the task to estimate how long it took them to do it. The individual incentive: it uses the worker's responses to build a model to estimate their effective hourly rate for all the other tasks in the marketplace. The global win: those workers' responses are used to produce estimates shown to all other workers.
+
 
+
*Rejection information. It shows the % of tasks for workers like you that get rejected, by influencing which workers get the requesters' future tasks based on rejection information. The individual incentive: the more of a worker's tasks a requester accepts, the earlier they get access to their future tasks. (This is a smaller effect than the reputation feedback above, but does have an impact.) This prevents "accept all submissions" degenerate behavior. The global win: workers can now see the % of tasks accepted for workers like them.
+
 
+
''' Boomerang: Incentivizing Information Disclosure in Paid Crowdsourcing Platforms'''
+
 
+
There is a massive amount of information necessary for a healthy crowdsourcing marketplace — for example accurate reputation ratings, skill tags on tasks, and hourly wage estimates for tasks — that is privately held by individuals, but rarely shared. We introduce Boomerang, an interactive task feed for a crowdsourcing marketplace, that incentivizes accurate sharing of this information by making the information directly impact their future tasks or workers. Requesters' ratings of workers, and their skill classifications of tasks, are used to give early access to workers who that requester rates highly and who are experts in that skill, so giving a high rating to a mediocre worker dooms the requester to more mediocre work from that worker. Workers' ratings of requesters are used to rank their high-rated requesters at the top of the task feed, and their estimates of active work time are used to estimate their hourly wage on other tasks on the platform.
+
 
+
'''The task feed hangouts from last week:'''
+
 
* Youtube link of the task feed meeting 1: [http://www.youtube.com/watch?v=OeBoiA07BBk watch]  
 
* Youtube link of the task feed meeting 1: [http://www.youtube.com/watch?v=OeBoiA07BBk watch]  
 
* Youtube link of the task feed meeting 2: [http://www.youtube.com/watch?v=42d5H0yoPPc watch]
 
* Youtube link of the task feed meeting 2: [http://www.youtube.com/watch?v=42d5H0yoPPc watch]
Line 46: Line 39:
 
== Task Authoring ==  
 
== Task Authoring ==  
  
Our goal for this week is to run experiments, then analyze the findings and write up a report. By the end of the week, we want to produce a report with findings from our pilot experiments - this will help us in making future decisions.  
+
Our goal for this week is to run our second pilot study to observe the variance between task authorship and task quality.  
  
'''First half of the week (Volunteer to be a requester)''': Pilot participants should try to finish their HTML task designs, that takes about an hour. Please message @catherine.mullings for that. Once we have your task designs, @dmorina will post them on Mechanical Turk for you. The taskauthoring folks are looking for a couple volunteers to create some tasks for four datasets that we have. We give you the task and some example input/outputs, and you try to write a task interface in Mechanical Turk that will get workers to produce the right answers.
+
'''Watch last week's task authoring meeting''' [https://www.youtube.com/watch?v=-hlgjuBS8us&feature=youtu.be here]
  
'''Second half of the week (analyze+report)''': We analyze the results from the experiment, conduct a Google hangout with @michaelbernstein, and produce a report on the findings. We can start making contributions to the report [https://docs.google.com/document/d/1B-gSbAn3atDvV8qZtsRWrJ0CdKhbMDO7B3dgFK_yZyo/edit?usp=sharing here on Google doc].
+
'''First half of this week (Volunteer to be a requester)''': Please message @catherine.mullings for that. Instructions will be released by Wednesday/Thursday.
 +
 
 +
'''By Wednesday, brainstorm a task or find dataset''' to give to requesters in our pilot study. Think of tasks that are more complex than image labeling or doing sentiment judgements. For example, one task might be to transcribe Youtube videos. Transcription is a much more involved task for workers and final results are subjective and up to interpretation.
 +
 
 +
'''By Saturday/Sunday, analyze results:''' After the task results are received, we will analyze the results from the experiment during a Google Hangout on Saturday/Sunday.  
 +
 
 +
The analysis for the study will be formally written [https://docs.google.com/document/d/1B-gSbAn3atDvV8qZtsRWrJ0CdKhbMDO7B3dgFK_yZyo/edit?usp=sharing here on Google doc].
  
 
== Open Gov and Design ==
 
== Open Gov and Design ==
  
[https://www.youtube.com/watch?v=QK2qaB-OutQ Check out this week's meeting], and based on the open gov discussions here - create a mock, minimal design within Daemo. Think about questions like: how does it work as a system? how it would fit in Daemo? Like, walk us through. I’m a new worker on Daemo. What do I do? Am I already part of a guild? How do I get into one? How do I get work once I’m in one? What if the requester doesn’t like what I do? And how does all this solve the reputation problem? Try to pitch a story/wireframe with specific design that has as few moving parts as possible.   
+
[http://www.youtube.com/watch?v=pxvJzdT2FvI Check out this week's meeting], and based on the open gov discussions here - create a mock, minimal design within Daemo. Think about questions like: how does it work as a system? how it would fit in Daemo? Like, walk us through. I’m a new worker on Daemo. What do I do? Am I already part of a guild? How do I get into one? How do I get work once I’m in one? What if the requester doesn’t like what I do? And how does all this solve the reputation problem? Try to pitch a story/wireframe with specific design that has as few moving parts as possible.  Please create a storyboard, to walk us through it step-by-step. Example below:
 +
 
 +
[[File:Storyboard.png| Storyboard example]]
  
 
Helpful links:  
 
Helpful links:  
 +
* [https://docs.google.com/document/d/1GiEzgOto9pP6YosJJ4dFxMgExgASiex72IqGgM-MXQc/edit?usp=sharing Google doc paper]
 
* [[Guilds and computational compatibility...The system]]
 
* [[Guilds and computational compatibility...The system]]
 
* [[Guilds Milestone 6]]
 
* [[Guilds Milestone 6]]
  
You can use [https://balsamiq.com/ balsamic] or Google slides to give shape to your ideas. Design folks, come join and help move this effort forward. Once you're done, post here: http://crowdresearch.meteor.com/category/open-gov
+
You can use [https://balsamiq.com/ balsamic] or Google slides or just pencil/paper drawing to give shape to your ideas. Design folks, come join and help move this effort forward. Once you're done, post here: http://crowdresearch.meteor.com/category/open-gov

Latest revision as of 23:08, 29 February 2016

Due date (PST): 8:00 pm 6th March 2016 for submission, 12 pm 7th March 2016 for peer-evaluation.

This week, we will refine methods and systems proposed last week further:

  • Youtube link of the meeting today: watch
  • Winter Meeting 8 slideshow: slides pdf

Task Feed

Last week we achieved near convergence on our taskfeed system proposal and generated some mocks for our new taskfeed!

This week we are beginning the push for our UIST submission. Our goals for this week are the following:

  • Design mocks for the data collection part of the taskfeed (i.e. how are we going to change the actual task interface for workers to report time spent working).
  • Frontend implementation of our taskfeed mocks from last week and this week (make basic HTML and CSS changes on the taskfeed to display and collect the necessary data: rejection rate and time spent).
  • Backend implementation of rejection rate (write the SQL needed to compute the rejection rates of each task for a given worker and return to the frontend).
  • Continued refinement and writeup of our system and methodologies - particularly the task time system. If enough cycles, begin outlining the paper or working on related work.

With the internal deadline on April 3rd, we are going to need all hands on deck for the next month. Whether you are a developer, designer, writer, thinker, etc. there will always be something for you to work on. Please pick up anything and give a yell in the channel so everyone knows who is working on what and so that we can help and guide each other. As always, feel free to reach out to @aginzberg or @dmorina individually with any questions.


Topic background

Check out our system proposal, mainly first few pages and comments to catch up on what we've been working on and thinking about.

The task feed hangouts from previous weeks:

  • Youtube link of the task feed meeting 1: watch
  • Youtube link of the task feed meeting 2: watch
  • Youtube link of the task feed meeting 3: watch

Michael's synthesized needs:

  • to find new tasks that will maximize income (reduce uncertainty in payment, rejection, maximize certainty in what will be asked of me and how quickly I can do it)
  • to find new tasks that fit my expertise profile
  • to refind old requesters' new tasks, since I know I like them

In addition...

  • to identify tasks I can do on my own time
  • to learn new skills

Task Authoring

Our goal for this week is to run our second pilot study to observe the variance between task authorship and task quality.

Watch last week's task authoring meeting here

First half of this week (Volunteer to be a requester): Please message @catherine.mullings for that. Instructions will be released by Wednesday/Thursday.

By Wednesday, brainstorm a task or find dataset to give to requesters in our pilot study. Think of tasks that are more complex than image labeling or doing sentiment judgements. For example, one task might be to transcribe Youtube videos. Transcription is a much more involved task for workers and final results are subjective and up to interpretation.

By Saturday/Sunday, analyze results: After the task results are received, we will analyze the results from the experiment during a Google Hangout on Saturday/Sunday.

The analysis for the study will be formally written here on Google doc.

Open Gov and Design

Check out this week's meeting, and based on the open gov discussions here - create a mock, minimal design within Daemo. Think about questions like: how does it work as a system? how it would fit in Daemo? Like, walk us through. I’m a new worker on Daemo. What do I do? Am I already part of a guild? How do I get into one? How do I get work once I’m in one? What if the requester doesn’t like what I do? And how does all this solve the reputation problem? Try to pitch a story/wireframe with specific design that has as few moving parts as possible. Please create a storyboard, to walk us through it step-by-step. Example below:

Storyboard example

Helpful links:

You can use balsamic or Google slides or just pencil/paper drawing to give shape to your ideas. Design folks, come join and help move this effort forward. Once you're done, post here: http://crowdresearch.meteor.com/category/open-gov