Difference between revisions of "Milestone 6 Research Proposal"

From crowdresearch
Jump to: navigation, search
(Research Contributions)
 
(36 intermediate revisions by 3 users not shown)
Line 1: Line 1:
This is one of the two options for [[Milestone 6]], which is geared towards teams which are more interested in the research side of things.
+
This is one of the two options for [[Milestone 6]], which is geared towards teams which are more interested in the research side of things. If your team is more interested in contributing to the basic code infrastructure for our platform, see [[Milestone 6 Code Infrastructure]]. Note that your choice of milestone this week will not limit your future options (ie, if you choose to write a research proposal this week, you will still be able to contribute to code in the future if you so desire).
  
 
In this milestone, your team will propose a new crowdsourcing platform by synthesizing some foundational ideas that have been proposed over the brainstorming and prototyping stages. You will propose it by writing an introduction to a mock research paper that summarizes the main contributions of a proposed platform, and how/why you expect that it improves on the current state-of-the-art.
 
In this milestone, your team will propose a new crowdsourcing platform by synthesizing some foundational ideas that have been proposed over the brainstorming and prototyping stages. You will propose it by writing an introduction to a mock research paper that summarizes the main contributions of a proposed platform, and how/why you expect that it improves on the current state-of-the-art.
Line 5: Line 5:
 
== Determine the foundational ideas in your proposed platform ==
 
== Determine the foundational ideas in your proposed platform ==
  
First, you should find a set of ideas from the milestones that you believe will work well together in combination as a platform. Try to focus on big ideas you believe are truly revolutionary and essential underpinning of your platform ("foundational ideas"), as opposed to minor features that are nice to have but are not particularly novel or necessary.
+
First, you should find a set of ideas from the milestones that you believe will work well together in combination as a platform. Try to primarily focus on big ideas you believe are truly revolutionary and essential underpinnings of your platform ("'''foundational ideas'''"), then to '''features''' that are great to have but are not the core of what represents the platform. Foundations should aim to solve problems of '''trust''' and '''power''' that motivated us in our needfinding. Features are ideas which improve the strength of any platform but aren’t holistic or don’t give it a high-level purpose. For example, "workers review all tasks before posting and also make all payment decisions" and "workers organize themselves into collectives" would be foundations, whereas task recommender systems, mediation strategies, and specific feedback and rating mechanisms would be features.
  
You can draw your set of foundational ideas by synthesizing ones that have been proposed in Milestones 3, 4, and 5:
+
One way to determine what the foundational ideas of your proposed platform are, are by thinking about how you would pitch your platform. If you had to describe why your platform is better than any of the countless existing crowd marketplaces in a sentence or two, what would you say? These are your foundational ideas. If your don't think you can explain the novelty and advantages of your system in just a sentence or two, you may want to revisit the set of ideas you have picked (see the meteor links above) - perhaps you are proposing a bag of features, with less novel/foundational ideas.
 +
 
 +
As a team, choose one to two main foundations for your platform, and one to three core features that you think will be important.
 +
 
 +
You can draw the set of foundational or feature ideas for your platform by synthesizing ideas and prototypes that have been proposed in Milestones 3, 4, and 5:
  
 
http://crowdresearch3.meteor.com/
 
http://crowdresearch3.meteor.com/
Line 15: Line 19:
 
http://crowdresearch5.meteor.com/
 
http://crowdresearch5.meteor.com/
  
One way to determine what the foundational ideas of your proposed platform are, are by thinking about how you would pitch your platform. If you had to describe why your platform is better than any of the countless existing crowd marketplaces in a sentence or two, what would you say? These are your foundational ideas. If your don't think you can explain the novelty and advantages of your system in just a sentence or two, you may want to revisit the set of ideas you have picked - perhaps you are proposing a bag of minor features, with no truly novel, foundational ideas.
+
For more on foundations vs. features, see this week's slides.
  
 
== Format of your research proposal ==
 
== Format of your research proposal ==
  
You will propose your platform in the form of an introduction to a mock research paper. Essentially, imagine you have built your system, incorporating in all the ideas that you wanted to have in it, and everything has gone as planned. How do you convince people (in the case of a paper, this is usually other researchers) that you have built a platform that is novel and that it will be more effective at addressing problems than existing ideas?
+
You will propose your platform in the form of an introduction to a mock research paper. Essentially, imagine you have built your system, incorporating in all the ideas that you wanted to have in it, have run your user studies and evaluations, and everything has gone as planned. How do you convince other researchers that you have built a platform that is novel and that it is more effective at addressing problems than any existing ideas that have been attempted in the past?
  
An introduction of a research paper summarizes the main contributions of the research. You usually want to address the following points:
+
This is an extremely effective strategy for making your ideas concrete and persuasive.
  
=== Background / Motivation ===
+
An introduction of a research paper summarizes the main contributions of the research. It is generally roughly 1 page (roughly 1000 words), and consists of the following components.
  
What are the core problems that your system will solve? Why are these problems important?
+
=== Title ===
 +
 
 +
Research papers should have titles which summarize, in one line, the primary research contribution (see below).
 +
 
 +
=== Abstract ===
 +
 
 +
A concise summary of the background and your research contributions (described below). Usually around 150 words or less.
 +
 
 +
Each of the following sections should be one to two paragraphs each. Paper introductions are brief and impactful.
 +
 
 +
=== Motivation ===
 +
 
 +
What is the problem that you are solving, and why is it important?
 
* Think about the [[Milestone_3#Michael_Bernstein.27s_synthesis | needs we synthesized from Milestone 2]] - ie, trust and power - when thinking about which problems you want to solve.
 
* Think about the [[Milestone_3#Michael_Bernstein.27s_synthesis | needs we synthesized from Milestone 2]] - ie, trust and power - when thinking about which problems you want to solve.
 +
* This should be a specific problem! Not just “crowdsourcing”. More like how trust and power are broken.
  
 
=== Related Work ===
 
=== Related Work ===
Line 33: Line 50:
 
* You may want to search around [https://scholar.google.com/ Google Scholar] to find existing work that is related to the ideas your system proposes.
 
* You may want to search around [https://scholar.google.com/ Google Scholar] to find existing work that is related to the ideas your system proposes.
  
=== Research Contributions ===
+
=== Insight ===
  
What are the foundational ideas that underly your system? Why are they novel and better than the current state-of-the-art solutions?
+
This section should lay out this the foundational idea(s). These big ideas are the things that you'll be known for, and what other platforms would want to replicate. Explain: Why/how are they novel and better than anything that has been attempted in the past?
 +
 
 +
=== System ===
 +
 
 +
The insight above should explain the high level idea (e.g., "All workers are paid in chocolate"). Here, you explain how it works in specifics. (e.g., "We built a crowdsourcing platform called Chococrowd that mails dark chocolate candies to workers at the conclusion of each month. Requesters choose the quality of the dark chocolate based on the quality of the work.")
  
 
=== Evaluation ===
 
=== Evaluation ===
  
 
Once the platform you propose has been implemented, how will you determine whether your system actually solves the problem you wanted to solve? What are the results you hope you can realistically achieve? Why do these results show that you have solved the problem?
 
Once the platform you propose has been implemented, how will you determine whether your system actually solves the problem you wanted to solve? What are the results you hope you can realistically achieve? Why do these results show that you have solved the problem?
 +
 +
=== References ===
 +
 +
The reference section is where you cite prior work that you build upon. If you are aware of existing related research papers, list them here. We also encourage you to borrow ideas from the past submissions (see the meteor links above). Please list the links of the ideas you used to create this proposal (there's no restriction in terms of number of ideas or whether its yours or others'). You can use the following template:
 +
 +
* [Foundation Idea] Link...
 +
* [Feature Idea] Link...
 +
* [Foundation Idea] Link...so on, and so forth...
  
 
== Read introductions of some existing papers ==
 
== Read introductions of some existing papers ==
  
To get yourself familiarized with the format, you should read the introductions of these papers to get a good idea for how HCI research papers that propose new platforms are structured. You will likely be familiar with the mClerk and MobileWorks papers, as they were part of the Milestone 1 readings.
+
To see concrete examples of how HCI research papers that propose new platforms are structured, you should read the introductions of these papers. You will likely be familiar with the first two papers (mClerk and MobileWorks), as they were part of the Milestone 1 readings.
  
 
=== mClerk ===
 
=== mClerk ===
Line 56: Line 85:
  
 
[[:Media:Twitch Crowdsourcing (private).pdf | Vaish R, Wyngarden K, Chen J, et al. Twitch crowdsourcing: crowd contributions in short bursts of time. Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 2014: 3645-3654.]]
 
[[:Media:Twitch Crowdsourcing (private).pdf | Vaish R, Wyngarden K, Chen J, et al. Twitch crowdsourcing: crowd contributions in short bursts of time. Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 2014: 3645-3654.]]
 +
 +
== Submitting ==
 +
 +
=== Create a Wiki Page for your Team's Submission ===
 +
 +
Please create a page for your team's submission at http://crowdresearch.stanford.edu/w/index.php?title=Milestone_6_YourTeamName&action=edit (substituting in YourTeamName with the team name), copy over the template at <strong>[[Milestone 6 Research Proposal Template]] </strong>.
 +
 +
=== [Team Leaders] Post the links to your research proposals until 8th April 11:59 pm===
 +
 +
We have a service on which you can post research proposals you generated, comment on them, and upvote ones you like.
 +
 +
http://crowdresearch.meteor.com/category/milestone-6-research-proposals
 +
 +
Post links to your research proposals <strong>only once they're finished</strong>. Give your posts the same title as your mock research paper. Do not include words like "Milestone", "Research Proposal", or your team name in the title.
 +
 +
-<strong>Please submit your finished research proposals by 11:59 pm 8th April 2015, and DO NOT vote/comment until 9th April 12:05 am</strong>
 +
 +
=== [Everyone] Peer-evaluation (upvote ones you like, comment on them) from 12:05 am 9th April until 9 am 10th April ===
 +
 +
Post submission phase, you are welcome to browse through, upvote, and comment on others' research proposals. We encourage you especially to look at and comment on submissions that haven't yet gotten feedback, to make sure everybody's submissions get feedback.
 +
 +
<strong> Step 1: </strong> Please use http://crowdresearch.meteor.com/needcomments to find submissions that haven't yet gotten feedback, and http://crowdresearch.meteor.com/needclicks to find submissions that haven't been yet been viewed many times.
 +
 +
<strong> Step 2: </strong> Once you find an idea of interest or less attended, please vote and comment upon it. Please perform this action from 3 to 5 submissions - this will help us balance the comments and votes. Please do not vote your team's research proposals. Once again, '''everyone''' is supposed to vote+comment, whether you're the team leader or not.
 +
 +
<strong>COMMENT BEST-PRACTICES</strong>: As on Crowdgrader, <strong>everybody</strong> reviews at least 3 submissions, supported by a comment. The comment should provide constructive feedback. Negative comments are discouraged - if you disliked some aspect of a submission, make a suggestion for improvement.
 +
 +
=== [Team Leaders] Milestone 6 Submissions ===
 +
 +
To help us track all submissions and browsing through them, once you have finished your Milestone 6 Research Proposal submission, go to the link below and post the link:
 +
 +
[[Milestone 6 Research Proposal Submissions]]

Latest revision as of 03:08, 6 April 2015

This is one of the two options for Milestone 6, which is geared towards teams which are more interested in the research side of things. If your team is more interested in contributing to the basic code infrastructure for our platform, see Milestone 6 Code Infrastructure. Note that your choice of milestone this week will not limit your future options (ie, if you choose to write a research proposal this week, you will still be able to contribute to code in the future if you so desire).

In this milestone, your team will propose a new crowdsourcing platform by synthesizing some foundational ideas that have been proposed over the brainstorming and prototyping stages. You will propose it by writing an introduction to a mock research paper that summarizes the main contributions of a proposed platform, and how/why you expect that it improves on the current state-of-the-art.

Determine the foundational ideas in your proposed platform

First, you should find a set of ideas from the milestones that you believe will work well together in combination as a platform. Try to primarily focus on big ideas you believe are truly revolutionary and essential underpinnings of your platform ("foundational ideas"), then to features that are great to have but are not the core of what represents the platform. Foundations should aim to solve problems of trust and power that motivated us in our needfinding. Features are ideas which improve the strength of any platform but aren’t holistic or don’t give it a high-level purpose. For example, "workers review all tasks before posting and also make all payment decisions" and "workers organize themselves into collectives" would be foundations, whereas task recommender systems, mediation strategies, and specific feedback and rating mechanisms would be features.

One way to determine what the foundational ideas of your proposed platform are, are by thinking about how you would pitch your platform. If you had to describe why your platform is better than any of the countless existing crowd marketplaces in a sentence or two, what would you say? These are your foundational ideas. If your don't think you can explain the novelty and advantages of your system in just a sentence or two, you may want to revisit the set of ideas you have picked (see the meteor links above) - perhaps you are proposing a bag of features, with less novel/foundational ideas.

As a team, choose one to two main foundations for your platform, and one to three core features that you think will be important.

You can draw the set of foundational or feature ideas for your platform by synthesizing ideas and prototypes that have been proposed in Milestones 3, 4, and 5:

http://crowdresearch3.meteor.com/

http://crowdresearch4.meteor.com/

http://crowdresearch5.meteor.com/

For more on foundations vs. features, see this week's slides.

Format of your research proposal

You will propose your platform in the form of an introduction to a mock research paper. Essentially, imagine you have built your system, incorporating in all the ideas that you wanted to have in it, have run your user studies and evaluations, and everything has gone as planned. How do you convince other researchers that you have built a platform that is novel and that it is more effective at addressing problems than any existing ideas that have been attempted in the past?

This is an extremely effective strategy for making your ideas concrete and persuasive.

An introduction of a research paper summarizes the main contributions of the research. It is generally roughly 1 page (roughly 1000 words), and consists of the following components.

Title

Research papers should have titles which summarize, in one line, the primary research contribution (see below).

Abstract

A concise summary of the background and your research contributions (described below). Usually around 150 words or less.

Each of the following sections should be one to two paragraphs each. Paper introductions are brief and impactful.

Motivation

What is the problem that you are solving, and why is it important?

  • Think about the needs we synthesized from Milestone 2 - ie, trust and power - when thinking about which problems you want to solve.
  • This should be a specific problem! Not just “crowdsourcing”. More like how trust and power are broken.

Related Work

What are the existing attempts to solve this problem that have been attempted in prior research papers and real-world systems? Why are their solutions unsatisfactory?

  • You may want to search around Google Scholar to find existing work that is related to the ideas your system proposes.

Insight

This section should lay out this the foundational idea(s). These big ideas are the things that you'll be known for, and what other platforms would want to replicate. Explain: Why/how are they novel and better than anything that has been attempted in the past?

System

The insight above should explain the high level idea (e.g., "All workers are paid in chocolate"). Here, you explain how it works in specifics. (e.g., "We built a crowdsourcing platform called Chococrowd that mails dark chocolate candies to workers at the conclusion of each month. Requesters choose the quality of the dark chocolate based on the quality of the work.")

Evaluation

Once the platform you propose has been implemented, how will you determine whether your system actually solves the problem you wanted to solve? What are the results you hope you can realistically achieve? Why do these results show that you have solved the problem?

References

The reference section is where you cite prior work that you build upon. If you are aware of existing related research papers, list them here. We also encourage you to borrow ideas from the past submissions (see the meteor links above). Please list the links of the ideas you used to create this proposal (there's no restriction in terms of number of ideas or whether its yours or others'). You can use the following template:

  • [Foundation Idea] Link...
  • [Feature Idea] Link...
  • [Foundation Idea] Link...so on, and so forth...

Read introductions of some existing papers

To see concrete examples of how HCI research papers that propose new platforms are structured, you should read the introductions of these papers. You will likely be familiar with the first two papers (mClerk and MobileWorks), as they were part of the Milestone 1 readings.

mClerk

Gupta A, Thies W, Cutrell E, et al. mClerk: enabling mobile crowdsourcing in developing regions. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2012: 1843-1852.

MobileWorks

Narula P, Gutheim P, Rolnitzky D, et al. MobileWorks: A Mobile Crowdsourcing Platform for Workers at the Bottom of the Pyramid. Human Computation, 2011, 11: 11.

Twitch Crowdsourcing

Vaish R, Wyngarden K, Chen J, et al. Twitch crowdsourcing: crowd contributions in short bursts of time. Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 2014: 3645-3654.

Submitting

Create a Wiki Page for your Team's Submission

Please create a page for your team's submission at http://crowdresearch.stanford.edu/w/index.php?title=Milestone_6_YourTeamName&action=edit (substituting in YourTeamName with the team name), copy over the template at Milestone 6 Research Proposal Template .

[Team Leaders] Post the links to your research proposals until 8th April 11:59 pm

We have a service on which you can post research proposals you generated, comment on them, and upvote ones you like.

http://crowdresearch.meteor.com/category/milestone-6-research-proposals

Post links to your research proposals only once they're finished. Give your posts the same title as your mock research paper. Do not include words like "Milestone", "Research Proposal", or your team name in the title.

-Please submit your finished research proposals by 11:59 pm 8th April 2015, and DO NOT vote/comment until 9th April 12:05 am

[Everyone] Peer-evaluation (upvote ones you like, comment on them) from 12:05 am 9th April until 9 am 10th April

Post submission phase, you are welcome to browse through, upvote, and comment on others' research proposals. We encourage you especially to look at and comment on submissions that haven't yet gotten feedback, to make sure everybody's submissions get feedback.

Step 1: Please use http://crowdresearch.meteor.com/needcomments to find submissions that haven't yet gotten feedback, and http://crowdresearch.meteor.com/needclicks to find submissions that haven't been yet been viewed many times.

Step 2: Once you find an idea of interest or less attended, please vote and comment upon it. Please perform this action from 3 to 5 submissions - this will help us balance the comments and votes. Please do not vote your team's research proposals. Once again, everyone is supposed to vote+comment, whether you're the team leader or not.

COMMENT BEST-PRACTICES: As on Crowdgrader, everybody reviews at least 3 submissions, supported by a comment. The comment should provide constructive feedback. Negative comments are discouraged - if you disliked some aspect of a submission, make a suggestion for improvement.

[Team Leaders] Milestone 6 Submissions

To help us track all submissions and browsing through them, once you have finished your Milestone 6 Research Proposal submission, go to the link below and post the link:

Milestone 6 Research Proposal Submissions