Difference between revisions of "Milestone 1"

From crowdresearch
Jump to: navigation, search
(Submitting)
(Submitting)
 
(14 intermediate revisions by 2 users not shown)
Line 5: Line 5:
 
* Get experience of what it is like to be a worker and requester on current crowd-labor marketplaces
 
* Get experience of what it is like to be a worker and requester on current crowd-labor marketplaces
 
* Read some research papers about crowdsourcing
 
* Read some research papers about crowdsourcing
 +
* <strong>Youtube</strong> link of the meeting today: [http://www.youtube.com/watch?v=GiGrCATDsHg watch]
 +
* <strong>Meeting 1 slideshow</strong>: [[:Media:Meeting1 Slideshow.pdf | pdf]]
  
 
== Experience the life of a Worker on Mechanical Turk ==
 
== Experience the life of a Worker on Mechanical Turk ==
  
Sign up as a worker for Mechanical Turk [https://www.mturk.com/ here]. Then, practice doing some tasks - you should do enough tasks to earn $1.
+
Sign up as a worker for Mechanical Turk [https://www.mturk.com/ here]. Then, practice doing some tasks - you should do enough tasks to earn $1. You can check out forums like [http://www.turkernation.com/ Turker Nation] to find tips about working on MTurk.
  
If having difficulty signing up as a worker, you can try the [https://workersandbox.mturk.com worker sandbox], or other sites like [http://www.crowdflower.com/ CrowdFlower] or [https://microworkers.com/ Microworkers]. Your peers will post some worker sandbox HITs at [[Milestone 1 Sandbox HITs]].
+
If having difficulty signing up as a worker, you can try the [https://workersandbox.mturk.com worker sandbox], or other sites like [http://www.crowdflower.com/ CrowdFlower] or [https://microworkers.com/ Microworkers] or [http://www.clickworker.com/en Clickworker]. Your peers will post some worker sandbox HITs at [[Milestone 1 Sandbox HITs]].
  
 
=== Deliverable ===
 
=== Deliverable ===
Line 18: Line 20:
 
== Experience the life of a Requester on Mechanical Turk ==
 
== Experience the life of a Requester on Mechanical Turk ==
  
Sign up as a requester for Mechanical Turk [https://requester.mturk.com/ here]. Then, post some HITs to tackle a task of interest to you. Please aim to follow best practices for designing these tasks --- be clear in your task description, and do not arbitrarily reject work. If you need funds to support your task, email crowdresearch@cs.stanford.edu and we can PayPal you a few dollars. If you don't want to spend money, you can post them via the [https://requestersandbox.mturk.com/ requester sandbox], list them on [[Milestone 1 Sandbox HITs]] and ask people to do them for you. Aim to have your HITs done by at least 15 different people.
+
Sign up as a requester for Mechanical Turk [https://requester.mturk.com/ here]. Then, post some HITs to tackle a task of interest to you. Please aim to follow best practices for designing these tasks --- be clear in your task description, and do not arbitrarily reject work. If you need funds to support your task, please create a paypal account and fill out [https://docs.google.com/forms/d/1m5ZeeHOfqy7WUUE9alYdKujATyeLrcXRFMa6t3p8-bE/viewform this form] so we can PayPal you a few dollars. If you don't want to spend money, you can post them via the [https://requestersandbox.mturk.com/ requester sandbox], list them on [[Milestone 1 Sandbox HITs]] and ask people to do them for you. Aim to have your HITs done by at least 15 different people.
  
 
Examples of tasks you might want to have done (you can pick one of the tasks below, or do a task of your own choosing):
 
Examples of tasks you might want to have done (you can pick one of the tasks below, or do a task of your own choosing):
Line 59: Line 61:
  
 
[[:Media:Flash Teams (private).pdf | Retelny D, Robaszkiewicz S, To A, et al. Expert crowdsourcing with flash teams. Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 2014: 75-85. ]]
 
[[:Media:Flash Teams (private).pdf | Retelny D, Robaszkiewicz S, To A, et al. Expert crowdsourcing with flash teams. Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 2014: 75-85. ]]
 +
 +
=== Other Recommended Readings ===
 +
 +
These readings are optional (you don't need to write up on them), but recommended:
 +
 +
[http://www.cs.cmu.edu/~jbigham/posts/2014/half-workday-as-turker.html My MTurk (half) Workday]
  
 
=== Deliverable ===
 
=== Deliverable ===
Line 69: Line 77:
 
== Submitting ==
 
== Submitting ==
  
Please create a page for your team's submission at http://crowdresearch.stanford.edu/w/index.php?title=Milestone_1_YourTeamName&action=edit (substituting in YourTeamName with the team name), copy over the template at [[Milestone 1 Template]] .
+
Please create a page for your team's submission at http://crowdresearch.stanford.edu/w/index.php?title=Milestone_1_YourTeamName&action=edit (substituting in YourTeamName with the team name), copy over the template at [[Milestone 1 Template]] . If you have never created a wiki page before, please see [http://www.mediawiki.org/wiki/Help:Starting_a_new_page this] or watch [https://www.youtube.com/watch?v=83-lCpAnaFw this].  
  
Then, after you have filled out the page, post the link to the wiki page you created on CrowdGrader!  
+
Then, after you have filled out the page, post the link to the wiki page you created on <strong>CrowdGrader</strong>!  
  
 
<strong>Step 1 for everyone</strong> - For the most of you, you don't have to enroll. I have done it for you. You can directly go to: http://www.crowdgrader.org/crowdgrader/venues/view_venue/860 . However, if you cannot access it, please self-enroll using this link: http://www.crowdgrader.org/crowdgrader/venues/join/860/tofypy_firice_dadepe_rutane  
 
<strong>Step 1 for everyone</strong> - For the most of you, you don't have to enroll. I have done it for you. You can directly go to: http://www.crowdgrader.org/crowdgrader/venues/view_venue/860 . However, if you cannot access it, please self-enroll using this link: http://www.crowdgrader.org/crowdgrader/venues/join/860/tofypy_firice_dadepe_rutane  
 
   
 
   
<strong>Step 2 for team leaders</strong>: Make sure all of your team members have enrolled into the system (though I have done it for you, please double check). ''Now add them to your group/team - there's an option to add collaborators''. You also have to keep a check on deadlines, and whether your team is co-operating with you.  
+
<strong>Step 2 for team leaders</strong>: Make sure all of your team members have enrolled into the system (though I have done it for you, please double check). ''Now add them to your group/team - there's an option to add collaborators'' (your team members might get an email, and have to confirm before it shows up as your collaborators). You also have to keep a check on deadlines, and whether your team is co-operating with you.  
 
   
 
   
 
<strong>Step 3 for team leaders</strong>: Make the submission and represent your team, only team leaders should make the submission.  
 
<strong>Step 3 for team leaders</strong>: Make the submission and represent your team, only team leaders should make the submission.  
 
<strong>Step 4 for everyone</strong>: Begin peer-evaluation. Everyone will be randomly assigned 3 submissions to grade, and 25% of your grades depend on your duty to peer-grade others - check Crowdgrader to find and grade the submissions. - Please comment and justify why you gave this score, and point out good/bad points about the submission. Team leaders, please make sure that every member of your team grades the submissions.
 
  
<strong>Step 5 for team leaders</strong>: Make an entry of your submission page here -> [[Milestone 1 Submissions]]
+
<strong>Step 4 for team leaders</strong>: Make an entry of your submission page here -> [[Milestone 1 Submissions]]
 +
 +
<strong>Step 5 for everyone</strong>: Begin peer-evaluation. Everyone will be randomly assigned 3 submissions to grade, and 25% of your grades depend on your duty to peer-grade others - check Crowdgrader to find and grade the submissions. - Please comment and justify why you gave this score, and point out good/bad points about the submission. Team leaders, please make sure that every member of your team grades the submissions.
  
 
== Milestone 1 Submissions ==  
 
== Milestone 1 Submissions ==  

Latest revision as of 12:43, 4 March 2015

Due date: 11:59 pm 4th March 2015 for submission, 9 am 6th March 2015 for peer-evaluation.

In this milestone we want you to:

  • Get experience of what it is like to be a worker and requester on current crowd-labor marketplaces
  • Read some research papers about crowdsourcing
  • Youtube link of the meeting today: watch
  • Meeting 1 slideshow: pdf

Experience the life of a Worker on Mechanical Turk

Sign up as a worker for Mechanical Turk here. Then, practice doing some tasks - you should do enough tasks to earn $1. You can check out forums like Turker Nation to find tips about working on MTurk.

If having difficulty signing up as a worker, you can try the worker sandbox, or other sites like CrowdFlower or Microworkers or Clickworker. Your peers will post some worker sandbox HITs at Milestone 1 Sandbox HITs.

Deliverable

Reflect on your experience as a worker on Mechanical Turk. What did you like? What did you dislike? Write it up on this wiki, as described here

Experience the life of a Requester on Mechanical Turk

Sign up as a requester for Mechanical Turk here. Then, post some HITs to tackle a task of interest to you. Please aim to follow best practices for designing these tasks --- be clear in your task description, and do not arbitrarily reject work. If you need funds to support your task, please create a paypal account and fill out this form so we can PayPal you a few dollars. If you don't want to spend money, you can post them via the requester sandbox, list them on Milestone 1 Sandbox HITs and ask people to do them for you. Aim to have your HITs done by at least 15 different people.

Examples of tasks you might want to have done (you can pick one of the tasks below, or do a task of your own choosing):

  • Transcribe an audio recording
  • Get descriptions for images
  • Label parts of speech in a sentence

Deliverable

Reflect on your experience as a requester on Mechanical Turk. What did you like? What did you dislike? Also attach the CSV file generated when you download the HIT results.

Explore alternative crowd-labor markets

Crowdsourcing is not only about Mechanical Turk, its much more than that. And a variety of platforms are trying to address different problems. To help you get a flavor of the diversity of the area, choose one of the following crowdsourcing sites:

Explore it and a get a feel for how the site works. If you want to, try doing a job on the site. You should not spend any more than 30-45min on this part.

Deliverable

Compare and contrast the crowd-labor market you just explored (TaskRabbit/oDesk/GalaxyZoo) to Mechanical Turk.

Readings

Please skim over the following papers. They're copyrighted, so please don't redistribute. You need to be signed in to view them.

MobileWorks

Narula P, Gutheim P, Rolnitzky D, et al. MobileWorks: A Mobile Crowdsourcing Platform for Workers at the Bottom of the Pyramid. Human Computation, 2011, 11: 11.

mClerk

Gupta A, Thies W, Cutrell E, et al. mClerk: enabling mobile crowdsourcing in developing regions. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2012: 1843-1852.

Flash Teams

Retelny D, Robaszkiewicz S, To A, et al. Expert crowdsourcing with flash teams. Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 2014: 75-85.

Other Recommended Readings

These readings are optional (you don't need to write up on them), but recommended:

My MTurk (half) Workday

Deliverable

For each system, please write down:

  • What do you like about the system / what are its strengths?
  • What do you think can be improved about the system?

Submitting

Please create a page for your team's submission at http://crowdresearch.stanford.edu/w/index.php?title=Milestone_1_YourTeamName&action=edit (substituting in YourTeamName with the team name), copy over the template at Milestone 1 Template . If you have never created a wiki page before, please see this or watch this.

Then, after you have filled out the page, post the link to the wiki page you created on CrowdGrader!

Step 1 for everyone - For the most of you, you don't have to enroll. I have done it for you. You can directly go to: http://www.crowdgrader.org/crowdgrader/venues/view_venue/860 . However, if you cannot access it, please self-enroll using this link: http://www.crowdgrader.org/crowdgrader/venues/join/860/tofypy_firice_dadepe_rutane

Step 2 for team leaders: Make sure all of your team members have enrolled into the system (though I have done it for you, please double check). Now add them to your group/team - there's an option to add collaborators (your team members might get an email, and have to confirm before it shows up as your collaborators). You also have to keep a check on deadlines, and whether your team is co-operating with you.

Step 3 for team leaders: Make the submission and represent your team, only team leaders should make the submission.

Step 4 for team leaders: Make an entry of your submission page here -> Milestone 1 Submissions

Step 5 for everyone: Begin peer-evaluation. Everyone will be randomly assigned 3 submissions to grade, and 25% of your grades depend on your duty to peer-grade others - check Crowdgrader to find and grade the submissions. - Please comment and justify why you gave this score, and point out good/bad points about the submission. Team leaders, please make sure that every member of your team grades the submissions.

Milestone 1 Submissions

To help us track all submissions and browsing through them. Once you create your milestone 1 page. Go to the link below and post it.

Milestone 1 Submissions