Welcome to the wiki for the Crowd Research project!
About the project
Whether you need help gathering data, labeling machine learning training examples, running experiments, or transcribing audio, today we use crowdsourcing platforms such as Amazon Mechanical Turk (www.mturk.com). However, these platforms are notoriously bad at ensuring high quality results, producing respect and fair wages for workers, and making it easy to author effective tasks. It’s not hard to imagine that we could do better.
This research will be a complete design, implementation, launch, and evaluation of a new crowdsourcing platform. What would it take to create an effective marketplace? One where workers have more power in the employment relationship, or could take additional responsibility for the result quality? How might we design such a market? Could we launch it and become the new standard? This research in human-computer interaction will involve a combination of design thinking, web development, and experimental design. This is far more ambitious than your typical project. It’s an entire marketplace design question. Thus, we’re banding together to solve it.
Well, first, there’s creating a crowdsourcing market that becomes the new standard. This could lead to a far better future for crowdsourcing and crowd work, and millions of people could eventually use it. It’s research, of course, so there’s always risk it might not work out — but if we knew it would work, it wouldn’t be research!
Second, we’ll be planning papers to top-tier conferences based on our work. If you are considering an MS or especially a PhD program, being a heavily contributing author on a paper can greatly improve your chances. How much you contribute to the project will determine author order. Last, I really do hope to build relationships with a diverse range of researchers.
Meetings and Slides
All timings in PST (California time)
Milestone 1 - 11:59 pm 4th March 2015 for submission, 9 am 6th March 2015 for peer-evaluation.
Milestone 2 - 11:59 pm 11th March 2015 for submission, 9 am 13th March 2015 for peer-evaluation. Everyone, please note that the Daylight Saving Time (United States) 2015 begins at 2:00 AM on Sunday, March 8. People out of the US might want to track the new changes.
Milestone 3 - 11:59 pm 18th March 2015 for submission, 9 am 20th March 2015 for voting and commenting on others' ideas.
Milestone 4 - 11:59 pm 25th March 2015 for submission, 9 am 27th March 2015 for voting and commenting on others' ideas.
Milestone 5 - 11:59 pm 1st April 2015 for submission, 9 am 3rd April 2015 for voting and commenting on others' prototypes.
Milestone 6 - 11:59 pm 8th April 2015 for submission, 9 am 10th April 2015 for voting and commenting on others' prototypes.
Milestone 7 - 11:59 pm 15th April 2015 for submission, 9 am 17th April 2015 for voting and commenting on others' prototypes.
Milestone 8 - 11:59 pm 22nd April 2015 for submission, 9 am 24th April 2015 for voting and commenting on others' prototypes.
Milestone 9 - 11:59 pm 30th April 2015 for submission, 9 am 2nd May 2015 for voting and commenting on others' prototypes.
Infrastructure - homepage for Infrastructure related efforts, contains separate milestones.
General Weekly Plan
We’ll meet weekly over videochat and lay out our goals for the next week. At the end of the week, you’ll submit what you’ve been working on. Your peers and a Ph.D. student here at Stanford will peer critique the work, and we’ll talk about the best stuff each week in our meeting. The sky’s the limit.
I’m sure we’ll adjust this as we go. Because, this entire crowdsourced research idea is a bit of a research project in itself, too.
- Saturday morning 9 am PST: Prof meeting with participants and milestone set for the next week (over Google Hangout on Air)
- Saturday after meeting - Wednesday midnight PST: participants work on their milestones (~5 days)
- Post Wednesday midnight PST - Friday morning 9 am PST: peer-evaluation by the participants (1+ days)
- Friday post 9 am PST - Friday evening PST: Research Assistants (RAs) check the top submissions and meet the Prof in evening. Pre-weekly meeting discussions happen. Milestones designed for the upcoming week etc.
- Saturday morning 9 am PST: Prof meets based on the input from RAs and top submissions. Participants receive their next milestone and a feedback survey after every meeting.
- Slack - used for chat and discussion
- Meteor - used for voting, evaluation and comments
- Forums - discussions about this project on external sites. Note that all official announcements and communications will occur via Slack and email.
Want to share some other resources? Create a wiki page, and post it at Resources.
About this wiki
You need to be logged in to edit this wiki and view private pages. If you already have an account but cannot login, try resetting your password. If you need an account, please contact Geza (@geza on slack) and you will be emailed your login details.
All pages on this wiki are public by default - so anyone visiting this site can see them.
Any page which ends with (private) will be private - so you need to be logged in to see them. See example.