Crowd Research Initiative

Open and Scalable University Laboratories



Read our paper

What is Crowd Research?

Research experiences today are limited to a privileged few at select universities. Providing open access to research experiences would enable global upward mobility and increased diversity in the scientific workforce. How can we coordinate a crowd of diverse volunteers on open-ended research? How could a PI have enough visibility into each person’s contributions to recommend them for further study? Crowd Research is a crowdsourcing technique for coordinating a large group of people in an open-ended research exploration, and a system for decentralized credit distribution to aid upward mobility and recognize contributions in publications.

Originally launched as the UC Santa Cruz+Stanford: The Aspiring Researcher Challenge in early 2015, the effort has provided access to over 1,500 people from 62 countries — 74% from institutions with low access to research — to work on three computer science research projects, led by professors from Stanford, UC Santa Cruz and Cornell Tech. Together, the crowd helped design the world's largest wisdom of crowd experiment, built hybrid human-computer vision algorithms, and developed a new pro-social crowdsourcing marketplace (Daemo). During this time, the crowd co-authored multiple papers at top-tier conferences and have gone on to undergraduate and graduate schools including MIT, UC Berkeley, Stanford and Carnegie Mellon.

Projects and Publications

Meta Publications: About the process, the science behind Crowd Research!

Crowd Research: Open and Scalable University Laboratories
Rajan Vaish, Snehalkumar (Neil) S. Gaikwad, Geza Kovacs, Andreas Veit, Ranjay Krishna, Imanol Arrieta Ibarra, Camelia Simoiu, Michael Wilber, Serge Belongie, Sharad Goel, James Davis, Michael S. Bernstein
ACM UIST 2017

Crowdsourcing the Research Process
Rajan Vaish, James Davis, Michael S. Bernstein
Collective Intelligence 2015

Project 1: Design the Next-Generation Crowdsourcing Platform - Daemo

Advisor: Michael Bernstein, Stanford Computer Science
Research Area: Human-Computer Interaction
Links: Project link, Daemo link

Crowd Guilds: Worker-led Reputation and Feedback on Crowdsourcing Platforms
Mark E. Whiting, Dilrukshi Gamage, Snehalkumar (Neil) S. Gaikwad, Aaron Gilbee, Shirish Goyal, Alipta Ballav, Dinesh Majeti, Nalin Chhibber, Angela Richmond-Fuller, Freddie Vargus, Tejas Seshadri Sarma, Varshine Chandrakanthan, Teogenes Moura, Mohamed Hashim Salih, Gabriel Bayomi Tinoco Kalejaiye, Adam Ginzberg, Catherine A. Mullings, Yoni Dayan, Kristy Milland, Henrique Orefice, Jeff Regino, Sayna Parsi, Kunz Mainali, Vibhor Sehgal, Sekandar Matin, Akshansh Sinha, Rajan Vaish, Michael S. Bernstein
ACM CSCW 2017

Boomerang: Rebounding the Consequences of Reputation Feedback on Crowdsourcing Platforms
Snehalkumar (Neil) S. Gaikwad, Durim Morina, Adam Ginzberg, Catherine Mullings, Shirish Goyal, Dilrukshi Gamage, Christopher Diemert, Mathias Burton, Sharon Zhou, Mark Whiting, Karolina Ziulkoski, Alipta Ballav, Aaron Gilbee, Senadhipathige S. Niranga, Vibhor Sehgal, Jasmine Lin, Leonardy Kristianto, Angela Richmond-Fuller, Jeff Regino, Nalin Chhibber, Dinesh Majeti, Sachin Sharma, Kamila Mananova, Dinesh Dhakal, William Dai, Victoria Purynova, Samarth Sandeep, Varshine Chandrakanthan, Tejas Sarma, Sekandar Matin, Ahmed Nasser, Rohit Nistala, Alexander Stolzoff, Kristy Milland, Vinayak Mathur, Rajan Vaish, Michael S. Bernstein
ACM UIST 2016

Prototype Tasks: Improving Crowdsourcing Results through Rapid, Iterative Task Design
Snehalkumar (Neil) S. Gaikwad, Nalin Chhibber, Vibhor Sehgal, Alipta Ballav, Catherine Mullings, Ahmed Nasser, Angela Richmond-Fuller, Aaron Gilbee, Dilrukshi Gamage, Mark Whiting, Sharon Zhou, Sekandar Matin, Senadhipathige Niranga, Shirish Goyal, Dinesh Majeti, Preethi Srinivas, Adam Ginzberg, Kamila Mananova, Karolina Ziulkoski, Jeff Regino, Tejas Sarma, Akshansh Sinha, Abhratanu Paul, Christopher Diemert, Mahesh Murag, William Dai, Manoj Pandey, Rajan Vaish, Michael S. Bernstein
AAAI HCOMP 2017 (Short Paper)

Designing A Constitution for a Self-Governing Crowdsourcing Marketplace
Mark E. Whiting, Dilrukshi Gamage, Shirish Goyal, Aaron Gilbee, Dinesh Majeti, Angela Richmond-Fuller, Mohamed Hashim Salih, Tejas Seshadri Sarma, Vinayak Mathur, Manoj Pandey, Snehalkumar (Neil) S. Gaikwad, Rajan Vaish and Michael S. Bernstein
Collective Intelligence 2017

The Daemo Crowdsourcing Marketplace
Snehalkumar (Neil) S. Gaikwad, Mark E. Whiting, Dilrukshi Gamage, Catherine A. Mullings, Dinesh Majeti, Shirish Goyal, Aaron Gilbee, Nalin Chhibber, Adam Ginzberg, Angela Richmond-Fuller, Sekandar Matin, Vibhor Sehgal, Tejas Seshadri Sarma, Ahmed Nasser, Alipta Ballav, Jeff Regino, Sharon Zhou, Kamila Mananova, Preethi Srinivas, Karolina Ziulkoski, Dinesh Dhakal, Alexander Stolzoff, Senadhipathige S. Niranga, Mohamed Hashim Salih, Akshansh Sinha, Rajan Vaish, and Michael S. Bernstein
ACM CSCW 2017 (Demo)

Daemo: a Self-Governed Crowdsourcing Marketplace
Snehalkumar (Neil) S. Gaikwad, Durim Morina, Rohit Nistala, Megha Agarwal, Alison Cossette, Radhika Bhanu, Saiph Savage, Vishwajeet Narwal, Karan Rajpal, Jeff Regino, Aditi Mithal, Adam Ginzberg, Aditi Nath, Karolina Ziulkoski, Trygve Cossette, Dilrukshi Gamage, Angela Richmond-Fuller, Ryo Suzuki, Jeerel Herrejon, Kevin Le, Claudia Flores-Saviaga, Haritha Thilakarathne, Kushagra Gupta, William Dai, Ankita Sastry, Shirish Goyal, Thejan Rajapakshe, Niki Abolhassani, Angela Xie, Abigail Reyes, Surabhi Ingle, Sofia Jaramillo, Martin Daniel Godinez, Walter Angel, Carlos Toxtli, Asmita Gupta, Vineet Sethia, D. Padilla, Kristy Milland, Kristiono Setyadi, Walikonthegei Nuwan Theekshana Wajirasena, Muditha Batagoda, Rolando Cruz, James Damon, Divya Nekkanti, Tejas Sarma, Mohammed Hashim Saleh, Gaby Gongora-Svartzman, Soroosh Bateni, G. Toledo-Barrera, Alex Pena, Ryan Compton, Deen Aariff, L. Palacios, Manuela Paula Ritter, Nisha K.K., Alan Kay, Jana Uhrmeister, Srivalli Nistala, Milad Nasr Esfahani, Elsa Bakiu, Christopher Diemert, Luca Matsumoto, Manik Singh, V. Jaramillo-Lopez, Krupa Patel, Ranjay Krishna, Geza Kovacs, Rajan Vaish, and Michael S. Bernstein
ACM UIST 2015 (Short Paper)

Project 2: Hybrid Human-Computer Vision Algorithms

Advisors: James Davis, UC Santa Cruz and Serge Belongie, Cornell Tech
Research Area: Computer Vision

On Optimizing Human-Machine Task Assignments
Andreas Veit, Michael Wilber, Rajan Vaish, Serge Belongie, James Davis, Vishal Anand, Anshu Aviral, Prithvijit Chakrabarty, Yash Chandak, Sidharth Chaturvedi, Chinmaya Devaraj, Ankit Dhall, Utkarsh Dwivedi, Sanket Gupte, Sharath N. Sridhar, Karthik Paga, Anuj Pahuja, Aditya Raisinghani, Ayush Sharma, Shweta Sharma, Darpana Sinha, Nisarg Thakkar, K. Bala Vignesh, Utkarsh Verma, Kanniganti Abhishek, Amod Agrawal, Arya Aishwarya, Aurgho Bhattacharjee, Sarveshwaran Dhanasekar, Venkata Karthik Gullapalli, Shuchita Gupta, Chandana G, Kinjal Jain, Simran Kapur, Meghana Kasula, Shashi Kumar, Parth Kundaliya, Utkarsh Mathur, Alankrit Mishra, Aayush Mudgal, Aditya Nadimpalli, Munakala Sree Nihit, Akanksha Periwal, Ayush Sagar, Ayush Shah, Vikas Sharma, Yashovardhan Sharma, Faizal Siddiqui, Virender Singh, Abhinav S., Pradyumna Tambwekar, Rashida Taskin, Ankit Tripathi, Anurag. D. Yadav
AAAI HCOMP 2015 (Short Paper)


Project 3: Exploring the Wisdom of Crowds

Advisor: Sharad Goel, Stanford Management Science and Engineering
Research Area: Data Science
Links: Project link

Studying the ‘Wisdom of Crowds’ at Scale
Camelia Simoiu, Chiraag Sumanth, Alok Mysore, Sharad Goel
AAAI HCOMP 2019


Investigating the ‘Wisdom of Crowds’ at Scale
Alok Shankar Mysore, Vikas S Yaligar, Imanol Arrieta Ibarra, Camelia Simoiu, Sharad Goel, Ramesh Arvind, Chiraag Sumanth, Arvind Srikantan, Bhargav HS, Mayank Pahadia, Tushar Dobhal, Atif Ahmed, Mani Shankar, Himani Agarwal, Rajat Agarwal, Sai Anirudh Kondaveeti, Shashank Arun-Gokhale, Aayush Attri, Arpita Chandra, Yogitha Chilukuri, Sharath Dharmaji, Deepak Garg, Naman Gupta, Paras Gupta, Glincy Mary Jacob, Siddharth Jain, Shashank Joshi, Tarun Khajuria, Sameeksha Khillan, Sandeep Konam, Praveen Kumar-Kolla, Sahil Loomba, Rachit Madan, Akshansh Maharaja, Vidit Mathur, Bharat Munshi, Mohammed Nawazish, Venkata Neehar-Kurukunda, Venkat Nirmal-Gavarraju, Sonali Parashar, Harsh Parikh, Avinash Paritala, Amit Patil, Rahul Phatak, Mandar Pradhan, Abhilasha Ravichander, Krishna Sangeeth, Sreecharan Sankaranarayanan, Vibhor Sehgal, Ashrith Sheshan, Suprajha Shibiraj, Aditya Singh, Anjali Singh, Prashant Sinha, Pushkin Soni, Bipin Thomas, Lokesh Tuteja, Kasyap VarmaDattada, Sukanya Venkataraman, Pulkit Verma, Ishan Yelurwar
ACM UIST 2015 (Short Paper)

Team

Rajan Vaish, Stanford

Snehal (Neil) S. Gaikwad, MIT

Geza Kovacs, Stanford

Andreas Veit, Cornell Tech

Ranjay Krishna, Stanford

Imanol Arrieta Ibarra, Stanford

Camelia Simoiu, Stanford

Michael Wilber, Cornell Tech

Serge Belongie, Cornell Tech

Sharad Goel, Stanford

James Davis, UC Santa Cruz

Michael Bernstein, Stanford

Updates and Impact

Press (10/23/2017):
A Stanford-led platform for crowdsourced research gives experience to global participants, Stanford News.

Press (08/23/2017):
Amazon's Turker Crowd Has Had Enough, Wired.

Daemo's Impact:
Daemo crowdsourcing marketplace has been successfully used to create a dataset of 100,000+ question­ answer pairs on 500+ Wikipedia articles — a paper based on this work recently won the best paper award at EMNLP 2016.

Daemo's Impact:
The HCI project's Daemo got into the finals of Knights News Challenge'15 (top 20 of 1,000+).

Upward Mobility:
Following members sought our letters to apply for various positions (random order):

- Karthik Paga (before: VIT University India; now: Carnegie Mellon)
- Chiraag Sumanth (before: PES Institute India; now: Stanford)
- Alok Shankar (before: PES Institute India; now: UCSD)
- Praveen Kumar Kolla (before: IIT Guwahati India; now: UC Davis)
- Bipin Thomas (before: Software Professional; now: RWTH Aachen University)
- William Dai (before: Tory High School USA; now: UC Berkeley)
- Ankita Sastry (before: SAP India; now: Carnegie Mellon)
- Radhika Bhanu (before: InMobi India; now: Cornell)
- Karan Rajpal (before: Independent developer India; now: Cornell)
- Aditi Nath (before: AIG India; now: Arizona State)
- Luca Matsumoto (before: Wilcox High School USA; now: UCLA)
- Rijul Magu (before: Jaypee Institute India; now: University of Rochester)
- Zhou Juechi (before: Beijing University of Posts and Telecommunications China; now: Cornell Tech)
- Nalin Chhibber (before: Independent developer India; now: University of Waterloo)
- Anshu Aviral (before: Indus OS India; now: Carnegie Mellon)
- Ankit Dhall (before: VIT University India; now: ETH Zurich)
- Venkata Karthik Gullapalli (before: VIT University India; now: Cornell)
- Prithvijit Chakrabarty (before: VIT University India; now: University of Massachusetts, Amherst)
- Anmol Agarwal (before: Jaypee Institute India; now: Convegenius Software)
- Naveen John (before: NITK India; now: University of Washington)
- Alan James (before: NIT Surat India; now: Oracle)
- Pranav Channakeshava (before: NITK India; now: Carnegie Mellon)
- Nikita Gupta (before: IIIT Delhi India; now: Adobe)
- Glincy Mary Jacob (before: Cochin University of Science and Technology India; now: NEC Technologies)
- Aditi Mithal (before: Indraprastha Institute India; now: UCLA)
- Ashrith Sheshan (before: BMS College India; now: UCSD)
- Akshat Mishra (before: Delhi Technological University India; now: University of Michigan)
- Tejas Sarma (before: University of Mumbai India; now: University of Illinois Chicago)
- Vinayak Mathur (before: Manipal University India; now: University of Massachusetts, Amherst)

Contact us!

crowdresearch@cs.stanford.edu

Founding Partners


     


Sponsors


ISSDM LANL   HPI  Toyota  ONR