Winter Milestone 5
Due date (PST): 8:00 pm 14th Feb 2016 for submission, 12 pm 15th Feb 2016 for peer-evaluation
This week, we will accept proposals to pursue different aspects of the project, and start a design test run.
- Youtube link of the task feed meeting 1 today: watch
- Youtube link of the task feed meeting 2 today: watch
- Youtube link of the task authoring meeting 1 today: watch
- Youtube link of the task authoring meeting 2 today: watch
- Archive of Milestone 4 submissions
- 1 Research: Dive deep into these topics and submit a proposal (All)
- 2 Research Engineering (All)
- 3 Design (Test Flight)
- 4 Submission
Research: Dive deep into these topics and submit a proposal (All)
Last week we asked you to pitch research ideas. This week we ask you to choose at least one the themes below, and dive deep into it. The goal is to propose a concrete research project idea: one that we could execute in the coming months and submit to UIST or CSCW. This may take the form of a system or social infrastructure we build into Daemo, or an experiment or study we run on it. To express this idea as a concise research concept, write it in the form of an introduction section to a research paper. Based on your ideas submitted last week, we have synthesized the themes into three. The submissions that are most viable and popular will be set as our direction for our research project, and the submitters may be asked to help us lead it!.
The typical narrative is that workers produce highly varying quality work. Nobody has studied the effect that requester quality has. Can we quantify the variance in requester quality, and introduce interventions to help improve it?
A specific idea raised in last week's submissions that gathered a lot of interest: could we run an experiment to demonstrate how much variance there is in requester quality for the same authoring task, vs. how much variation there is in worker quality?
b. Task ranking
Requesters don't get ideal workers, and workers don't get ideal requesters - can we rank the relevant tasks, on the basis of reputation, skills, and other necessary aspects?
A specific idea raised in last week's submissions that gathered a lot of interest: could we design an task feed ranking interface and algorithm for Daemo? A combination of user-centered work and machine learning/data mining?
c. Open gov
In current systems, worker and requester voices remain unheard; and the platform is run by a central organization with all control. Can we infuse the idea of open governance in Daemo?
A specific idea raised in last week's submissions that gathered a lot of interest: could workers form guilds for specific expertise areas, then run their own reputation and ranking operations? Like how doctors and lawyers administer their own tests for determining whether to license you?
We've synthesized some of the most popular ideas for each area. Grab at least one area (task authoring, task ranking, open gov), and an idea (not necessarily yours), and develop it further into a concrete research proposal! There are two type of proposals you can write - systems (where the novel contribution is a software system or platform to solve problems) and science (where the novel contribution is a phenomenon or an approach or a study to understand certain behavior that solves the problem). In your write up, follow the outline format below. Check out this really helpful paper.
After you choose one of the themes above, and decide to solve it through a system, write the system focussed introduction section using the outline below. You can see example introductions below.
- Example1: Gupta A, Thies W, Cutrell E, et al. mClerk: enabling mobile crowdsourcing in developing regions. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2012: 1843-1852.
- Example2: Narula P, Gutheim P, Rolnitzky D, et al. MobileWorks: A Mobile Crowdsourcing Platform for Workers at the Bottom of the Pyramid. Human Computation, 2011, 11: 11.
- Example3: Vaish R, Wyngarden K, Chen J, et al. Twitch crowdsourcing: crowd contributions in short bursts of time. Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 2014: 3645-3654.
After you choose one of the themes above, and decide to solve it through a science, write the phenomenon focussed introduction section using the outline below. You can see example introductions below.
- Example1: Steven P. Dow, Alana Glassco, Jonathan Kass, Melissa Schwarz, Daniel L. Schwartz, and Scott R. Klemmer. 2010. Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. ACM Trans. Comput.-Hum. Interact. 17, 4, Article 18 (December 2010)
- Example2: Carrie Cai, Shamsi Iqbal, and Jaime Teevan. Chain Reactions: The Impact of Order on Microtask Chains. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2016), San Jose, CA, May 2016
- Example3: Cheng, J., Teevan, J. & Bernstein, M.S. (2015). Measuring Crowdsourcing Effort with Error-Time Curves. CHI 2015.
Research Engineering (All)
Great job on the issues from last week. For those of you who have been doing tutorials and reading documentation now it's time to put that into practice. This week we will improve and finalize some of the issues we started last week, and we will also complete new ones.
- issue-forms (done), issue-profile (almost there, finalizing), issue-csv (cont'd this week) and issue-task (to be finalized)
- new issues: #645 (timer), #646 (advanced project properties), #647 (template item data_source), #648 (bug, task creation)
announce in #research-engineering that you are working on a particular issue and please let the others know about the progress of the issues you are working on (so that we don't do duplicate work). You are encouraged to work together.
For any questions ping @aginzberg, @dmorina, and @shirish.goyal on Slack #research-engineering
Design (Test Flight)
DRI: @karolina and @michaelbernstein
Calling all enthusiastic designers onboard. After a long wait, we're starting the design test run this week. The goal is to design a mockup/wireframe of a messaging system of how workers and requesters would talk to each other, specially while working on the tasks. Messaging could be key to Daemo, since it would help requester clarify a given task, and worker can reach out to them to get feedback on their work. Though MTurk has basic email system for communication, Upwork has borrowed ideas from Slack and created their messaging system. What else can we do?
Use Balsamiq or any tool of your choice, and share your unique mockup/wireframe of the messaging system on Daemo.
Create a Wiki Page for your Team's Submission
Create a wiki page with the introduction section, diving deep into one the three themes (look at the outline format above). If you're participating in design test run, create another wiki and paste screenshots or mockup/wireframe files. If you have never created a wiki page before, please see this or watch this.
We have a [Reddit like service] on which you can post the links to the wiki-pages for the submissions, explore them, and upvote them.
Please use the same login avenue (Facebook, Twitter, or email address) as you’ve done in the past with Meteor. This will help us identify and track your contributions better.
For newcomers joining Crowd Research, when it asks you to pick your username, pick the same username as your Slack. Please DO NOT forget to mention the milestone contributors' slackid below each wiki page.
On Meteor, there are 4 submission categories, 3 for research theme, and 1 for design test run.
1- [One of three mandatory] http://crowdresearch.meteor.com/category/task-rank where you can post a link to the wiki page for your task ranking proposal
2- [One of three mandatory] http://crowdresearch.meteor.com/category/task-author where you can post a link to the wiki page for your task author proposal
3- [One of three mandatory] http://crowdresearch.meteor.com/category/open-gov where you can post a link to the wiki page for your open gov proposal
4- [Test flight] http://crowdresearch.meteor.com/category/design-messenger where you can post mockup/wireframe of the Daemo messenger.
Give your posts titles which summarize your idea. Viewers should be able to get the main point by skimming the title ("Automatic Suggestion for Tasks based on Average Completion Time" is a good title. "YourTeam TrustIdea 1" is a bad title).
-Please submit your finished ideas by 8:00 pm 7th Feb 2016, and DO NOT vote/comment until then
[Design Test Flight]
For your Messenger Design Wiki submissions, please create Google Slides showing the flow of your designs and link it in your Wiki page. For example, see this slide show
When you create the "Share" link, make sure to click “Anyone with the link can view"
[Everyone] Peer-evaluation (upvote ones you like, comment on them) from 8:05 pm 7th Feb until 12 pm 8th Feb 2016
Post submission phase, you are welcome to browse through, upvote, and comment on others' ideas. We encourage you especially to look at and comment on ideas that haven't yet gotten feedback, to make sure everybody's ideas gets feedback. You can use http://crowdresearch.meteor.com/needcomments to find ideas that haven't yet gotten feedback, and http://crowdresearch.meteor.com/needclicks to find ideas that haven't been yet been viewed many times.
COMMENT BEST-PRACTICES: Everybody in the team reviews at least 3 ideas, supported by a comment. The comment has to justify your reason for upvote. The comment should be constructive, and should mention positive aspect of the idea worth sharing. Negative comments are discouraged, rather make your comment in the form of a suggestion - such as, if you disliked an idea, try to suggest improvements (do not criticize an idea, no idea is bad, every idea has a scope of improvement).