Winter Milestone 4
Due date (PST): 8:00 pm 7th Feb 2016 for submission, 12 pm 8th Feb 2016 for peer-evaluation
This week, we will accept proposals to pursue different aspects of the project, and start a design test run.
- 1 UX + Research: Dive deep into these topics and submit a proposal (All)
- 2 Research Engineering (All)
- 3 Design (Test Run)
- 4 Submission
UX + Research: Dive deep into these topics and submit a proposal (All)
Last week we asked you to pitch ideas in the space of representation, reputation and dark horse one's. This week we ask you to choose one the themes below, and dive deep into it. The goal is to propose experimental design and research direction of one of these ideas/themes, in the form of the "introduction" section. Based on your ideas submitted last week, we have essentially synthesized the themes into three. The one's that are more viable and popular will be set as the baseline for our next milestones, and the submitters may become the DRI.
Requesters are not expert at posting tasks, can we improve the process by helping them and involving workers?
b. Task ranking
Requesters don't get ideal workers, and workers don't get ideal requesters - can we rank the relevant tasks, on the basis of reputation, skills, and other necessary aspects?
c. Open gov
In current systems, worker and requester voices remain unheard; and the platform is run by a central organization with all control. Can we infuse the idea of open governance in Daemo?
We’ve synthesized some of the most popular ideas for each area. Grab at least one area (task authoring, task ranking, open gov), and an idea (not necessarily yours), and develop it further into a concrete research proposal! There are two type of proposals you can write - systems (where the novel contribution is a software system or platform to solve problems) and science (where the novel contribution is a phenomenon or an approach or a study to understand certain behavior that solves the problem). Check out this really helpful paper.
After you choose one of the themes above, and decide to solve it through a system, write the system focussed introduction section using the outline below.
- Example1: Gupta A, Thies W, Cutrell E, et al. mClerk: enabling mobile crowdsourcing in developing regions. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2012: 1843-1852.
- Example2: Narula P, Gutheim P, Rolnitzky D, et al. MobileWorks: A Mobile Crowdsourcing Platform for Workers at the Bottom of the Pyramid. Human Computation, 2011, 11: 11.
- Example3: Vaish R, Wyngarden K, Chen J, et al. Twitch crowdsourcing: crowd contributions in short bursts of time. Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 2014: 3645-3654.
After you choose one of the themes above, and decide to solve it through a science, write the phenomenon focussed introduction section using the outline below.
- Example1: Steven P. Dow, Alana Glassco, Jonathan Kass, Melissa Schwarz, Daniel L. Schwartz, and Scott R. Klemmer. 2010. Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. ACM Trans. Comput.-Hum. Interact. 17, 4, Article 18 (December 2010)
- Example2: Carrie Cai, Shamsi Iqbal, and Jaime Teevan. Chain Reactions: The Impact of Order on Microtask Chains. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2016), San Jose, CA, May 2016
- Example3: Cheng, J., Teevan, J. & Bernstein, M.S. (2015). Measuring Crowdsourcing Effort with Error-Time Curves. CHI 2015.
Research Engineering (All)
Great job on the issues from last week. For those of you who have been doing tutorials and reading documentation now it's time to put that into practice. This week we will improve and finalize some of the issues we started last week, and we will also complete new ones.
- issue-forms (done), issue-profile (almost there, finalizing), issue-csv (cont'd this week) and issue-task (to be finalized)
- new issues: #645 (timer), #646 (advanced project properties), #647 (template item data_source), #648 (bug, task creation)
announce in #research-engineering that you are working on a particular issue and please let the others know about the progress of the issues you are working on (so that we don't do duplicate work). You are encouraged to work together.
For any questions ping @aginzberg, @dmorina, and @shirish.goyal on Slack #research-engineering
Design (Test Run)
DRI: @karolina and @michaelbernstein
Calling all enthusiastic designers onboard. After a long wait, we're starting the design test run this week. The goal is to design a mockup/wireframe of a messaging system of how workers and requesters would talk to each other, specially while working on the tasks. Messaging could be key to Daemo, since it would help requester clarify a given task, and worker can reach out to them to get feedback on their work. Though MTurk has basic email system for communication, Upwork has borrowed ideas from Slack and created their messaging system. What else can we do?
Use Balsamiq or any tool of your choice, and share your unique mockup/wireframe of the messaging system on Daemo.
Create a Wiki Page for your Team's Submission
Please create 5 wiki pages for each topic - 1) reputation, 2) representation, 3) dark horse, 4) actionable and 5) long-term. See the template here for more details on "how to title and link different pages" WinterMilestone 3 Template . If you have never created a wiki page before, please see this or watch this.
We have a [Reddit like service] on which you can post the links to the wiki-pages for the submissions, explore them, and upvote them.
NEW INSTRUCTIONS - Create 5 wiki pages
Please use the same login avenue (Facebook, Twitter, or email address) as you’ve done in the past with Meteor. This will help us identify and track your contributions better.
For newcomers joining Crowd Research, when it asks you to pick your username, pick the same username as your Slack.
On Meteor, there are 5 submission categories, for each, create a separate wiki page - see WinterMilestone 3 Template for tips on how to set the title of your submission:
1- [Mandatory] http://crowdresearchw16-m3.meteor.com/?cat=reputation where you can post link to the wiki page for 1 reputation related idea you generated in the "Dive Deeper into Specific Ideas" stage
2- [Mandatory] http://crowdresearchw16-m3.meteor.com/?cat=representation where you can post link to the wiki page for 1 representation related idea you generated in the "Dive Deeper into Specific Ideas" stage
3- [Mandatory] http://crowdresearchw16-m3.meteor.com/?cat=dark-horse where you can post a link to the wiki page for your dark horse idea
4- [Test flight] http://crowdresearchw16-m3.meteor.com/?cat=design-test-flight-actionable-ideas where you can post actionable ideas based on your usability study/experience.
5- [Test flight] http://crowdresearchw16-m3.meteor.com/?cat=design-test-flight-long-term-idea where you can post long term ideas based on your usability study/experience.
Give your posts titles which summarize your idea. Viewers should be able to get the main point by skimming the title ("Automatic Suggestion for Tasks based on Average Completion Time" is a good title. "YourTeam TrustIdea 1" is a bad title).
-Please submit your finished ideas by 8:00 pm 31st January 2016, and DO NOT vote/comment until then
[Everyone] Peer-evaluation (upvote ones you like, comment on them) from 8:05 pm 31st Jan until 12 pm 1st Feb 2016
Post submission phase, you are welcome to browse through, upvote, and comment on others' ideas. We encourage you especially to look at and comment on ideas that haven't yet gotten feedback, to make sure everybody's ideas gets feedback.
COMMENT BEST-PRACTICES: Everybody in the team reviews at least 3 ideas, supported by a comment. The comment has to justify your reason for upvote. The comment should be constructive, and should mention positive aspect of the idea worth sharing. Negative comments are discouraged, rather make your comment in the form of a suggestion - such as, if you disliked an idea, try to suggest improvements (do not criticize an idea, no idea is bad, every idea has a scope of improvement).