WinterMilestone 2 spontaneous optionalreadings
- 1 Recommended (Optional) Readings
- 1.1 Experiences surveying the crowd: Reflections on methods, participation and reliability
- 1.2 Exploring the Use of Crowdsourcing to Support Empirical Studies in Software Engineering
- 1.3 We Are Dynamo: Overcoming Stalling and Friction in Collective Action for Crowd Workers
- 1.4 TurKit: Human Computation Algorithms on Mechanical Turk
Recommended (Optional) Readings
Experiences surveying the crowd: Reflections on methods, participation and reliability
- The capacity of completing long tasks increase as long as the requester is able to keep them engaged and their efforts are respected.
- Education is an orthogonal factor.
- Workers are more attracted to survey taking tasks.
- Turking cannot be done in isolation. Workers may talk to one another in external forums.
- Sophisticated Turkers are aware that surveys that are completed too quickly stand a good chance of being rejected. Hence, they advise each other to hold back on survey submission.
- Turkers are looking for surveys on topics they care about, both because it’s easier to complete these surveys and because they want their opinions to be heard.
- Catching fraudsters is important.
- Longer exposure to survey design offers additional opportunities to attract fraudsters.
- Chief concern is maintaining goodwill and promoting the Turkers’ engagement via entertaining (but realistic) situations and provocative open-ended questions.
Exploring the Use of Crowdsourcing to Support Empirical Studies in Software Engineering
- RECRUITING PARTICIPANTS: Recruiting the right type and number of participants is a common challenge. The conjecture that crowdsourcing could address that challenge by providing access to a large pool of candidate participants doesn’t hold true because the researcher has less control over the users participating in the study.
- RESPONSE QUALITY: People who “game” the system may impact the response quality. Since Amazon anonymizes participant identities, it is also difficult to control for certain aspects of the population, such as age, gender, and education or training level. A qualification test, which undoubtedly may result in limited participation can allow some control over the users the requester is hiring.
- SUPPORT FOR STUDY MANAGEMENT: By providing a framework on which the study can be conducted, things can be more easily managed by a researcher. Even if the researcher has access to programmers to assist in the implementation of the study, some understanding of the framework's capabilities and constraints is required to determine how to best adapt the study to it.
- ADJUSTMENTS IN DESIGN AND OPERATION: If a user study contains multiple tasks, it cannot be guaranteed that each user will perform each task.
Mechanical Turk can be used as a front end tool to recruit users and manage payment. The actual study shoudl be implemented at a thirdparty site which may include a pointer to that site within a Mechanical Turk HIT. Once the worker finishes the activity at the site, they could collect a token and provide it to Mechanical Turk to complete the HIT.
We Are Dynamo: Overcoming Stalling and Friction in Collective Action for Crowd Workers
- The design of Dynamo is based on three affordances:
- trust and privacy,
- assembling a public, and
- Dynamo's success summary:
- Using the Dynamo Wiki, the members collaborated to come up with a comprehensive online guide that covered matters such as fair pay, how to respect Turker privacy, and how to respect Turker communities online.
- The letter writing campaign to Jeff Bezos- "This is a writing campaign for Turkers to let Jeff Bezos, head of Amazon and brainchild behind mTurk, and the rest of the world know all about who we are. The intent is to get Bezos to see that Turkers are not only actual human beings but people who deserve respect, fair treatment, and open communication.”
- Dynamo's failures:
- Stalling: Whenever volunteers perceive that an effort was losing momentum, their activities tend to decline i.e. participation is lowered.
- Friction: Some volunteers who didn't agree with the direction of the effort threatened to remove their support if the changes that they propose are not addressed.
- Stalling and Friction were experienced to be a linked occurrence.
- The Labor of Action: Overcoming Stalling and Friction
- Debates with deadlines: Deadlines restricted the arguing volunteers to come up with a consensus in a stipulated amount of time. This helped in preventing delays in the decision-making process.
- Act and undo: Early drafts by the team pushed forward momentum in collective action.
- Produce Hope: Portrayal of a successful image helped in attracting more workers towards an issue. The workers could then trust that their efforts in the movement will not get wasted.
- Reflect and Propose: Assistance in simplifying the arguments helped in productive discussions.
Dynamo provided a perfect platform to workers for collective action. Its two major issues of Stalling and Friction were also found to be addressed successfully.
TurKit: Human Computation Algorithms on Mechanical Turk
- TurKit Script: It provides a thin wrapper around the basic features of creating and removing HITs. It also provides crucial higher-level calls not part of the MTurk API.
- Crash-and-Rerun Programming Model:
- Incremental programming
- Easy to implement
- Retroactive print-line-debugging- It is possible to add/edit the code to program parts that have already executed.
- Users remained unclear about which aspects of TurKit script are stored in the execution trace and which could be modified or re-ordered.
- It favors usability over efficiency.
- TurKit Online: It's interface facilitates for managing projects, editing files, viewing output, and managing the execution trace.