Milestone 6 Pumas Infra

From crowdresearch
Revision as of 21:30, 8 April 2015 by Saiphsavage (Talk | contribs)

Jump to: navigation, search

Template for Milestone 6 Code Infrastructure. If your team did Milestone 6 Research Proposal, see Milestone 6 Research Proposal Template



If this is a collaboration between multiple teams, list them here.


PlatformCode.jpg Please document your overall system architecture. Please explain how your framework is modular and makes it easy to extend the platform and add new features.

System Overview

EmpathySourcing:Triggering Values in Crowd Work

Crowdsourcing platforms have become increasingly popular and used widespread. However, there is currently a disconnect between workers and requestors. This has lead to unsatisfactory and mediocre work; as well as lawsuits and conflicts between workers and requestors. We introduce EmpathySourcing, a platform that via smart mechanisms triggers empathy between requestors and workers. Our design takes advantage of related sociology theories to foster community and values, and as a result produce higher quality work. We report a multi-month field deployment of EmpathySourcing where 82 requestors made 11,240 tasks, and 700 workers finished those tasks. Our results suggest that empathy mechanisms can indeed improve crowd work.

Motivation & Insight

Tasks and systems that leverage the crowd have become commonplace. Crowdsourcing platforms have become increasingly popular [1]. This has lead to the development of several different types of platforms that use the crowd to collect and/or analyze massive amounts of data. However, Crowdsourcing platforms are currently in a crisis [1,2,3].

Workers frequently have doubts about being adequately compensated for their work. Many also suffer from what psychologists have deemed "inferiority complexes" [7]. Workers feel they are not respected. This is especially accentuated when requestors constantly reject their work without any explanation. Many workers even have doubts on whether they are performing illegal crime-related tasks for requestors [2].

This apparent lack of trust and socialization between workers and requestors is also affecting requestors. Requesters are hesitant concerning the validity and accuracy of the results. They have to ask multiple workers to perform one same task over and over again to eliminate spam and ensure quality input. Requestors also live in constant fear. Many times they do not speak up about "bad work" because they are afraid of obtaining a bad review and affecting all of their future requests [3].

There have been several investigations that have tried to improve crowdsourcing platforms, and the dynamics between workers and requestors. Salehi et al. proposed Dynamo a platform that allows crowd workers to organize themselves and produce collective action. Nevertheless, these attempts to design better crowdsourcing platforms are at present unsatisfactory. Previous research has not addressed in their designs the well-established sociology topic of social support and empathy [6]. Sociology theory has established that to produce quality work a community between workers and their managers needs to be established. Values need to be implanted in the system.

This paper explores the idea of designing mechanisms that facilitate the creation of empathy between workers and requestors. We present EmpathySourcing, a platform designed to trigger empathy between crowd workers and requestors, and as a result produce higher quality work.


Image Caption: Examples of the smart mechanisms that EmpathySourcing creates to foster community and values between workers and requestors.

Related Work

Trust Problems between workers and requestors.

Workers pose various needs. At times tasks may be lacking proper tagging and description, which translates into unpaid time the workers have to spend time finding and identifying the tasks they would like to work on. They need a quicker way of searching. The vast majority of crowd workers are motivated by economic incentive, which are inaccurate when it comes to HITs, as they do not adapt to market's supply and demand changing conditions. A second need derives from this, i.e. workers requiere a trustworthy compensation for their work. Turkopticon identified a common complaint from workers: their work gets rejected without any explanation. Workers need to be treated equally and with respect so that they are more encouraged to complete good quality tasks. A platform or channel for expressing their concerns and be able to communicate with other workers and even their requesters would be a yearned environment. Subsequently, new workers need to be treated equally and trusted, they need feedback from requesters to get more HITs and gain reputation. This calls for an accurate exposure of their skills and qualifications for requesters to seek. Finally, workers themselves avoid dubious tasks, as to avoid making mistakes and getting rejected, so they need to understand the goal of the task and without delay.

On the other side of the Tuker market lay the requesters, which also present several needs. Their HITs need to be solved accurately and fast, but when this does not happen they are unaware of the reasons behind. A communication channel with the workers could enable them to negotiate so that both parts get their need fulfilled. Requesters are hesitant concerning the validity and accuracy of the results. If they are uncertain about the correctness of the results then they have to discard them. A confidence system where the requester gets to know the workers would be of great in solving this need. Another common problem seen on AMT, MTurk or similar platforms is that requesters find it too difficult to post and process their HITs, they sometimes even outsource them. They requiere a far simpler and direct platform. Requesters are also ignorant about the appropriate price for their HITs. They usually ask around forums, but there is no significant information to help them and they can't always wage in the worker's experience with the complexity of the task they need them to perform. Finally, requesters can also get bad reviews thereby workers become hesitant to take their HITs i.e. trust is a common need between workers and requesters.

What is the problem that you are solving, and why is it important?

After analyzing both sides of the workforce we come down to two main issues: power and trust. Power to post task, to edit already posted tasks and to return the results already obtained. A constant battle between workers and requesters aggrandizes the problems and disrupts the basic concept of trust, and the consequences involved. It is said that trust must be gained, but how is this possible in a platform where all interaction is computer-based and not human-based? Requesters need to be able to trusts the quality of the results from their HITs, as well as workers need to be trusted to do the work they were hired to do and be paid appropriately. When both parts feel trusted more synergy and motivation exists, therefore producing better quality work.

Explain problems of motivation.

Under this scheme two types of motivation need to exist: the one given by workers and the one given given by requesters. There are different ways a requesters can motivate workers, which in turn results in happier workers producing better quality work. Based on sociological research the proposed platform ca recommend requesters how to motivate workers. To begin with, the platform can recommend economical incentives or personalized messages to workers that excel in their performance, thereby stimulating confidence. The platform can also incorporate various mechanisms to conduct opportunities for workers to life advancements through their career uplift and type of tasks performed. Another motivational tool is using direct feedback, that way workers can know what is right or wrong with the tasks they are carrying out, and improve upon those comments. Therefore, feedback becomes motivating, inspirational and a whole support system for better quality work. Finally, the most intuitive motivational tool: giving thanks to workers for the tasks they complete.

What are the existing attempts to solve this problem that have been attempted in prior research papers and real-world systems? Why are their solutions unsatisfactory?

Collective action aim at change generation, it being by profit, progress or other means. The downside is that many of these effort are unsuccessful. Amazon Mechanical Turk (AMT) is a crowd work platform where businesses go to seek scalable workforce, while workers get to choose between thousands of tasks and offers whenever they find it favorable. The advantages come from both sides as the requesters have a 24x7 workforce and pay until pleased with the final outcome, while workers get to chose their hours, perform the tasks from home and earn money. Nonetheless, some have argued [3] that this collective work platform is questionable in terms of ethical work relations, values and relies too much on the assumption that all workers produce work with the same quality as computers do. Aside from AMT other platforms exist, such as TurkerNation or MTurkGrind, but present an additional layer of communication between requesters and workers [8].

Ross, Irani and et al. [9] present a study on crowdworker's demographics. The study done on MTurk, that characterizes the turkers involved i.e. their gender, age, income, education and impact of turking to their situation. The common outcome - also portrayed by Martin, Hanrahan and et al. [5]- is that turkers are unsatisfied with requesters rejecting their work with no explanation, they are underpaid and turking has no impact on their career and life advancement.

Silberman, Ross and et al. [10] present the problems sellers have in this new era of human computation. Followed by Silberman, Irani and et al. [11] who present the issues, with work ethics in crowdwork, giving us a scenario where values should be imposed in turker environments. Irani and Silberman [12] argue about the "work invisibility" present in turker platforms such as AMT, since workers can be underpaid, work extremely hard and still have their tasks rejected. AMT's essence is based on computer interaction with as little human interaction as possible. Irani and Silberman introduce their option "Turkoticon" where workers can expose, criticize and evaluate their relationships with their employers. This platform is the most similar product to TurkerSociety, as it enables collaboration and communication between turkers. The downside to their platform is that the communication is stablished only on one end and their platform look more like a complaints site than a socialization and community fostering platform. TurkerSociety does go further and looks at two-way relationships: workers and requesters. A duplex communication enable a more understanding community with higher work quality results.


PipelineSystemEmpathy.jpg Image Caption: Overview of EmpathySourcing.

EmpathySourcing will create a community between workers and requesters with the main purpose of facilitating the communication between them and fostering labor values and socialization mechanisms. This platform will identify workers with low ratings (i.e. lacking something that does not enable them to produce good work) and people (e.g. more experienced workers or requesters) who can help them. The worker will then be introduced to or connected with the helper. In this manner communication is established, community values are promoted and improved quality work is procured. This platform is based on (a) having everybody knowing each other (communication) and (b) abide by values. In this format two types of motivation will exist as well: (a) the motivation given by other workers and (b) the motivation given by requesters.

There are five values this community aims to foster:

1) Confidence. The worker (or requester) must feel that he/she is an important part of the team (yet not essential) for the success of the community. In this sense, feeling part of a community and a support system will be a key factor.

2) Fraternity. The relationships between workers must always be direct but respectful. A failure is not individually-based but collective, and everyone should help each other to overcome it.

3) Equity. There needs to be Fair treatment between the members of the team.

4) Mutual Support. This is about moving human resources to the workers who might have a weakness so that they can better perform their task.

5) Inclusion. Everyone should feel they are part of a large collective effort, regardless of their gender, ideology, beliefs, capacities. Everyone is responsible for the community's advancement.

All values are as important and not affected that they order in which they are explained.

EmpathySourcing will also incorporate the mechanisms, previously explained. The mechanisms are color-labelled according to the values that they intend to strengthen. At the same time, these values come from the former research on worker's and requester's problemas and the motivation issues and solutions that derive from them. Motivation can be encouraged through this platform, subsequently this platform is able to provide that motivation by establishing the five values in the last figure and by implementing the proposed mechanisms through data visualization and crowd sourcing. EmpathySourcing aims at empathy between workers and requesters to improve upon their communication, their work quality and ultimately their life advancement.

Community Values

Image Caption: Sample values that the platforms via smart mechanisms seeks to help foster.


To have a comprehension of the type of work that our system improves, we launched our tool to the public and studied its usage from 6 months. Our aim was to have a large number of natural usages to test the robustness and weaknesses of creating empathy mechanisms. We conducted interviews and surveys at the end to understand subjective perceptions of our system versus other platforms. We also measured:
The quality of the work produced (we asked external people unrelated to the research to evaluate the quality)
Measure time to complete the task.

Requestor using EmpathySourcing Platform


[1] Crowdsourcing user studies with Mechanical Turk Aniket Kittur , Ed H. Chi , Bongwon Suh, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, April 05-10, 2008, Florence, Italy. [1] Review of advantages and limitations of AMT.

[2] "Who are the Crowdworkers?: Shifting Demographics in Amazon Mechanical Turk". Ross, J., Irani, I., Silberman, M. Six, Zaldivar, A., and Tomlinson, B. (2010). In: CHI EA 2010. (2863-2872).

[3] M. Six Silberman , Joel Ross , Lilly Irani , Bill Tomlinson, Proceedings of the ACM SIGKDD Workshop on Human Computation, July 25-25, 2010, Washington DC.

[4] Ethics and tactics of professional crowdwork, XRDS: Crossroads M. Six Silberman , Lilly Irani , Joel Ross. The ACM Magazine for Students, v.17 n.2, Winter 2010.

[5] Martin D, Hanrahan B V, O'Neill J, et al. Being a turker. Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 2014: 224-235.

[6] Corsun, David L., and Cathy A. Enz. "Predicting psychological empowerment among service workers: The effect of support-based relationships." Human relations 52.2 (1999): 205-224.

[7] Riezler, Kurt. "Comment on the social psychology of shame." American Journal of Sociology (1943): 457-465.

[8] Aniket Kittur , Ed H. Chi , Bongwon Suh, Crowdsourcing user studies with Mechanical Turk, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, April 05-10, 2008, Florence, Italy. [1]

[9] Ross, J., Irani, I., Silberman, M. Six, Zaldivar, A., and Tomlinson, B. (2010). "Who are the Crowdworkers?: Shifting Demographics in Amazon Mechanical Turk". In: CHI EA 2010. (2863-2872). [2]

[10] M. Six Silberman , Joel Ross , Lilly Irani , Bill Tomlinson, Sellers' problems in human computation markets, Proceedings of the ACM SIGKDD Workshop on Human Computation, July 25-25, 2010, Washington DC.[3]

[11] M. Six Silberman , Lilly Irani , Joel Ross, Ethics and tactics of professional crowdwork, XRDS: Crossroads, The ACM Magazine for Students, v.17 n.2, Winter 2010. [4]

[12] Lilly C. Irani , M. Six Silberman, Turkopticon: interrupting worker invisibility in amazon mechanical turk, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, April 27-May 02, 2013, Paris, France.

[13] Social Turkers

[14] TurkNation

[15] Dynamo