Milestone 1 pumas

From crowdresearch
Jump to: navigation, search

Team Pumas Milestone 1 submission.

Experience the life of a Worker on Mechanical Turk

Overall, being a turker had its immediate benefits which we enjoyed greatly. For instance, gaining money easily and whenever we wanted. However, as time advanced we witnessed injustices that were hard to not acknowledge. None the less, we personally found these injustices to be beneficial triggers which forced us to think how we could help turkers to work together to oppose the injustices. We provide more detail in the following:


(1) The flexibility of work hours. Most of the tasks we conducted were at midnight before bed. Having the opportunity to gain a living from the comfort of your bed while in pijamas is very motivating!

(2) Gaining money! Who does not like to make money from simple clicks! That was the best part of all! $$$ :)


(1) The petitions from some requesters appeared to be: (a) underpaid; (b) shady.

It was a bit frustrating to not have a direct channel where one could work collectively with other turkers to ban ASAP suspicious requesters. It felt e.g., morally wrong to allow others to continue doing an underpaid job. Here we feel there is a need for mechanisms that allow turkers to unite and rapidly conduct together collective action.

There were some tasks for which we would have liked to have more feedback on what our work would be used for. It was unclear whether our input would be used for an illegal activity. For instance, one task was about writing down the emails that appeared in a photo. Here it seemed the requester was working in an illegal spamming activity. More information about the work we are asked to do could be beneficial. It was again morally frustrating to not have ways of rapidly challenging a task collectively.

It was also frustrating to see the apparent lack of rules in terms of tasks and payments.

We created a survey where we asked turkers about their experiences on Facebook and requested from them sample Facebook posts. Our csv file can be found here:

Experience the life of a Requester on Mechanical Turk


(1) Having easily instance access to a pool of people who we can study.

(2) Paying extremely low prices to obtain input from participants.


(1) Having to create mechanisms to detect cheating.

(2) Not that simple to filter participants based on their expertise or traits.

Explore alternative crowd-labor markets

Compare and contrast the crowd-labor market you just explored (TaskRabbit/oDesk/GalaxyZoo) to Mechanical Turk.

TaskRabbit: This platform in contrast to Mechanical Turk facilitates requesting offline tasks. For instance, having a person report about a local event in their neighborhood.

oDesk: This platform in contrast to Mechanical Turk facilitates finding experts in particular areas who can conduct a particular task.

GalaxyZoo: This platform in contrast to Mechanical Turk motivates people to contribute work (knowledge) not due to a monetary incentive but rather due to a community based one (helping science.) The platform also offers a space where participants can actively discuss their contributions. The participants of GalaxyZoo appear to be more knowledgeable in the tasks they conduct than the ones contributing in Amazon Mechanical Turk.


Flash Teams

  • What do you like about the system / what are its strengths?

We like the fact that the system views the crowd as an elastic, on-demand set of diverse and high-quality participants.

We also admire that the system is moving toward more expert work. The system can efficiently pipeline large tasks by assembling crowds of experts.

Some of the strengths are that: -it's designed so that crowd experts can complete complex and interdependent tasks quickly and reliably.

-It facilitates dynamic recruitment.

-It overcomes typical problems of teamwork such as last minute dropouts, or having team members with timezone differences.

  • What do you think can be improved about the system?

-The system does not facilitate the building of camaraderie. This can be very important for building a strong working team in the long run. The system could add mechanisms through which team members build friendship and solidarity.

-The system should be evaluated for other tasks.

-enable automatic mechanisms to help resolve disputes.


  • What do you like about the system / what are its strengths?

The work took the time to design its platform so that tasks could be conducted on small mobile devices, and in groups to control the accuracy and efficiency.

  • What do you think can be improved about the system?

A user study with more participants and run more in the "wild" The user study also needs improvement. In particular, evaluation of more tasks could provide a better picture of where the system fails and where it has its strengths.


  • What do you like about the system / what are its strengths?

We like that the system devised ways in which people in developing countries with simple cell phones (i..e, non smart phones) could become crowd workers and easily make a living. The paper also studied well the dynamics of India to devise how the payments to workers would work.

  • What do you think can be improved about the system?

Designing ways in which the system can motivate contributions from volunteers. For instance, perhaps via gamification.

Overall there is the moral debate of whether it is right to crowdsource work at low cost in developing regions. We believe here it might be worth to integrate mechanisms through which crowd workers in the region could visualize and compare the payments they receive with those obtained in other regions for the same task. This could help the workers unite and fight for fair pricing.

We believe the study could also be improved by conducting a pre-evaluation with participants to understand whether they are extrinsically or intrinsically motivated. This could help understand their contributions and contextualize results.