Milestone 10 Pumas

From crowdresearch
Jump to: navigation, search

Template for Milestone 10

Summary of what idea this prototype is attempting to test

We wanted to understand whether Workers+Requestors would like mechanisms that allowed them to share proposals and select the voting method they wanted associated with their proposal.
Our thought process was that this type of design would allow workers+requestors to decide when they wanted the whole community to be involved in the voting process of a proposal; or when they wanted only a select committee to decide etc.
We considered that different proposals could impact the platform differently, and it was thus important to weight the actors differently in the decision making.
We conducted interviews with 2 requestors from Amazon Mechanical Turk; and an Odesk worker. We presented the idea to them and had brief discussion.

Updated Prototype

Votism2.jpg
Votos.jpg

We updated the design of our prototype based on a good discussion we had among the team.Below we include some of the discussion:
I like to start with a top level approach to things, so my immediate question is what is the goal you are trying to achieve. And I believe the goal is to expand workers' involvement past simple task completions. You basically want to give them a voice. This way the requester has a better understanding of the needs of the workers and can receive feedback that would allow the requester to create better tasks. Happy workers = good workers. So your goal with Votism is to give workers a place to suggest changes for the requester. Workers submit a proposal, and a requester will then take the most popular proposals and decide to implement the changes. I think this is a good mechanism that should be part of any crowdsourcing project. As a requester, I have often had to iterate the design of the task until I reached a point were, for the worker, the task was easy to understand and easy to accomplish quickly and accurately. This feedback loop your proposing would be awesome. I would suggest changing to up/down votes instead of stars. I'm not sure, but I'm thinking up/down votes is a better way to rank ideas. I have some Luis-science hypothesis in my head which may or may not be true. Where I work we have a suggestion forum, ideas are voted up or down. I like how it works. Star system makes me think of foursquare or netflix.

Possible additions to Votism:

  • Ideas should have a space for discussion. Comment thread with ability to vote on comments.
  • For idea implementation, I'm thinking of an "Agile-style" approach. Ideas which are voted the highest are reviewed by the requester(s) and he/she will determine how important and how easy it is to implement the new idea. Based on those two criteria, every two weeks or so the requester will choose a few of those ideas and implement them to the satisfaction of the workers. And so on until the whole project is completed.
  • Perhaps a rating system that shows how well a requester responds to worker's ideas (demands?).

Several issues I can think of:

  • Changing the task mid-way might affect the results depending on what you're trying to do. If a requester is conducting a research experiment, you probably can't change the task halfway because that might skew your results. You'd have to complete a cycle, and implement changes in the next iteration. Perhaps the feedback received from the workers will allow the requester to get better results for her research.
  • Two-week sprints for implementing changes assumes the whole project will last much more than two weeks. Perhaps a sprint could be every couple of days. Or perhaps you just prioritize which idea from the workers you want to implement, you do it, and then you move on to the next.
  • There is also some assumption that the worker will repeatedly work on the requesters tasks. Sometimes workers just work on tasks for a couple of days and go on to the next project. I think this happens with Mechanical Turk. The more repeated interaction there is between a worker and a requester, the greater the interest and the benefit in having two-way communication and feedback. Perhaps something that a platform could emphasize is the ability to repeatedly work with the same requester. Maybe even have reputation scores on requesters ("This requester is cool to work with 5/5 would work again").

One of the main features of open governance is transparency: If you can have clear, transparent, and easy to use channels of communication between the workers and the requesters, you will have more transparency, which in theory should lead to a better platform.

Those are my thoughts for now. I will keep trying to be involved as much as I can. I will at least be a little more present in slack, and you can always feel free to respond here, or reach out via email.

Individual User Study Details

We interviewed 2 requestors (computer science graduate students who used Amazon Mechanical Turk for their studies (ages 26,23); and a freelance editor from Odesk (age 23).) The overall findings from the interviews are:

  • Most are not familiar with the different voting methods.
  • Difficult for users to understand impact of choosing a particular voting method.
  • People were worried if participating in the voting could be a time sink for them. They discussed they would need to review proposals , think about them, and that could take their time

Improvements to Idea based on Prototype Testing

We need to think how we could make voting methods intuitive for people so they can easily incorporate them into their proposals.