Milestone 3 Team Innovation 2
Team Innovation 2 is comprised of experienced MTurk workers, which is the perspective from which the below material is written.
- Provide an open forum where workers and requesters can interact with each other seamlessly. This will allow discussion of new features and of issues not already addressed by the existing design.
- Rate workers using multiple aspects of their performance and history. Part of this rating is influenced by requesters, and part of this rating is calculated automatically. For instance, how long has a worker been on the site? How many days have they been active? How many work hours have they logged? What is their performance rating in different categories of work? How much work have they done in what categories? Display this information on workers' profiles so requesters can see it, and allow requesters to invite workers to their tasks based on rating criteria. [This provides requesters with more information they can use to find workers that are trustworthy.]
- Require a base level of payment for certain tasks. This base level of payment could even be automatically calculated by the site based on information within the system, like workers' pay-related rankings on tasks in a given work category. [This allows workers to trust that their payment will always be above a certain level. They also have the ability to influence their base level of pay, so this is also related to power.]
- Allow workers to publicly recommend or highlight good requesters, and allow requesters to do the same for workers. [This will help increase the trust between the two groups.]
- Provide a training program or tutorials to take requesters through the process of designing a good task, one that will get positive responses and useful results. [This increases trust on both sides: Workers feel more confident that they understand the tasks they work on, and requesters feel more confident that they will get good results for their money.]
- Make it easy for workers and requesters to collaborate repeatedly with those they find it useful to work with. Allow requesters to create groups or lists of workers they trust, so they can draw from that pool of workers and invite them directly to perform work. Also allow workers and requesters to favorite workers/requesters that they like, so they can either notify those requesters when they have new work, or get notifications (automatic or otherwise) when those requesters post new work.
- Allow workers and requesters to (optionally) increase their ratings by posting links to their profiles on sites where they already have feedback (like Guru.com or oDesk), or by providing sample work or tasks, or by presenting a list of their personal policies, standards, and qualifications. Workers and requesters can review these profiles and vote on whether they invite confidence and trust (like an upvote on Reddit).
- Allow workers to challenge work they feel was rejected unfairly. Process such challenges fairly and quickly, to allow for efficient use of everyone's time, and give requesters some kind of incentive (like a rating?) for spending the time to handle these issues in a satisfactory manner.
- Provide a robust, complex, user-friendly system for searching for work. Make the site browsable by job category, skills involved in a given task, pay scale, worker rating, number of tasks available, time estimated for completion, etc. [This gives workers more power to choose good work, and work that suits their skills.]
- Allow workers to give direct commentary and feedback (including numerical rankings, if they choose) on the quality of a task. Feedback includes such things as how well the task pays, how long it took for them to complete the task, whether the directions are well-written, etc. This feedback will be visible, in a requester's work history, to other workers and requesters on the site. The ratings of a requester's tasks could also affect how the site interacts with the requester, for instance, allowing requesters who have better overall task feedback to post more tasks than requesters with low feedback. Possibly only tasks that get above a certain rating are allowed to be reposted? [This gives workers the power to influence the quality of work on the site. It also helps educate requesters on how workers view their tasks, which could increase trust between workers and requesters, so it's also related to trust.]
- Allow only work that meets quality control standards to be posted. For instance, all work must have a clear description, good tags, be properly categorized, and tagged intelligently. Should there be a review board that approves all work? Should the system itself simply require that certain information must be included before it's posted? Maybe each item should be approved by volunteer workers before it goes live on the site. [This restricts the power to post work to those who can post quality work, but is also related to trust, because workers will be more confident that work will be worth the time it takes to do it. It also seems likely to get requesters better results, so they will trust their results more.]
- Make delivery and review of work transparent to both workers and requesters. When completed work is submitted, have it remain visible to the worker. Allow both the requester and the worker to suggest edits to submitted work. Allow requesters to rate work independently of rejecting or approving it. [This gives the worker more power over their own work and allows the requester the final approval, but also gives both the worker and the requester more power to discuss changes in and quality of work.]
- Allow workers and requesters to flag items of concern, for instance, spam worker accounts, work that breaks guidelines, and have an independent, rotating board of both workers and requesters to review these items, make decisions, and provide resolution. [This checks the power of both workers and requesters to behave maliciously, and also empowers both groups to feel their concerns will be addressed quickly and hopefully without bias.]
Dive Deeper into Specific Ideas
- Milestone 3 Team Innovation 2 TrustIdea 1: Robust User Rating and Profile System - http://crowdresearch.meteor.com/posts/TaNbEkmeoSSiv5vDB
- Milestone 3 Team Innovation 2 TrustIdea 2: Worker and Requester Forum - http://crowdresearch.meteor.com/posts/ShEsGS3tx2Gdq9noJ
- Milestone 3 Team Innovation 2 PowerIdea1: Worker Feedback on Tasks - http://crowdresearch.meteor.com/posts/HEiG2GyJWLFPzThYc
- Milestone 3 Team Innovation 2 PowerIdea 2: Efficient Search System - http://crowdresearch.meteor.com/posts/qsv2bcRNjSL3PC8sC
Dark Horse idea
Our team came up with two dark horse ideas that we liked. One is described in detail at the link below.
- Milestone 3 Team Innovation 2 DarkHorseIdea: Bonus Pool Payment System - http://crowdresearch.meteor.com/posts/3xiwnx7ir35hgW3PS
The second dark horse idea, not expanded in detail, is for a sort of "curated work experience" — where only requesters who receive the approval of a board of selected, high-quality workers are permitted to post work on the site. Maybe there could be something like the current MTurk Masters certification, but for requesters? Something that gives requesters special privileges, like special access to a pool of the best workers?