Milestone 3 Singularity
The ideas you brainstormed, (at least 10 ideas for trust, and at least 10 ideas for power). Provide them in whatever format you want - diagrams, sketches, descriptions, or a combination (the wiki supports images, see here for instructions on uploading them).
- Have a platform that acts as a more social medium and consists of requester profiles for workers to view. These requester profiles contain all the relevant data that a worker might need to assess the trustworthiness of a particular requester. This might include information such as the type of jobs the requester usually puts up, the fairness rating of the requester (as voted upon by workers who've done their work), history of rejections, average payment given out and a lot more. Such a platform could also let requesters build a list/group of trusted workers who he or she can rely on. This platform has several other features which we've elaborated upon as our main idea. It could let workers be a part of groups and cliques of people that have similar interests or skills which could let them share opportunities and organize themselves and do a lot more.
- Creating an issue tracker by which a requester can resend a work for review if he is unsatisfied with the quality of work. The worker will have a deadline to get done with the issue or face rejection for the task.
- Creating different levels of workers , a worker's level increases with experience and doing a specific amount of work for the community.This is another major idea that we think could be incredibly useful. A worker could specify the skills he possesses and level up in those skills by doing tasks. On each level up in a particular skill, the worker gets more challenging and higher paying tasks. This addresses two major needs. The need for workers to find jobs of their own skill level and The need for requesters to allocate tasks to trusted/reliable skilled workers.
- Requesters need to TRUST the work they get. We came up with a system that can accomplish that to a reasonable extent. For brevity, I'll describe it in short over here. We ensure a higher percentage of verifiable “checking” questions. For a survey that requires 90 objective questions that need to be answered, we sprinkle in 10 survey questions to which the requester/website/moderator/bot know the answers. The correctness of these check questions could give a good estimate of the the correctness of the entire submission. The best part is that this "verification" system could be automated. A bot could be assigned the task of adding random survey questions to a task OR checking the answers to the "check" questions of a task. This bot could then easily assemble the results and send them to the requester who could have a look at the dubious submissions and "bounce" them back to workers asking for improvements.
- Requesters need to TRUST the work they get. In the above system, "verification" questions could be nicely complemented with the the idea of redundantly allocating tasks to a higher number of workers. This way it would be easier to point out what worker's answers is clearly different from the majority which would indicate suspicious (random) or poor performance. This is another measure for an entity to get an estimate of the quality of work which could be used to let worker improve or reject radically different hits that don't conform to the majority AND have performed poorly in the verification questions/
- The idea of letting workers downvote or upvote a requester after completing his/her work based on the fairness of the requester. This would let the workers influence the requesters' credibility/fairness ratings. It would lead the requester to be more wary of rejecting hits as they might fear getting downvoted and not being able to find workers in the future who would be able to see the low approval rating they received.
- Requester statistics would definitely go a long way in establishing trust between requesters and workers. A worker who can easily check and scrutinize a requester's statistics pertaining to their rejection rates, payment rates, acceptance rates etc would find it immensely easy to assess whether he or she can trust the requester or not.
- Standardization of tasks and job designs using pre-defined standard templates, formats and interfaces designed and refined by the community for the workers AND the requesters.
- Workers getting to understand tasks conveniently. They would be familiar with the templates, interfaces and UI designs from beforehand therefore wouldn't face a lot of problems when viewing a task that has been presented to them in a familiar format.
- Requesters can expect a higher quality of work in lesser time from workers considering that they'll understand tasks easily. Standardization will highly increase productivity and efficiency due to cutting down on time spent understanding tasks and lesser number of errors made due to a higher probability of them having understood it correctly. It would be incredibly convenient too because requesters can follow a predefined template when DESIGNING a task. Requesters have often noted that they'd like to spend less time making the task which can be solved using this idea as they would just have to follow a few steps when incorporating a task into a standard.
- Requesters need to trust results they get, but workers need to trust that their tasks won’t be rejected. There can be a small “test set”(known to the requesters and the mturk platform but unknown to the workers) which is a fraction of the task but the answers to which the requestor already knows. The platform would evaluate the performance of the workers on this test set and the task would automatically be accepted or rejected based on the performance in this set.(this only applies to tasks such as tagging which are easily decomposable).
- Better reliability of results can be gained if multiple workers redundantly solve the same tagging task. Then results of different workers can be composed to arrive at the ‘correct’ answer. Workers whose answers diverge too much from those of others will be rejected.
- Combining the above two ideas, the entirety of a worker’s task need not be replicated to another. If say 10% of each worker’s task is being replicated, then their performance can be evaluated on the basis of how much this 10% diverges from others. This keeps overheads low.
- People who give valuable feedback are rewarded and their feedback has higher importance and visibility.
- The worker has the option of making his earnings public. He can keep them private if he wants to, but putting his earnings on display is one way to actually attract requesters who can trust such a worker. The money earned is calculated by the system, and there is no way to forge these statistics (obviously).
- The worker has the ability to stack up streaks for every day of his contribution (similar to github streaks).
- In this way, there are multiple paths that a worker can explore in order to rise to the top and build trustworthiness. Those ways include - getting stars from requesters, getting stars from the general public, publicly displaying the number of contributions in a visually aesthetic way, stacking up streaks and displaying earnings on the main profile page.
- Creation of mods to intermediate between requester and workers. They can verify if the work has been rejected with just cause or not.We could use a system similar to that of mods on reddit ( these could be the turkers with a vast experience and an excellent hit rate who have been quite active on turkopticon) the mods could act as an intermediary between the requesters and the turkers.
- Creating of a better UI so that the worker can navigate through the site easily and does tasks that he/she is convenitent with.
- Integrating a forumn inside platform , so that workers can clarify their queries.
- Crucial information about tasks such as rejection rate and average completion time have a direct effect on expected earnings. Requesters exercise their power in choosing not to disclose this info. Making this info available for tasks gives workers some decision making power.
- Although turkopticon exists, there is no intrinsic rating system for requesters on the platform itself. This hurts newbie workers who have no idea about such fora/addons/tools. Including such a feature on mturk would help newbies avoid making mistakes.
- A turker should be able to put his job failure under review and the mods should verify his/her claim within a certain amount of time.
- The system should constantly be always evolving. The mods/developer team needs to take into account the feedback of both parties to ensure that we don’t become stagnant.
- We could design a system where rejections are themselves rejected or approved by a netural third party which could comprise of moderators or other workers.
- Reduce the amount of power requesters have when it comes to allocating money. To tackle scammers who don't pay, we implement some sort of ESCROW system. The mods have to pay earlier on to ensure that they can't disappear without paying later on. Once the money from a requester is up in a secure escrow then their task could be approved by mods or by a bot that can check that the requester has put up sufficient funds (or the claimed amount of funds in the task). The workers could start working on the job and get paid from that escrow. Now the allocation of payments may or may not be done by requesters. They could certainly approve payment but to withhold payment, they have to raise an issue/problem/complaint with the worker or the mods.
- Reduce the amount of power requesters have when it comes to rejecting work unfairly. Instead of rejecting work straightaway,make it mandatory for the requesters to "BOUNCE" unsatisfactory work back to the worker who can review it, improve it and then turn it back. This has three major advantages:
- A worker gets the chance to improve instead of getting rejected out of the blue in a sudden manner.
- Workers get to ensure that they have a decent shot at getting paid.
- Improved work is returned which will significantly increase the overall quality of the final work that the requesters get.
- The system comes with a built in issue tracker. In the issue tracker, the requester can review the work done by the worker, and open new issues depending on where he feels that the work done by the worker missed the mark. He can open multiple issues with the worker’s work and keep checking the off as the worker fixes them. This leads to fewer rejections, and less heartburn.
- The requester assigns the issues an “importance rating” from 1 to 10.
- If the work really needs to be rejected, the worker can be given one chance to withdraw his work without affecting his reputation. (Haven’t thought this feature through.)
- An issue opened by one requester can be worked upon by someone who wasn’t initially affiliated to the task. The payment of that worker depends on the issue rating given to the issue solved by that particular worker.
- A markdown supported summary creation system in order to promote brevity among users ala github
Dive Deeper into Specific Ideas
For each of the 4 ideas (2 for trust, 2 for power), describe (using diagrams, sketches, storyboards, text, or some combination) the ideas in further detail.
- Milestone 3 Singularity TrustIdea 1: Issue Tracker for tasks - http://crowdresearch.meteor.com/posts/oFsCt2Nig5BcC2iEb
- Milestone 3 Singularity TrustIdea2:Different Levels of workers - http://crowdresearch.meteor.com/posts/KkregTBo5M95t6Fik
- Milestone 3 Singularity PowerIdea 1: Moderators as arbiters of disputes - http://crowdresearch.meteor.com/posts/PJSvYp9MCezeQPnQ8
- Milestone 3 Singularity PowerIdea2:Requester Profiles - http://crowdresearch.meteor.com/posts/RSSEabntSiDJyeg3t
Dark Horse idea
Describe your dark horse idea (using diagrams, sketches, storyboards, text, or some combination).