Milestone 3 PixelPerfect

From crowdresearch
Revision as of 20:15, 18 March 2015 by Kushagramehta (Talk | contribs) (What powers do workers have to find HITs quickly?)

Jump to: navigation, search

Team Pixel Perfect's submission for Milestone 3.

Initial Brainstorm

Crux of idea goes here.

Trust-related Ideas

How do requesters trust workers and vice versa?


  1. Strict identification system : Each worker or requester is requested to submit valid ID and proofs at time of application, to verify their genuineness. This may be a combination of Social Security Number, Voting Card, Driving License, Ration Card, Aadhaar card (in India) etc. Worker details are by default, private and disclosed only if the worker wishes to. Each application is reviewed before users can join the platform.
  2. Worker-Requester Social Graph : We use the Facebook social graph model to conceptualize worker-requester relations and compatibility. We develop a crowdworker graph (bipartite for now i.e no relations within workers or requesters) whose nodes are requester and workers, and compatibility between them modeled by the edge weights.
Compatibility develops through a two-way feedback system between workers and requesters after completion of a task.
It is centralized around a simple Yes/No question :
Would you like to do a task from this requester again? [For workers]
Would you delegate a task to this worker again? [For requesters]
Compatibility plays a role in how visible a given requester’s HIT is, to various workers. Workers having higher compatibility with the requester are more likely to see the HIT and therefore take it up.

How do requesters trust good results on their tasks?


  1. Demographics check : The platform shall have a mechanism where only workers with appropriate demographic are eligible for the task. These might be helpful in certain HITs which are demographic specific (eg. Surveys involving only native American women). Such tasks can be filled anonymously, with the system performing the demographic check based on the identification provided. If the tasks require disclosure of details, workers can only take up such tasks if they willing to forgo anonymity. This preserves the choice to be anonymous.
  2. Task templates : Good task design can go a long way in reducing spam in submissions. These can be done by including questions which require correct answering before completing rest of the survey/task. Correct answering forces spammers to read and appreciate the content, spending more of their time, and hence discouraging bad replies. Therefore, pre-designed task templates for different task classes are available for requester use, thus standardizing task quality. Some templates may have scripts/tools to help with evaluation. This helps requesters develop HITs quickly and get more trustworthy results. Workers also benefit from consistent clarity of standardized tasks.

How do requesters trust they will get their results on time?


  1. HIT visibility algorithm : We model HITs analogous to posts on Facebook, liking or commenting a post is analogous to accepting a HIT by a worker. On Facebook, the posts visible to users are determined by the EdgeRank algorithm, which is a function of affinity, weight and time decay.
On the platform we develop, the HIT visibility is determined by :
  • Worker-requester compatibility determined from the Social Graph (Affinity)
  • Importance of task to worker determined from skillset, task preference and time/money payoff (Weight)
  • Tasks whose deadlines are approaching given more preference (Time Decay)
Finally a randomized algorithm scrambles and sorts these HITs and chooses which ones are visible to the worker.
Thus, requesters can have confidence in receiving their results on time.

How do requesters trust the skillset and expertise of workers?


  1. Skill levels : Workers are free to register for certain skills and fields of expertise, although a machine learning algorithm can automatically do the same, by analyzing the tasks undertaken. Each worker will have a radar graph associated with him or her, which shows at a glance, expertise levels in the registered skillsets. Skill levels grow with successful and accurate completion of tasks in that skill. Requesters can adjust lower and upper bounds on these skill levels for their tasks, to filter qualified people.

How do workers trust they are not being exploited?


  1. Report exploitation system : A common issue is work theft, where requesters do not acknowledge work submitted, however incorporate it in the results. This happens because requesters enjoy intellectual property rights over the work submitted and not the workers. A workaround might be to allow workers to submit “Exploitation Forms” to the requester with appropriate evidence attached. These might be analysed by moderators who may decrease the rating of the requester if found guilty. Workers may be penalized if found to submit false claims. This legal facility may be included as part of the terms and conditions of the platform.
  2. Penalty system : Workers would be compensated for the delay in the payment as quite a large percentage of people rely on these platforms for their daily living. The penalty amount would be a function of delay time and task amount.

How do workers trust they will be gainfully employed at all times?


  1. Rating system : Rating is assigned to each worker or requester and is a function of several factors. Rating is indicative of quality of work done/given and credibility.
  • Workers earn rating for accurate and fast completion of tasks, skill level and knowledge, time spent on platform, compatibility with requesters.
  • Requesters ratings are similar to that on Turkopticon – generosity, promptness, professionalism, fairness and communicability, task clarity and feedback history.
A common issue on current crowdsourcing platforms is, certain weeks generate good payoff for workers while other weeks don’t. Auto-generated tasks whose solutions are known can be made available at all times. During idle scenarios (low task demand), these can be taken up for work. This enables workers to improve their experience and ratings during downtime, which in turn makes them eligible for tougher and more intricate tasks later on.

How do new workers get started quickly and earn the trust of their counterparts?


  1. Tutorial Mode : Consider workers just starting out with the platform and having little experience. Lot of tasks might not be available to them at this point, since they are relatively unknown. Therefore, computer generated tasks whose solutions are known, will be visible to them – we can gauge their skillset, accuracy and efficiency. This acts as an initial seed for rating/experience.

Power-related Ideas

What powers do workers have to feel respected and voice their opinion?


  1. Two-Stage Feedback : There will be a two-stage feedback for a particular HIT and requester – one during completion of task and the other, after its completion.
    • First stage of feedback for workers involves them rating the clarity of the instructions, if the task was appropriately priced and whether optimum time was given for the task. Also a single question 'Would you like to work for this requester again?' signifying the overall experience is asked.
    • Second stage involves requesters being rated on the promptness of their payment and their professionalism in general (mass rejections, feedback given to workers)
    Moreover, workers can send "exploitation forms" as mentioned above in case they feel they have been wronged.
  2. Comment Feature : Each HIT will have a comment section, where workers can suggest improvement or provide tips on task completion, or even publicly declare their overall experience with that HIT and requester. Requesters may use this feature to positively critique task design and other aspects.

What powers do workers have to find HITs quickly?


  1. Requester and Worker Profiles : Requester and workers could have a profile page for purposes of communication and knowledge. This is similar to the concept of a wall on Facebook.
    • Requester profiles may carry information about rating, tier, level and experience points, typical allotted work, contact information (email/github/IRC) and perhaps a button to IM. An option to subscribe to this requester may be allowed – to increase visibility of tasks from this requester.
    • Worker profile pages carry information about rating, tier, level, skillset and task preferences. Registering for more skills, can increase task visibility.
  2. Dashboard sorting  : Workers can sort HITs on the dashboard according to the level of complexity, skills required, estimated time taken to complete a HIT, nature of task etc. This can help increase productivity. Sorting by payoff and requester is a debatable issue.

What powers do experienced users of the platform have?


  1. New rating system, separte for both workers and requesters. This system basically involves grading based on different factors like for workers it can be, efficiency, accuracy, skill level, credibility, compatibility, approval rate, experience, social graph, etc.
  2. Top-level worker/requesters have moderator like privileges.
  3. Moderators can classify tasks into various categories by tagging tasks with keywords. This would ease down the HITs filter process, making it much more accurate.
  4. High-skilled workers can access tasks having higher complexity and payscale.

Dive Deeper into Specific Ideas

For each of the 4 ideas (2 for trust, 2 for power), describe (using diagrams, sketches, storyboards, text, or some combination) the ideas in further detail.

Please create a separate wiki page for each of your ideas, so we can link to them individually. The title of the wiki page should be Milestone 3 followed by your team name and a description of the idea itself (ex: Milestone 3 YourTeamName TrustIdea 1: Automatic Pricing for Tasks based on Average Completion Time). Once done, post a link to each of your trust-related ideas to http://crowdresearch.meteor.com/category/milestone-3-trust-ideas and a link to each of your power-related ideas to http://crowdresearch.meteor.com/category/milestone-3-power-ideas when done, following the instructions at http://crowdresearch.meteor.com/posts/bXSNbqihjajASBQEL

Trust-related Ideas

Power-related Ideas

Dark Horse idea

Describe your dark horse idea (using diagrams, sketches, storyboards, text, or some combination).

Please create a separate wiki page for your dark horse idea so we can link to it individually. Post the link on http://crowdresearch.meteor.com/category/milestone-3-dark-horse-ideas when done