Difference between revisions of "Milestone 3 PixelPerfect"

From crowdresearch
Jump to: navigation, search
(How do requesters trust workers and vice versa?)
(How do requesters trust workers doing their tasks and the results they get?)
Line 9: Line 9:
 
==== How do requesters trust workers doing their tasks and the results they get? ====
 
==== How do requesters trust workers doing their tasks and the results they get? ====
 
----
 
----
# Mechanism on the platform where workers who have the appropriate skillset and demographics are eligible for the tasks.
+
 
# Unique identification requires each worker, requester to submit valid ID proof to verify their genuineness. This might be helpful in certain HITs which are demographics specific (eg. Surveys involving only native American women).  
+
# '''Strict identification system''' : Each worker or requester is requested to submit valid ID and proofs at time of application, to verify their genuineness. This may be a combination of Social Security Number, Voting Card, Driving License, Ration Card, Aadhaar card (in India) etc. Worker details are by default, private and disclosed only if the worker wishes to. Each application is reviewed before users can join the platform.
 +
 
 +
# '''Demographic and skillset check''' : The platform shall have a mechanism where only workers who have the appropriate skillset and demographic are eligible for the task. These might be helpful in certain HITs which are demographic specific (eg. Surveys involving only native American women). Such tasks can be filled anonymously, with the system performing the demographic check based on the identification provided. If the tasks require disclosure of details, workers can only take up such tasks if they willing to forgo anonymity. This preserves the choice to be anonymous.
 +
 
 +
# '''Task templates''' : Good task design can go a long way in reducing spam in submissions. These can be done by including questions which require correct answering before completing rest of the survey/task. Correct answering forces spammers to read and appreciate the content, spending more of their time, and hence discouraging bad replies. Therefore, pre-designed task templates for different task classes are available for requester use, thus standardizing task quality. Some templates may have scripts/tools to help with evaluation. This helps requesters develop HITs quickly and get more trustworthy results. Workers also benefit from consistent clarity of standardized tasks.
  
 
==== How do requesters trust workers and vice versa? ====
 
==== How do requesters trust workers and vice versa? ====

Revision as of 19:13, 18 March 2015

Team Pixel Perfect's submission for Milestone 3.

Initial Brainstorm

Crux of idea goes here.

Trust-related Ideas

How do requesters trust workers doing their tasks and the results they get?


  1. Strict identification system : Each worker or requester is requested to submit valid ID and proofs at time of application, to verify their genuineness. This may be a combination of Social Security Number, Voting Card, Driving License, Ration Card, Aadhaar card (in India) etc. Worker details are by default, private and disclosed only if the worker wishes to. Each application is reviewed before users can join the platform.
  1. Demographic and skillset check : The platform shall have a mechanism where only workers who have the appropriate skillset and demographic are eligible for the task. These might be helpful in certain HITs which are demographic specific (eg. Surveys involving only native American women). Such tasks can be filled anonymously, with the system performing the demographic check based on the identification provided. If the tasks require disclosure of details, workers can only take up such tasks if they willing to forgo anonymity. This preserves the choice to be anonymous.
  1. Task templates : Good task design can go a long way in reducing spam in submissions. These can be done by including questions which require correct answering before completing rest of the survey/task. Correct answering forces spammers to read and appreciate the content, spending more of their time, and hence discouraging bad replies. Therefore, pre-designed task templates for different task classes are available for requester use, thus standardizing task quality. Some templates may have scripts/tools to help with evaluation. This helps requesters develop HITs quickly and get more trustworthy results. Workers also benefit from consistent clarity of standardized tasks.

How do requesters trust workers and vice versa?


  1. We use the Facebook social graph model to conceptualize worker-requester relations and compatibility.
We develop a crowdworker graph (bipartite for now i.e no relations within workers or requesters) whose nodes are requester and workers, and compatibility is modeled by the edge weights.
Compatibility develops through a two-way feedback system between workers and requesters after completion of a task. It is centralized
around a simple Yes/No question :
Would you like to do a task from this requester again? [For workers]
Would you delegate a task to this worker again? [For requesters]
Compatibility plays a role in how visible a given requester’s HIT is, to various workers. Workers having higher compatibility with the
requester are more likely to see the HIT and therefore take it up.

How do workers trust they are not being exploited?


  1. Implementation of Multi-level rejection system which also involves workers. Involvement of requesters and their counter parts makes rejection process much more fair and justifiable.
  2. Penalty system for delayed worker payment. Workers would be compensated for the delay in the payment as quite a large percentage of people rely on these platforms for their daily living.
  3. Changing the legal system. A common issue is work theft, where requesters do not acknowledge work submitted, however incorporate it in the results. This happens because requesters enjoy intellectual property rights over the work submitted and not the workers. A legal solution might be tough to implement considering the vast multitude of countries workers come from.

How do requesters trust they will get their results on time?


  1. Implementation of a Priority Queue system ,where tasks nearing their deadlines are given higher priority.

How do new workers get started quickly and earn the trust of their counterparts?


  1. New workers go through some initial computer generated tasks whose solutions are provided to them - we can gauge their skill set, accuracy and efficiency. This acts as their initial seed.
  2. Templates for design of HITs can be given to new requesters to enhance the clarity in their tasks.

Power-related Ideas

What powers do workers have to feel respected and voice their opinion?


  1. Two-Stage Worker feedback system for a particular HIT and requester.
    This is 2 line review of the idea.
  2. Issue column attached to every HIT where requesters can suggest improvement or give tips on completing the task.
    This is a 2 line review of the idea.

What powers do workers have to find HITs quickly?


  1. The visibility of HITs on a workers dashboard will be determined by previous work, skill set etc.
  2. Workers can arrange HITs according to the time taken to complete the HIT, in order to facilitate their work plan in that particular sitting.
  3. Reducing idle worker time. A common issue on current crowdsourcing platforms is, certain weeks generate good payoff for workers while other weeks don’t. While there is no effective way to predict trends in task demand, we can look for compensation techniques. Auto-generated tasks whose solutions are known can be made available at all times. During idle scenarios (low task demand), these can be taken up for work. This enables workers to improve their experience and ratings during downtime, which in turn makes them eligible for tougher and more intricate tasks later on.

What powers do experienced users of the platform have?


  1. New rating system, separte for both workers and requesters. This system basically involves grading based on different factors like for workers it can be, efficiency, accuracy, skill level, credibility, compatibility, approval rate, experience, social graph, etc.
  2. Top-level worker/requesters have moderator like privileges.
  3. Moderators can classify tasks into various categories by tagging tasks with keywords. This would ease down the HITs filter process, making it much more accurate.
  4. High-skilled workers can access tasks having higher complexity and payscale.

Dive Deeper into Specific Ideas

For each of the 4 ideas (2 for trust, 2 for power), describe (using diagrams, sketches, storyboards, text, or some combination) the ideas in further detail.

Please create a separate wiki page for each of your ideas, so we can link to them individually. The title of the wiki page should be Milestone 3 followed by your team name and a description of the idea itself (ex: Milestone 3 YourTeamName TrustIdea 1: Automatic Pricing for Tasks based on Average Completion Time). Once done, post a link to each of your trust-related ideas to http://crowdresearch.meteor.com/category/milestone-3-trust-ideas and a link to each of your power-related ideas to http://crowdresearch.meteor.com/category/milestone-3-power-ideas when done, following the instructions at http://crowdresearch.meteor.com/posts/bXSNbqihjajASBQEL

Trust-related Ideas

Power-related Ideas

Dark Horse idea

Describe your dark horse idea (using diagrams, sketches, storyboards, text, or some combination).

Please create a separate wiki page for your dark horse idea so we can link to it individually. Post the link on http://crowdresearch.meteor.com/category/milestone-3-dark-horse-ideas when done