Milestone 3 PixelPerfect TrustIdea 2: User Rating System

From crowdresearch
Jump to: navigation, search

User Rating System


We draw inspiration from the cult video game Need For Speed : Hot Pursuit, where players can choose to play either as a racer or a cop. Players level up in their respective careers and earn rewards and tools to help with their racing and chasing. They earn XP based for successful getaways and busts and completion of career-specific milestones.

We adapt the Hot Pursuit model to crowd working platforms : Workers and requesters are the racers and cops. Both classes progress along their careers in their own way. Workers gain experience/rating for accurate and fast task completion, while requesters earn it for prompt payments, clarity of tasks and professionalism in their work.

Leveling up (experience) earns more privileges on the platform and unlocks complex and higher-paying tasks.
Higher ratings are indicative of quality of work and trustworthiness.

Experience and Rating

Experience points are earned through task completion and generation without considering quality. Earning a set amount of experience allows a worker or requester to level up. Levels help unlock higher privileges and productivity tools. Experience points are indicative of the quantity of work done/given.

Rating is assigned to each worker or requester and is a function of several factors. Rating is indicative of quality of work done/given and credibility.

  • Workers earn rating for accurate and fast completion of tasks, skill level and knowledge, time spent on platform, compatibility with requesters.
  • Requesters ratings are similar to that on Turkopticon – generosity, promptness, professionalism, fairness and communicability, task clarity and feedback history.

Workers and requesters are classified into tiers based on experience and rating.


Tier System

On completion of a set number of milestones by increasing experience and rating, workers and requesters move from and into tiers. Tiers are broad classes of platform users, grouped mainly for convenience of distributing benefits and assistance. We propose a 3-tier system for both workers and requesters.


Worker Tiers

  • Professional or expert (Tier 1) : Workers who have lot of experience with the platform and are focused on efficient completion of tasks. Some tools designed to increase productivity become available to them – next task becomes available immediately after completion of tasks without task selection screen. Machine learning data is used for task recommendation. Options to skip over to another task or disable this feature can be provided.
  • Mature or intermediate (Tier 2) : Workers who have established themselves on the platform and now qualify for taking on actual tasks. A machine learning algorithm interacts with their task selection choices and learns about their preferences. Preferences can also be manually tweaked. This helps generate HITs suited to them later on.
  • Novice or entry-level (Tier 3) : Workers with a few days experience with the platform and just starting out. Lot of tasks might not be available to them at this point, since they are relatively unknown. Computer generated tasks whose solutions are known are provided to them – we can gauge their skillset, accuracy and efficiency. This acts as an initial seed for rating and experience.

Requester Tiers

  • Professional or expert (Tier 1) : Requesters who require large volume of human computation and use the platform on a regular basis. Tools are made available to increase their productivity and help with the analysis of results.
  • Mature or intermediate (Tier 2) : Requesters who have understood the platform and are now capable of generating well-designed tasks with appropriate pay.
  • Novice or entry-level (Tier 3) : These are requesters who are just starting out with the platform and have no idea how much to pay for their tasks. Preferably, early requesters must work as a worker to understand the hardships faced and give due respect to the worker.