Milestone 8 TuringMaching Foundation3

From crowdresearch
Revision as of 20:23, 22 April 2015 by Avrimmit (Talk | contribs) (Solutions)

Jump to: navigation, search

Foundation 3: External quality ratings

Metaphor of credit ratings: rather than just people rating each other, have an (external?) authority or algorithm responsible for credit ratings (A, B, C, etc.)
 Benefit: this reduces the incentive to get positively-biased five star ratings on everything — those ratings become meaningless


  • Is this a group/authority? For example, Wikipedia reviews are subjective and based on voting. Or is it an algorithm?

  • If it’s a group, who pays for their time to review you?

  • From Anand: “How do you do skills-based ratings, etc., without hindering tasks with a requirement to categorize them?”


We proposed the semi-automated rating mechanism in Trust Rank, Sustainable Reputation Framework for the Crowdsourcing Marketplace

We believe the cost of setting up external rating mechanism is high. However, internal rating mechanism can be made more robust. This system can evolve and learn from the historical rating behaviors. For instance, to overcome the problem of bias voting we propose following mechanism:

Hypothesis: The worker will not prefer to continue business with the bad requestor. Requestor will not prefer to continue business with the bad workers.
No worker or requestor will prefer to continue working after a bad experience with each other.

  • We tune our recommendation system based on the ratings given by workers/requestors.
  • If worker gives good rating to the requestor, algorithm pairs them for the next 3-5 tasks. In this situation for the good agents, truth telling is the dominant strategy. Please see the payoff matrix. This enforces workers and requestor to provide honest judgement about the ratings.
  • During the sign up, the system will present the detail demo of Ranking System to Both requestors and workers
Game Theoretic Setting with Nash Equilibrium


  • Rating Parameters for Requestors: Generosity, Fairness, Promptness, and Communicativity are widely used ranking parameters Irani 2013. In addition, we introduce Probability of Payment Defaults, PPD as a parameter to indicate historical track record of the requestors.
  • Finally, we associate weights to each parameter and determine the final score as a function of the parameters. In long run, as we collect more parameters, we can reduce the dimension of the data using PCA. Then we can select the components that explain high variance in the ranking and derive loadings (weights) from that.
  • Another method for rank prediction is Latent factor recommendation system which gives ability to automatically learn the features require for the rating prediction.
  • We can also incorporate extra parameters used in recruitment sites such as Glassdoor.
  • Workers: Similarly rating for workers can be derived based on Honesty, professionalism, past performance, commitment, and skill sets, widely used parameters while hiring the workers in the Fortune 500 companies.


Score Prediction Analytics


Rank Classification

The System: For further details please see Trust Rank, Sustainable Reputation Framework for the Crowdsourcing Marketplace

Trust Rank, Sustainable Reputation Framework for the Volatile Crowdsourcing Marketplace