Reputation System Idea from James

From crowdresearch
Jump to: navigation, search

This post is mostly concerned with creating generic scores for workers and requesters. It does not discuss how the score will be used on the system. It may be compatible with other reputation systems.

Initial Rating

Anonymous logs of worker/requester interactions are rated by workers. The workers rate the quality of both the requester and the worker on a scale. This leads to a basic rating for both requesters and workers.

Worker Comparison

If a particular log is reviewed by multiple workers, the ratings they assign can be compared. Workers who consistently disagree with the majority of their peers will have their rating lowered while those who consistently align with the majority will have their rating increase. This task will effectively act as a milestone 0.


I suggest having little or no payment for this task. Workers will be building their rating when they perform the task so they have a form of compensation. If monetary compensation is needed it can be generated by imposing a tax on requester payments.


The data generated with this approach can be used to train a machine learning algorithm in order to identify positive and negative interactions without relying on the crowd. This future system would be able to give immediate warnings to users with negative behavior (or those deemed 'at risk' of behaving negatively) before their scores are severely impacted.

Other Thoughts

This proposal is an attempt at crowd sourcing the reputation system.

PRO: It is anonymous and thus it should prevent inflation and create meaningful ratings.

CON: Since workers and requesters do not directly rate each other, they will not be able to form preferential relationships.