Milestone 8 teamtrojan Foundation 3

From crowdresearch
Jump to: navigation, search

Foundation 3: External quality ratings

Group or authority ratings OR algorithm based ratings

Implicit ratings have the disadvantage of being biased. This gives rise to an external group or authority in the ratings system. Though, the group or authority may not be biased, there are other factors that may tender the ratings invaluable. These factors include:

1. Since the external authority or group are not a part of the platform, they may not give true ratings.
2. The group may not know all factors to be considered before rating a worker
3. They may provide dummy ratings if they fail to understand a particular situation or a particular mechanism

The platform can allow both, implicit and explicit ratings. The weighted average of the two can be computed to obtain the final rating. As an alternative, each type of rating can contribute a fraction to the final rating. The mechanism of anonymous ratings can be introduced to limit the partiality of implicit ratings.


In order to have explicit ratings, having a single admin or an authority will not scale. A committee can be present to provide ratings, given a list of factors to be considered.

Though the idea of having an algorithm provide ratings has several advantages, the potential problems that may arise are:

1. It may be difficult to provide the rating parameters such as, consistency, skill set etc, to the algorithm.
2. The initialization and updation of these parameters may need complex and accurate machine learning algorithms.
3. It may not be possible to accurately test the parameter values. It may be more challenging to obtain the train and test data sets for the algorithm.

If it’s a group, who pays for their time to review

Money can be collected from requesters and turkers, example registration fees, in order to pay reviewers.

How do you do skills-based ratings, etc., without hindering tasks with a requirement to categorize them?

The advantages ratings can provide requesters:

1. Requeters trust high rated workers more
2. There will be less scrutinization of a highly rated worker's response by the requester
3. High ratings improve the probability of work acceptance.


Though ratings play an important part, they are solely not responsible for fetching work. The main factor is a worker's quality of submitted work. A posted task is visible to all workers. Requesters cannot limit work to highly rated workers only.