Milestone 8 teamtrojan Foundation 3

From crowdresearch
Revision as of 08:51, 22 April 2015 by Rashmiputtur (Talk | contribs) (If it’s a group, who pays for their time to review)

Jump to: navigation, search

Foundation 3: External quality ratings

Group or authority ratings OR algorithm based ratings

Implicit ratings have the disadvantage of being biased. This gives rise to an external group or authority in the ratings system. Though, the group or authority may not be biased, there are other factors that may tender the ratings invaluable. These factors include:

1. Since the external authority or group are not a part of the platform, they may not give true ratings.
2. The group may not know all factors to be considered before rating a worker
3. They may provide dummy ratings if they fail to understand a particular situation or a particular mechanism

The platform can allow both, implicit and explicit ratings. The weighted average of the two can be computed to obtain the final rating. As an alternative, each type of rating can contribute a fraction to the final rating. The mechanism of anonymous ratings can be introduced to limit the partiality of implicit ratings.

In order to have explicit ratings, having a single admin or an authority will not scale. A committee can be present to provide ratings, given a list of factors to be considered.

Though the idea of having an algorithm provide ratings has several advantages, the potential problems that may arise are:

1. It may be difficult to provide the rating parameters such as, consistency, skill set etc, to the algorithm.
2. The initialization and updation of these parameters may need complex and accurate machine learning algorithms.
3. It may not be possible to accurately test the parameter values. It may be more challenging to obtain the train and test data sets for the algorithm.

If it’s a group, who pays for their time to review

Money can be collected from requesters and turkers, example registration fees, in order to pay reviewers.

Challenge question 3