Moderators as arbiters of disputes

From crowdresearch
Jump to: navigation, search


It is our thesis that both professional workers and fair requesters want to rid the system of requester scammers and low effort/lazy/cheating workers. The current system gives complete power to the requester to accept or reject tasks as they see fit. However professional workers and fair requesters would both want all requesters to be fair in rejecting tasks. This benefits the worker directly by increasing their earnings and the requester indirectly by improving trust and reliability of worker acceptance ratios.

Raising disputes on rejected tasks

Currently, workers have no power to protest rejection of tasks except talking about their experiences on fora and rating requesters on platforms such as Turkopticon. We propose that in every case of rejected tasks, workers should be allowed to "raise disputes". It is in the interest of all altruistic stakeholders that these disputes be resolved fairly. If a worker frivolously raises a dispute, they should be fined and similarly for a requestor who unfairly rejects a task, is disputed and then found to be in the wrong.

Selecting the arbiters

This is the hardest part of the entire idea. We need to choose people who are selfless and impartial, and are committed to the cause of making the system function fairly and efficiently.

Arbiters subject to review

There is the option of relaxing the criteria for selecting arbiters and instead have a review process for every disputed decision, but that again adds a layer of authority/power. If we do not want to add another layer of review on top of these moderators, the only option left is peer review in case of disputed rulings. This sacrifices moderators' independence(if they were unaccountable to anyone, they would rule the way they felt rather than the way they felt the majority of the moderators feel) for consistency(the outcome of a particular dispute would not be as sensitive to the choice of moderator).

Unaccountable arbiters

If we consider a system where these arbiters or moderators are unaccountable to anyone, then there must be an excellent vetting and selection process which must also somehow defend against future corruption(they may accumulate bias over time, or get demotivated and stop putting in effort) of these moderators. Although the selection procedure would be hard to perfect, it is simpler that implementing a review process hence we will be working with this model.

Eliminating bias

Not every dispute is clear cut, and different people, given the same dispute, may side with worker or requester. e.g. a worker may be more sympathetic towards workers, and requesters towards requesters. Thus there is an inherent conflict of workers' and requesters' interest when trying to eliminate bias in future moderators. In fact one who seems biased to a worker may seem a neutral to a requester, and vice versa.

One claim we can make, however, is that highly rated requesters on Turkopticon are probably fair and 'liked' by workers, and workers with high approval ratios are professionals and liked by requesters. Thus these stakeholders are probably invested in the fair function of the system.

Election of moderators by altruistic stakeholders

There could be a process where some subset of these stakeholders are allowed to vote for moderators, and the best k candidates are selected. Moderators have the option to resign at any time leaving their post vacant and open to a vote. In addition to voluntary resignation moderators should also have a maximum term. The value of each individual's vote could somehow be weighted by their acceptance ratio/Turkopticon approval rating.

Each decision of a moderator should be made public so as to provide data on how good of a job they are doing.

Resolution of a dispute

When an issue comes to an arbiter it should be anonymised as much as possible, i.e. it shouldn't reveal the identity of the worker and requester. The arbiter decides which party is in the wrong and chooses the fine from a range of 0 to a precalculated maximum(a function of the reward for the task?).

Although requesters in general deal with more money than workers(on average, the ratio would be equal to the ratio of workers to requesters), since they are also open to proportionally more dispute charges, we believe that the maximum fine amount for workers and requesters should be the same.

The maximum fine amount should be low enough to :

1. Not force a requester to just accept all submissions, even low quality ones

2. Not force workers to accept mass rejections

while at the same time high enough to

1. Deter requestors from rejecting moderate effort work

2. Deter workers from disputing a fair rejection just for a chance of salvaging their acceptance ratio

We propose that the maximum fine should be a sublinear function of task reward (10 x (reward)^0.5 seems fine). A fraction of this amount can go to the moderators, however we do not think it wise for a fraction to be paid to the erred party as reparations as this may encourage frivolous disputes and unfair rejections.