Milestone 6 Opera

From crowdresearch
Jump to: navigation, search


Crowd Justice : A crowdsourcing platform with dispute resolution


Crowdsourcing platforms are emerging online labor markets, but they do not mandate any labor laws or government control. Workers of crowdsourcing platforms like Amazon Mechanical Turk (AMT) do not encourage government or 3rd party intervention [1] . As a result , workers often face unjust rejection from requesters and no one takes responsibility of resolving such disputes. We introduce a new platform which has an inbuilt redressal forum by handling disputes like another normal tasks. The platform will have an inbuilt ranking system which will benefit the redressal procedure.


Crowdsourcing platforms like AMT are typically biased towards requesters [1] since they are the source of income to the platform. Hence workers take help of add-on services like Turkopticon [2] to perform requester rating which gives the worker an indication on the fairness of the requester. While such ranking tools are preventive actions, there is no way to actually address an unfair rejection. Hence workers feel disrespected and may tend to move away from the system.

This motivates us to come up with a platform that is fair not just to workers but also requesters such that neither gets exploited or lives under fear of rejection. Having such a dispute resolution mechanism will make the platform more trustworthy and will attract and retain requesters and workers.

Related Work

We studied some papers and articles [4] [5] which talk about features that should be part of a crowdsourcing platform, but none really talks about dispute redressal forum. However, Herik and Demov [6] propose that crowdsourcing can be used to perform dispute resolution, in general.


The system will have an inbuilt dispute resolution platform. It will also have a ranking system that will automatically rank the workers and requesters based on certain attributes. The dispute redressal will also consider the rankings while making decision and outcome of the dispute can affect the rankings of the guilty party. Hence both parties will try to be fair in order to avoid a dispute situation in the first place.


The system will have a set of workers either with knowledge of law or workers/requesters with higher rankings which will have access to disputes. Both requester and workers will have the power to raise dispute for a given task which in itself will create a task which will be available to the set of workers eligible for dispute redressal.

Like other tasks, the dispute task will also have a priority and time limit. The outcome of the dispute task is a verdict that will mark either of the other parties as guilty. The guilty party is supposed to pay the dispute task price to the dispute resolver. The party found guilty will experience decrease in ranking points.

In order to ensure that the dispute resolver is fair, the same dispute task will be peer reviewed by an odd number of other dispute resolvers to give verdict.

Ranking will be done based on their karma points[8].


To measure the effectiveness of the system, an experiment will be conducted with a set of requesters and workers , out of which some will be genuine users and some will be malicious by intent. The experiment will start with high ranking given to the malicious users and over time the task they generate/work on will be made sure to come under dispute by deliberate mishandling. If overtime their ranking comes down, it’s an indication that the dispute resolution is working and effective.


  1. Turkopticon - Interrupting Worker Invisibility on Amazon Mechanical Turk
  2. mCrowd - A Platform for Mobile Crowdsourcing
  3. Managing the Crowd: Towards a Taxonomy of Crowdsourcing Processes
  4. The Case of A Crowdsourcing Platform
  5. Crowdsourcing for Enterprises
  6. Towards Crowdsourced Online Dispute Resolution
  7. Crowdsourced Online Dispute Resolution
  8. Milestone 3 Opera Karma Points: Worker/Requester Performance Rating