Milestone 7 Opera

From crowdresearch
Jump to: navigation, search


Crowd Justice : A crowdsourcing platform with dispute resolution


Crowdsourcing platforms are emerging online labor markets, but they do not mandate any labor laws or government control. Workers of crowdsourcing platforms like Amazon Mechanical Turk (AMT) do not encourage government or 3rd party intervention [1] . As a result , workers often face unjust rejection from requesters and no one takes responsibility of resolving such disputes. We introduce a new platform which has an inbuilt redressal forum by handling disputes like another normal tasks. The platform will have an inbuilt ranking system which will benefit the redressal procedure.


Crowdsourcing platforms like AMT are typically biased towards requesters [1] since they are the source of income to the platform. Hence workers take help of add-on services like Turkopticon [2] to perform requester rating which gives the worker an indication on the fairness of the requester. While such ranking tools are preventive actions, there is no way to actually address an unfair rejection. Hence workers feel disrespected and may tend to move away from the system.

This motivates us to come up with a platform that is fair not just to workers but also requesters such that neither gets exploited or lives under fear of rejection. Having such a dispute resolution mechanism will make the platform more trustworthy and will attract and retain requesters and workers.

Related Work

We studied some papers and articles [4] [5] which talk about features that should be part of a crowdsourcing platform, but none really talks about dispute redressal forum. However, Herik and Demov [6] propose that crowdsourcing can be used to perform dispute resolution, in general.


The system will have an inbuilt dispute resolution platform. It will also have a ranking system that will automatically rank the workers and requesters based on certain attributes. The dispute redressal will also consider the rankings while making decision and outcome of the dispute can affect the rankings of the guilty party. Hence both parties will try to be fair in order to avoid a dispute situation in the first place.



The following challenges were identified after panel discussion:

  1. Guaranteed Dispute Handling - How do we make sure that disputes raised are addressed and don't lie unattended ?
  2. Fair Judgement - How do we ensure that the party handling disputes is not biased ?
  3. Bounded response time for disputes - How do we ensure that disputes get resolved within reasonable time ?
  4. Dispute overflow - How do we ensure that there are not too many fake disputes created ?


The worker, while registering will have an option to opt for premium account or standard account. With a premium account, the worker will be eligible to raise disputes. We can have range like Silver, Gold, Platinum, with increasing cost that the worker needs to pay , and with each option, he gets to raise more disputes per month/quarter. The worker can upgrade his standard account to a premium account and can also renew his premium account per quarter/year.

The requester when raising a task will need to pay dispute reserve in advance to the system, which will get refunded once the task is successfully closed. Dispute reserve is safety net against the unfair rejection by requesters. In case worker gets unfair rejection and requester found guilty then some portion of this reserve will be paid towards the dispute resolution expenses.

With this money that system gets from worker and requester, it can sustain a panel to handle disputes. The platform can decide whether to have the panel managed by 3rd party or have in-house members for it who are not workers or requesters but have knowledge about crowd sourcing.

It may happen that a worker does not have money to afford a premium account but still wants to raise a dispute, in case of which, the dispute reserve will be taken from the worker by waiving off the task price for the next n tasks he will work on. This reserve only be collected if worker raises disputes and will be released if the dispute is proven to be correct otherwise retained by the system.

Once the worker raises dispute, the panel is responsible to guarantee a resolution in the bounded time frame. The verdicts of the panel are subject to upvote/downvote by the users of the platform so that the system knows the quality of verdicts overtime.


  • There is guaranteed timely response for disputes.
  • Since the panel is not a subset of system users, the verdict is unbiased.
  • Since there is a cost to raise a dispute, the worker will raise it only when needed.


  • The panel may need some learning curve to know the kind of tasks and issues faced.
  • If the panel is a 3rd party, there should be a pact to not leak the intellectual property of the requester outside the system


To measure the effectiveness of the system, an experiment will be conducted with a set of requesters and workers , out of which some will be genuine users and some will be malicious by intent. The experiment will start with high ranking given to the malicious users and over time the task they generate/work on will be made sure to come under dispute by deliberate mishandling. If overtime their ranking comes down, it’s an indication that the dispute resolution is working and effective.


  1. Turkopticon - Interrupting Worker Invisibility on Amazon Mechanical Turk
  2. mCrowd - A Platform for Mobile Crowdsourcing
  3. Managing the Crowd: Towards a Taxonomy of Crowdsourcing Processes
  4. The Case of A Crowdsourcing Platform
  5. Crowdsourcing for Enterprises
  6. Towards Crowdsourced Online Dispute Resolution
  7. Crowdsourced Online Dispute Resolution
  8. Milestone 3 Opera Karma Points: Worker/Requester Performance Rating