Milestone 6 pentagram

From crowdresearch
Jump to: navigation, search

This is the submission page for Milestone 6 by Team pentagram.

Title

ReCo - An iterative CrowdSourcing solution

Abstract

ReCoTurker is an iterative crowdsourcing platform which focuses on establishing clarity in tasks posted on the platform and a reviewal system for resolving conflicts in HITs submitted for a task. The platform also provides additional features like worker and task recommender system, user ratings based on reviews/performance and a priority system to establish implicit mutual trust between workers and requesters along with an assurance for quality in HITs. Unlike the existing platforms ReCoTurker provides a very interactive interface to review the rejection for a submitted HIT which is supervised by a third party Moderator. It also includes a Task Review System whose panel is comprised of highly rated and experienced workers reviewing the tasks submitted by a requester before hosting the same on the platform. The recommender system suggests suitable tasks for a worker and potential workers for a requester.

Motivation

There are some very apparent problems with existing crowdsourcing platforms like Amazon MTurk and Microworkers etc. For example, reviews of submissions sometimes takes a very long time, of the order of 4-6 weeks. When HITs are rejected, the feedback given to the worker are not informative enough to justify the rejection. In addition to this, there are user experience design issues like non-intuitive GUI, inaccessability in certain countries etc. We ourselves worked in a few existing crowdsourcing platforms and faced few of these very problems. Speaking to veteran workers also provided some insight to the problems they face in finding HITs, working timings etc. Based on these observations, we arrived at the conclusion that the needs of the workers and requesters on these micro-task platforms boils down to two primary issues viz. trust and power. Workers are not in a position to verify the authenticity of the requester or be assured of pay for the work done. In addition to this, there is no explicit power given to any person to edit/modify posted work, or verify HITs for correctness. Since these issues are long-standing, and of critical importance, there is a need for a new platform which addresses these issues. We have suggested some solutions to overcome these issues.

Related Work

A lot of work has gone into improving the algorithms for checking errors and weed out spammers and correct the results of biased workers. There have been attempts made at creating a reputation system as well.

However, these systems are all trained and tested on the crowdsourcing data available and may not be capable to ensure the clarity of tasks presented by requesters as workers range from novices to experts and their understanding of the tasks vary. The feedback taken from workers/requesters may not be trustworthy as it utilises some of their time that may be spent elsewhere. The idea to bring in more interaction has not taken off in current crowdsourcing platforms due to the their indifference towards promoting solidarity as they do not see this as a motivator for better work quality.

Insight

The ReCoTurker platform is starkly different from the other existing platforms through an exhaustive worker and task recommender system. It provides a strong reviewal mechanism to fairly solve a conflict(if any) regarding a HIT's acceptance by a requester. The platform also ensures a smooth and unbiased functionality of the entire process . The cash flow in the system is transparent and closely monitored(through a third party Moderator)

System

The work-flow of the system is described by the following components :

Task and worker recommender system

Tasks are recommended to the worker based on his history of completed tasks and the skills that he has been endorsed for.These recommendations considerably reduce the time taken for searching tasks.On the other hand the requester is recommended workers based on their skill levels,their current priority and other requirements of requesters.

Task review system

Before the task is posted on the platform, it is reviewed by a panel of reviewers.The panel might itself consist of experienced requesters and workers.This is to ensure that the task posted is possessed of clarity.This process is iterative and is even extended after the task is posted on the forum.

Conflict resolution system

Once a worker completes a HIT, it is judged and paid/rejected by the requester.If the worker feels he has been judged wrongly,he appeals for a reviewal to the moderator.The moderator,himself is a designated third party, judges the work done and gives a conclusion.If judged unfair, the moderator gets money from the requester and compensates the worker accordingly.The rating of that requester is correspondingly reduced. Else the moderator reduces the rating of the worker.

Priority system

In the background, a process continually checks for the worker/requester rating.If the rating is reduced below a certain threshold,then requester/worker is pushed to a low priority zone.

Evaluation

  • Feedback can be taken from the worker after a task has been submitted, regarding the clarity of the task , the amount of involvement of the requester during the process, promptness and fairness of requester in payment. To ensure authenticity, the survey is to be completed after the requester has assessed the quality of the work, to know the result of his work and activate the money transfer.
  • This feedback can be presented to the requester which may improve his requesting practices. If a requester gets poor reviews repeatedly, his rating will go down and his tasks will not be recommended to the skilled workers. This makes him strive to set clear tasks, pay workers well and increases workers trust on him. Frequent malicious feedback from workers (giving poor reviews while majority have given good feedback) will reduce their rating.
  • The recommender and priority systems can be improved frequently based on the satisfaction of workers and requesters. A forum or social networking platform may be created to publicise tasks, ask for help, express grievances and also to increase the cohesion and bonding in the crowdsourcing community.

References