WinterMilestone 4 Enhance work quality without paying much (science) by Ahmednasser

From crowdresearch
Jump to: navigation, search

Crowdsourcing and (price vs quality)

crowdsourcing becomes a good way for make the tasks done by another people and there are many available platforms now like Mtruk, Microworkers, clickworkers and many other platforms that help requesters in finishing their work, this work maybe image tagging, filling surveys or translating some text or many other forms of work.

In any platform a requester posting some tasks (HITs) and offer a price for that, then workers who interested in this work accept it and start doing it, after workers finish the tasks if the requester see that it match what he want he pay for workers but the problem is that requesters suffer from lack of quality in the delivered work as the workers not in the same level of experience, they don’t have the same skills and some of them don’t follow the instruction they just want to finish tasks and get paid. and If they want a high quality work they have to hire expert workers to make this work done which will make requesters pay much money.

The questions that I want to answer

  • Will the quality of work be improved if we used experts to review what non experts did ?
  • Could we get high quality work without paying much ?
  • What do requesters prefer high quality work with much money or acceptable quality with acceptable money or poor quality with few money ?

The experiment

This experiment based on assumption that we have two types of workers Experts and Non-Experts
I prepared a task about English text and I want it to be translated to Arabic, a HIT is consists of 3 lines and I tried 3 scenarios

1) Non expert workers translated it and experts revised it
so I posted them and the work was done by 100 worker and I paid .05 $ for each worker after that I post the results to a new HIT and hired 1/10 of workers = 10 experts, so I had 10 experts each one had 10 paragraph to revise and enhance it if needed, each one of the experts got $ .5 for the whole work, the total I spent is .05 * 100 for non-experts + .5 * 10 for the experts which equals $ 10.

2) The whole translation was done by experts
I created the HITs with constraint that this work must be done by experts not normal workers and for each HIT I paid .2 $ (which is more than what I paid for the normal user in the first scenario) and the work was done by 100 expert and the cost was 100 * 0.2 equals $ 20.

3) The whole translation was done by non-experts (normal workers)
After creating and posting the 100 HITs and offer $ 0.05 for each HIT the work was done by 100 worker and I paid $ 5 for the whole HITs.

The results (Hypothesis)

  • The work in the 3rd case which was done by normal workers is poor quality one but has cheaper price.
  • In the 2nd case the work has a high quality as it was done by experts but the cost was high I had to pay much for the experts.
  • The HITs that was done by workers and revised by experts had high quality and cheaper price than the HITs done by experts only.
  • Requesters would prefer the first case as it provides acceptable price and quality
  • In the first scenario requesters won’t pay much for experts as they don’t perform the HITs from scratch, they just revise others work.


@ahmednasser - Ahmed Nasser Hussien