WinterMilestone 4 by Team - witty

From crowdresearch
Jump to: navigation, search

Modifying task ranking used by Boomerang

In today's crowdsourcing platforms, workers and requesters are often unable to trust each others’ quality, due to a flawed repuatation system. To address this problem, Daemo introduced Boomerang[1], a reputation system that differs from traditional rate-and-leave systems by “boomerang-ing” the accuracy of the rating decision back to directly impact the user.This feedback loop means that giving someone a high rating increases the likelihood of working with that individual again, while giving a low rating reduces that likelihood. Boomerang, though solves the problem of reputation and combats the widespread rating inflation ,does not take into account the interests of the workers, and has no measure by which the various tasks can be ranked.

In the current setting of Daemo, a worker's task feed appears as follows : their task feed is ordered with the group of requesters which they have rated the highest at the top, then the group they have rated mediocrely, and then the group of brand new requesters, and finally the group they have rated the least at the bottom. Nowhere in the current setting, the worker's interests are taken into consideration. Consider a case where a worker has rated a requester badly for some task, and hence she will appear at the bottom of worker's task feed henceforth; even if she posts a task which is of worker's interest. The worker will end up doing tasks appearing at the top of his feed; and may never even reach the task of his interest! Also, the requester will never be able to get the highest quality results since the tasks are not performed by the workers who actually like doing that task! So, we see that requesters are worried about not getting high quality results and workers are disappointed that they don't get to work on the tasks suited to their preferences,hence are not able to perform to their full potential; and as a result produce low quality results. This leads to a 'vicious cycle' in the marketplace.

We present a modified 'task ranking' system which would incorporate worker's interests, skills and will provide further recommendations based on the tasks she recently performed.

  • Task division into categories

Every time a new worker joins Daemo, she will be asked to choose certain categories suiting her interest. For example, she can choose data entry, labeling, web development etc. Not only this, she can also add categories in this pool. When a requester creates new tasks, she will be asked to choose certain tags which associate with those tasks. Then we can easily filter out workers who would be interested in performing these tasks based on the categories chosen by them earlier. This would account for the worker's interest in task ranking system.

  • Skill endorsements

We introduce an endorsement system where every time a requester accepts someone's work, she will be asked to endorse the worker for certain skills based on the task she has completed. For example, if the worker has completed some web development assignment then the requester should endorse the worker for skills like html ,css ,web development etc. This would help us to match tasks with workers according to their specialization.

  • Recommendations based on recently performed tasks

Every time a worker completes some task, he would be notified with other similar tasks (can be found via categories) which are currently active. Carrie J. Cai et al. found that participants completed tasks faster when they were preceded by the same type of task, whereas they found challenging tasks less mentally demanding when preceded by tasks on the same content.

Result

This is roughly how the worker's feed will look like :
Ranking.JPG

References

1. http://crowdresearch.stanford.edu/w/img_auth.php/e/e8/Daemo_white_paper_%28private%29.pdf
2. Cai, Carrie J., Shamsi T. Iqbal, and Jaime Teevan. "Chain Reactions: The Impact of Order on Microtask Chains." Proceedings of the 34th Annual ACM Conference on Human Factors in Computing Systems (CHI’16). ACM. Vol. 6. 2016.

Milestone Contributors

@vrinda1994 , @witty123