Winter Milestone 5 Team - Witty

From crowdresearch
Jump to: navigation, search

Boomerang-an Interactive Task Feed

Brief introduction of the system

Boomerang is an interactive task feed for a crowdsourcing marketplace, that incentivizes accurate sharing of the information necessary for maintaining a healthy crowdsourcing platform, like accurate reputation ratings, skill tags on tasks, and hourly wage estimates for tasks, by making the information directly impact their future tasks or workers, hence the name 'Boomerang'.

How is the system solving critical problems

There is a massive amount of information necessary for a healthy crowdsourcing marketplace. Workers on the crowdsourcing platforms need to find new tasks that will maximize their income (reduce uncertainty in payment and rejection), find tasks that fit their expertise profile and are of their interest and refind old requesters' new tasks. Requesters need to get their work done by the workers who can guarantee high quality results and get their tasks completed as quickly as possible. So we need techniques for providing accurate reputation ratings, classification of tasks into various categories, and hourly wage estimates for tasks. Our system 'boomerangs' the accuracy of rating/classification decisions back to impact the user in the future. Workers' ratings of requesters is used to rank their task feed in a way that top ranked requesters appear at the top, then the low ranked requesters. Their task completion estimates are used to determine their hourly wage. Requesters' ratings of workers and endorsing them for skills are used to give early access to workers who that requester has rated highly and workers with suitable expertise on that task. Skill tagging on tasks ensures that the tasks go to workers with the required skillset.

Introducing modules of the system

Below, we introduce the different modules or functionalities that Boomerang supports. The first module presents a boomerang ranking system, which prevents inflated scores. The second module attaches tasks with various categories. The third module estimates hourly wage of a worker using her previous inputs and the last module is a user-friendly feature for getting notified of tasks/requesters a worker likes.

Module 1: Workers' and Requesters' ratings

Problem/Limitations

Workers need to find requesters who are reputable and can be trusted with payments. Requesters need to find workers who complete the tasks properly, give high quality results and possess suitable skills.

Module preview

Currently, workers either discuss among themselves about the requester reputation or seek the help of forums like Turkopiton. Requesters can't bet on the fact that the tasks they post are given access to workers best suited for that task. There is no universally accepted technique of doing so. Workers' ratings on requesters will rank their task feed such that their high rated requesters come on top, followed by mediocre rated requesters, then the new requesters(who haven't been rated by them yet) , followed by the requesters they have rated the least. Requesters' ratings on workers will ensure that the workers whom they have rated the highest get earliest access to the tasks. Along with simple rating, they will also 'endorse' the workers with skills on pertaining to the tasks they have successfully completed. For example, if the worker has completed a web development assignment then the requester should endorse the worker for skills like Web programming, HTML, CSS and so on. This endorsement will be combined with skill tagging(Module 2) to ensure task goes to the worker with suitable skills.

System details

After a worker has completed an assignment , she will be asked to rate the requester. Based on her experience, she can either rate the requester <check plus>, <check> or <check minus>. When the requester has evaluated a worker's task result, she will be asked to rate the requester. She will also be asked to endorse the worker for particular skills.

Module 2: Skill tagging

Problem/Limitations

Workers have no means by which they can find the tasks that interest them, and they always have to be on the edge of the seat to find exciting tasks! Requesters on the other hand have no control over who gets access to tasks they post and whether a task will be completed by a worker with the required skills.

Module details

When the requester posts a task, along with the required information she will be asked to 'tag' the assignment with appropriate categories this task belongs to, or the suitable skills required to complete this task. The worker who has highest endorsement for those skills (Module 1) will get earliest access to those tasks. This will also be helpful to the workers as they can sort the tasks by those tags and find the tasks of their interest in least time.

Module 3: Estimating the hourly wages

Problem/Limitations

Workers try to track their hourly wages using some third party platforms. These platforms have limited information and therefore, might not be reliable. The problem of estimating the hourly wage can easily be solved if, we can somehow, calculate the average time in completing that task.

Module details

  • Tracking the time a user takes in completing a task will be monitored implicitly with the help of a timer. If a user navigates to some other tab, the timer would be paused automatically. Additionally, the timer can be paused by the worker himself.
  • For estimating the time a worker takes in completing a task 'T', we will average the time taken by 'similar users' on the same task 'T'.
  • To calculate how similar two users are: we calculate the similarity index using some metric like Pearson Correlation using the time taken by them in completing the same set of tasks.
  • So, in this way we will find similarity scores for a worker with all the other workers who have completed the task 'T', and then we will calculate average of the completion time of workers who have a similarity score greater than a threshold 'K' with the said worker.

Module 4: Notifications

Problem/Limitations

Workers are often quoted saying that crowdsourcing platforms are 'unpredictable' as they never know when the tasks of their preference are posted and they have to be active 24X7.

Module Details

Workers can 'follow' the requesters they like and categories of their interest. Whenever their followed requesters post a task or tasks of followed categories are posted, they get a notification! Additionally, workers can follow other workers too; and when their followed workers complete a task, they get notified.

Milestone contributors

@vrinda1994 , @witty123