WinterMilestone 5 stormsurfer

From crowdresearch
Revision as of 17:53, 14 February 2016 by Shreygupta (Talk | contribs)

Jump to: navigation, search

System

Please use the following template to write up your introduction section this week.

Brief introduction and problem being solved

For workers, task ranking boils down to two things:

  • Finding good requesters: e.g. requesters that pay well, requesters that accept/reject HITs fairly/on-time, requesters that maintain strong/positive communication.
  • Finding good tasks: e.g. interesting tasks, tasks that match the worker's abilities, tasks that allow the work to learn something new, tasks that pay a high wage (money/time).

For requesters, task ranking boils down to simply one thing:

  • Finding good workers: e.g. workers that do a task proficiently.

Task ranking preferences, however, largely vary from user to user, and at different points in time, the same user may have different preferences. How do we maximize utility for both parties at any given moment? Currently, other systems face a similar problem: Facebook allows the user to choose rank between "top stories" and "most recent," and our very own voting platform allows users to rank between various sorting methods, including "top," "new," "best," and "needs comments/views."

Why not allow the user to choose in Daemo?

Modules of the system

I thus propose allowing the user to choose between several different rankings, including:

  • Dynamic ranking, based on a combination of requester and task quality.
  • Requester ranking, based on requester quality (as with Boomerang).
  • Wage ranking, based on estimated wage (from workers who have performed similar tasks) for a task.
  • Discovery ranking, which includes new tasks that the worker can explore.

Module 1: dynamic ranking

Problem and limitations

Crowdsourcing platforms such as Amazon Mechanical Turk, Clickworker, and Microworkers fail to address all aspects of reputation and ranking between workers and requesters. Workers cannot easily search for and find requesters and tasks they are interested in, and requesters cannot easily be matched with high-quality workers who are proficient in their specific tasks.

Currently, Boomerang solves one aspect of the problem by introducing "a reputation system that differs from traditional rate-and-leave systems by 'boomeranging' the accuracy of the rating decision back to directly impact the user. This feedback loop means that giving someone a high rating increases the likelihood of working with that individual again, while giving a low rating reduces that likelihood." Boomerang thus effectively matches high-quality workers with high-quality requesters (and vice versa), yet no match is made in regard to task category (classification) and worker proficiency with respect to that category.

Module preview

In this module, we introduce the dynamic ranking feature, an algorithmic approach to categorizing and ranking requester tasks. Requesters can tag tasks into categories such as "transcribing," "summarizing," and "image tagging," and workers can search for tasks based on tags using the search utility. Machine learning will be used to train the system to automatically categorize tasks.

The dynamic ranking feature is unique in that it builds on Boomerang to take into account not only worker and requester ratings but also the task completed. For example, a worker who may have completed tasks in two different categories for the same requester may be proficient in one area (e.g. "transcribing") but not another (e.g. "image tagging"). The requester, therefore, is likely to rate the worker high with respect to the tasks in the first area ("transcribing") but low with respect to the second ("image tagging"). Given that the worker completed a roughly equal amount of each type of task, with the current state of Boomerang, all of the requester’s tasks would most likely appear in the middle of the worker’s task feed, and the requester’s tasks would be available to the worker at approximately an average decay.

However, the dynamic ranking feature will have an updated ranking algorithm that takes into account task category, resulting in a divergence in both the position where a requester’s different tasks will appear in the worker’s task feed and the decay at which they will be available. In the above example, the specific requester’s "transcribing" tasks will appear at the top of the worker’s task feed, and the worker will have early access to these tasks, because the worker was rated high by the requester for this specific category of tasks. On the other hand, the specific requester’s "image tagging" tasks will appear at the bottom of the worker’s task feed, and the worker will have delayed access to these tasks, because the worker was rated low by the requester for this specific category of tasks.

As more requesters rate the worker with respect to the various task categories, the feature will learn which types of tasks the user is interested and proficient in. In the previous example, tasks categorized as "transcribing" by new requesters will begin to populate the worker’s task feed, below the "transcribing" tasks create by (high) rated requesters but above "image tagging" tasks created by all requesters.


System details

Module 2: requester ranking

Problem and limitations

Module preview

System details

Module 3: wage ranking

Problem and limitations

Module preview

System details

Module 4: discovery ranking

Problem and limitations

Module preview

System details

Milestone contributors

Slack usernames of all who helped create this wiki page submission: @shreygupta98