WinterMilestone 5 stormsurfer

From crowdresearch
Jump to: navigation, search

System

Please use the following template to write up your introduction section this week.

Brief introduction and problem being solved

For workers, task ranking boils down to two things:

  • Finding good requesters: e.g. requesters that pay well, requesters that accept/reject HITs fairly/on-time, requesters that maintain strong/positive communication.
  • Finding good tasks: e.g. interesting tasks, tasks that match the worker's abilities, tasks that allow the worker to learn something new, tasks that pay a high wage (money/time).

For requesters, task ranking boils down to simply one thing:

  • Finding good workers: e.g. workers that do a task proficiently.

Task ranking preferences, however, largely vary from user to user, and at different points in time, the same user may have different preferences. How do we maximize utility for both parties at any given moment? Currently, other systems face a similar problem: Facebook allows the user to choose rank between "top stories" and "most recent," and our very own voting platform allows users to rank between various sorting methods, including "top," "new," "best," and "needs comments/views."

Why not allow the user to choose in Daemo?

Modules of the system

I thus propose allowing the worker to choose between several different ranked feeds, including:

  • Dynamic ranking, based on a combination of requester and task quality.
  • Requester ranking, based on requester quality (as with Boomerang).
  • Wage ranking, based on estimated wage (from workers who have performed similar tasks) for a task.
  • Discovery ranking, which includes new tasks that the worker can explore.

However, task availability will always be dynamically ranked.

Module 1: dynamic ranking

Problem and limitations

Crowdsourcing platforms such as Amazon Mechanical Turk, Clickworker, and Microworkers fail to address all aspects of reputation and ranking between workers and requesters. Workers cannot easily search for and find requesters and tasks they are interested in, and requesters cannot easily be matched with high-quality workers who are proficient in their specific tasks.

Currently, Boomerang solves one aspect of the problem by introducing "a reputation system that differs from traditional rate-and-leave systems by 'boomeranging' the accuracy of the rating decision back to directly impact the user. This feedback loop means that giving someone a high rating increases the likelihood of working with that individual again, while giving a low rating reduces that likelihood." Boomerang thus effectively matches high-quality workers with high-quality requesters (and vice versa), yet no match is made in regard to task category (classification) and worker proficiency with respect to that category.

Module preview and system details

In this module, we introduce the dynamic ranking feature, an algorithmic approach to categorizing and ranking requester tasks. Requesters can tag tasks into categories such as "transcribing," "summarizing," and "image tagging," and workers can search for tasks based on tags using the search utility. Machine learning will be used to train the system to automatically categorize tasks.

The dynamic ranking feature is unique in that it builds on Boomerang to take into account not only worker and requester ratings but also the task completed. For example, a worker who may have completed tasks in two different categories for the same requester may be proficient in one area (e.g. "transcribing") but not another (e.g. "image tagging"). The requester, therefore, is likely to rate the worker high with respect to the tasks in the first area ("transcribing") but low with respect to the second ("image tagging"). Given that the worker completed a roughly equal amount of each type of task, with the current state of Boomerang, all of the requester’s tasks would most likely appear in the middle of the worker’s task feed, and the requester’s tasks would be available to the worker at approximately an average decay.

However, the dynamic ranking feature will have an updated ranking algorithm that takes into account task category, resulting in a divergence in both the position where a requester’s different tasks will appear in the worker’s task feed and the decay at which they will be available. In the above example, the specific requester’s "transcribing" tasks will appear at the top of the worker’s task feed, and the worker will have early access to these tasks, because the worker was rated high by the requester for this specific category of tasks. On the other hand, the specific requester’s "image tagging" tasks will appear at the bottom of the worker’s task feed, and the worker will have delayed access to these tasks, because the worker was rated low by the requester for this specific category of tasks.

As more requesters rate the worker with respect to the various task categories, the feature will learn which types of tasks the user is interested and proficient in. In the previous example, tasks categorized as "transcribing" by new requesters will begin to populate the worker’s task feed, below the "transcribing" tasks create by (high) rated requesters but above "image tagging" tasks created by all requesters.

As a result, workers and requesters will not only be matched on the basis of ratings between workers and requesters but also the type of task that was completed. This will allow workers to easily search for and find requesters and tasks they are interested in and requesters to easily be matched with high-quality workers who are proficient in their specific tasks.

This ranking is similar to the "top news" ranking on Facebook and the "top" ranking on our voting system.

Dynamic Ranking Winter Milestone 5 stormsurfer.png

Module 2: requester ranking

Problem and limitations

Crowdsourcing platforms such as Amazon Mechanical Turk, Clickworker, and Microworkers fail to address all aspects of reputation and ranking between workers and requesters. Workers cannot easily search for and find requesters and tasks they are interested in, and requesters cannot easily be matched with high-quality workers who are proficient in their specific tasks.

Module preview and system details

Currently, Boomerang solves one aspect of the problem by introducing "a reputation system that differs from traditional rate-and-leave systems by 'boomeranging' the accuracy of the rating decision back to directly impact the user. This feedback loop means that giving someone a high rating increases the likelihood of working with that individual again, while giving a low rating reduces that likelihood." Boomerang thus effectively matches high-quality workers with high-quality requesters (and vice versa).

This module/feature is simply Boomerang, taking into account only general requester quality for task feed. Task availability will still be dynamically ranked.

Module 3: wage ranking

Problem and limitations

At times, workers are interested in maximizing their wage, above all other inputs. Workers should be able to access a feed that ranks tasks by wage. Task availability will still be dynamically ranked.

Module preview and system details

We'll certainly have to experiment and design how to determine estimated wage for a task, keeping in mind that different workers are more/less proficient in different tasks. Here are some ideas:

  • As Professor Bernstein earlier suggested, calculate average wage based on the HITs during the prototype tasks. Have a condition that requires workers to be focused on this task, and trust that they will continuously work on this task from start to submit.
  • Have a running timer for when workers are working on a task, and have them start/stop the timer to indicate during what periods they are actually working on the task. Take into account only times within the first standard deviation when computing the estimated wage.
  • Make the estimated wage even more specific to the worker by basing the estimation on workers who have completed similar tasks to the ones that the worker did. This will, at least to an extent, ensure that the workers who contribute to the calculation of the "estimated wage" (in this case, specific to each worker) are proficient in the same areas as is the worker.

Module 4: discovery ranking

Problem and limitations

Workers want to discover new tasks, either because they are bored with old ones or are interested in learning something new. Therefore, workers should be able to discover new tasks. Task availability will still be dynamically ranked.

Module preview and system details

In this feed, rank tasks based on how "new" the task is to the worker. After categorizing the different tasks, we can create an index system for each category that maps how similar a certain category is to other categories (e.g. 0.99 as essentially the same category and 0.01 as completely different categories). Classify the different categories that the tasks that the worker has done fall in, rank them based on how many times the worker has completed a task in a given category, and display the tasks in the discovery feed as a "flip-flop" of those tasks that the worker has done.

For example, if a worker has done tasks in categories (1) "image tagging" and (2) "transcribing," with frequencies in that order, in the discovery feed, tasks will appear in such an order that the category most distant from "image tagging" will appear at the top and the category most distant from "transcribing" right below it.

Milestone contributors

Slack usernames of all who helped create this wiki page submission: @shreygupta98