Milestone 3 Team Innovation 2 TrustIdea 1: Robust User Rating and Profile System

From crowdresearch
Jump to: navigation, search

One of the most oft-mentioned problems with MTurk is that neither workers nor requesters trust the people they're working with. One way to solve this problem is by giving those in both groups more information about the people they might potentially work with, so they can choose to work with those who seem like a good match. This information could be presented on profile pages for reference by other users, and it could also be used to build a searchable database of information that users can filter by criteria of their choice. A worker can look for requesters who posts tasks that receive a minimum average quality rating; a requester can search for workers who have a certain skill set.

The goals of a robust rating system are a) to collect relevant information about users, and b) to present this information in a clear, understandable way that helps users accurately judge the qualities that are important to them in a worker or requester.

As in the current MTurk design, some information about users would be collected automatically by the system. For example, it might record how long a worker has been using the site, the number of tasks they've completed, the number of work hours they've logged, and what categories of work they perform. For a requester, it might record how many tasks they've posted, how many workers they've hired, what types of work they post, and how long users take to complete their tasks.

Users could also choose to present information about themselves, to make themselves more attractive to other users they think would work well with them. A worker could list copywriting and image tagging in her skills list (even if she hasn't worked on any copywriting tasks yet), and include samples of copywriting work as proof. A requester might list transcription and surveys in his work categories section (even if he hasn't posted any work yet), and post standard guidelines for his transcription work, so workers can easily access and review his policies without needing to go through his tasks to get there.

Another key component of a robust rating system is allowing users to give intentional feedback on each other's performance, and to allow other users to view this feedback. It's important that this feedback not be limited to system-calculated work rejection and approval rates. Requesters should be able to rate users' skill in performing certain types of work, and workers should be able to rate the quality of requesters' tasks.

The system could even be designed to give users slightly different levels of influence on each other's ratings depending on their own ratings and achievements. This would reward users both for longevity and for earning good ratings from other users, but wouldn't unfairly penalize new users, who would still be able to give ratings. For instance, if a worker is very highly rated in transcription and has completed 5000+ transcription tasks, their rating of the quality of transcription tasks might have slightly more weight than the rating of a worker who has completed only 10 transcription tasks. A requester who has posted 2,000 writing tasks and whose tasks receive high ratings would be more influential in rating completed work involving the writing skill.

In a useful feedback system, certain ratings will be present and viewable for all users, to provide a basis for comparison. However, a well-designed profile system also allows users to customize their profiles, including for privacy reasons. A system like the one described here would potentially allow users to preserve their anonymity if they choose, but to share information they feel comfortable sharing. Certain profile fields should have customizable privacy settings, allowing users to, for instance, make the information available only upon request, visible only to other registered users of the site, or visible to the public.

With a system like this, users can choose to work with those whose profiles contain information that encourages trust, and they also contribute rating information that allows others to make their own trust-related judgements.