WinterMilestone 3 westcoastsfcr ReputationIdea: Badges and Categories

From crowdresearch
Jump to: navigation, search


How do workers find relevant work, and requesters find ideal workers?

Although an interest in the subject matter of the HITs may play a factor in the HITs that workers choose, their primary motivation for using Mechanical Turk is money, so the HITs that they choose will reflect that motivation. Workers constantly ask themselves "Are these HITs worth my time?".

Workers look for HITs that pay well and that they can quickly finish, as this tends to maximize the amount of payment they will receive. But each worker is different; they might finish certain tasks faster than other tasks, and what type of task this is varies from worker to worker. The question is then, how can workers find work relevant to their motivations. Currently Amazon's Mechanical Turk displays there available HITs all in one section. The worker can filter by the amount that the HIT pays or whether they are qualified to do the HITs. There is no filter that allows you to filter by for a specific type of job. There's also search function, but it's not entirely clear what the worker can search for, especially to a newcomer. I propose a solution where categorization of the type of task is prominently displayed on the worker interface. There should be some functionality that allows you to filter for the type of task the worker would like to do i.e. surveys, categorization, image tagging, etc. They offer these types of tasks as templates on the Amazon Mechanical Turk's requester site, why not display them on the worker site as well. This way workers can find tasks most relevant to their interests on Mechanical Turk, making money. Workers know what types of tasks they can do that will optimize the amount of money they make.

Additionally, if we employ this sort of categorization, workers can then be classified by the type of work that they do. Workers who do well in a certain category can earn badge or some kind of qualification that speaks to their ability in those particular type of tasks. This way, requesters who want a really high standard of quality in the HITs they put out can require people to have a badge in the particular category that the task falls under. Currently, there's a masters qualification on Mechanical Turk that requesters can use to ideally obtain higher quality results from workers. This looks at workers' performance across all different types of tasks, but does not say anything about any specific type of task. And since workers perform differently across different types of tasks, we can't say how well the worker will perform on that specific task. And ideally, results will come quicker since the workers will be qualified for that specific type of task and can work quickly on those types of tasks.

The main goal of workers on Mechanical Turk is to make money. By allowing workers to easily search for tasks that they are good and quick at, we allow them to more easily accomplish this goal. And what follows through that categorization, regulated with badges, will accomplish requesters' goals to obtain high quality results, quickly.


Categorization of Tasks

Categorization Storyboard.jpg


Badges Storyboard.jpg

What are the goals of the design?

The goal of this design is to:

  • allow workers to find HITs that are more suitable their abilities
  • simplify the HIT searching process for workers
  • allow workers to maximize their earnings
  • allow requesters to obtain high quality results
  • allow requesters to find workers who work quickly while still producing good quality results

Which aspects of your design reflect each goal? How does your design solution addresses the users' needs?

Allow Workers to Find HITs Suitable to their Abilities

Allowing workers to filter or categorize HITs by the type of task, allows them to pick tasks that are more suitable for themselves (i.e. can complete quickly).

Simplify the HIT Searching Process for Workers

Workers employ a scripts and browse forums in order to find HITs that will pay well. Categorization and greater organization of the website will allow workers to better search HITs that maximize their earnings.

Allow Workers to Maximize their Earnings

Workers can ideally earn money in a shorter amount of time due to the categorization of tasks, since they can focus on types of tasks they are good and quick at.

Allow Requesters to Obtain High Quality Results

By apply a qualification (badge) based on the task type, we no longer have this generalization of abilities through the Masters qualification. Requesters will have less to worry about whether or not the worker will satisfactorily complete the HIT or not. Whereas before, workers might have been good at some tasks, but not others.

Allow Requesters to Obtain Results Quickly

Since workers can search by task type, they will pick ones that will maximize the amount of profit they can make per time spent. And so, they are likely to take on task types that they are exceptionally good and quick at. Requesters are then likely to get results quicker, as a result of badges.

Ideas/Current Questions/Potential Problems (Things I didn't have time to think about this week)

  • How should the workers be allowed to look at these categories?
 - Filter 
 - Tabs for each task type 
 - Customization of main page 
 - Navigation bar offering the option to search by task type 
  • What will the interface for the categories look like? How can we make it easy for the workers?
  • Main HIT page defaults to HITs that workers have a badge for
  • At what point do workers get a badge? After performing certain amount of HITs with a certain approval rating?
  • How does this system affect newcomers? And how can we make it easier for them to join the market?
  • Will the badge system make it difficult for workers to switch tasks they want to focus on?
  • Do workers generally do a variety of tasks or stick to one task?
  • Will it be difficult for requesters to classify their task? (i.e. will there be many tasks in the other section)
  • This may be detrimental to tasks that fall under the "Other" section, which may cause improper classification of task.
  • This does not take care of the fact that workers still base a lot what HITs they take on the requester's reputation (found on forums like TurkerNation) and whether or not the worker has had good experiences with them (i.e. responsive, fair)and their approval rating.
  • Like the Master's qualification, requesters can abuse it (i.e. requiring the badge when it is not needed for the task)