From crowdresearch
Jump to: navigation, search

Boomerang Reputation System

You may see a Boomerang qualification score on tasks posted through the early beta of the Daemo crowdsourcing platform. Boomerang scores are used to keep track of how workers have done on past tasks, and give great workers early access to new tasks. New tasks are available to workers with the highest Boomerang score for that task first, then made available to other workers over time if the high-scoring workers don't take it. So, the workers who did the best previously get early access. If a task isn't available to you yet, wait a bit and it may become available!

How does it work?

Boomerang gives all workers a score as they work on the task. Scores typically vary from 100 to 300, and yours probably started around 200. As you do the task, your score will go up or down depending on your HIT submissions. Your score is based on feedback that the requester gives. There are three possibilities: (1) the requesters may rate your work directly, or (2) the requester may ask other workers to rate your work via peer review, or (3) the requester might rate your work automatically based on automatically-observable signals like comparing a subset to known answers. A few examples:

  • If you are labeling images, your score may be based on the % of pre-labeled images that you matched
  • If you are writing stories, your score may be based on how many other workers liked your story
  • If the requester is using worker peer review, other workers compare your submission to other workers' submissions, and we calculate a score using TrueSkill, which is just like the Elo approach that is used to rank chess players.

You get a new score for each new HIT type you do, as well as for each new requester. We average across HIT types and requesters to initialize your score, however. (So, if the first requester you work with rates you highly, your Boomerang score will start high for other requesters too. Likewise, if you do well on one task for a requester, you'll start with a high score for the next task from that requester too.)

What does the qualification score mean?

Each HIT requires a minimum Boomerang rating for the workers to be able to accept it. When posted, the minimum rating is 300. After that it drops as needed. Eventually it may drop to 100, which means that everyone is allowed to work on the task. For every HIT, you will also be assigned a Boomerang qualification score based on how well you performed on the previous HITs, and it will be updated as you submit new work.

How do I change my score?

Wait until a Boomerang task becomes available to you, and then do it well! If it goes well, Boomerang will automatically raise your score, and give you earlier access to the next task. If you're not sure how the requester is determining their scores...each task may have a different scoring function, and you're welcome to reach out to the requester to find out what it is if the task doesn't say.

For more details check out the full paper here