WinterMilestone 3 BPHC ReputationIdea : Referral Network for Workers

From crowdresearch
Revision as of 09:40, 31 January 2016 by Adityanadimpalli (Talk | contribs)

Jump to: navigation, search

One of the major needs identified in existing platforms for crowd-sourcing is the ability of the platform to effectively match capable workers with suitable jobs or HITs (Human Intelligence Tasks). The Stanford Crowd Research Team has thus far developed their Daemo Platform, which makes a significant step in meeting this requirement through the Boomerang Ranking System.

We propose a solution consisting of three design aspects, that attempt to solve needs related to - worker reputation, matching requester and worker requirements and the high bar for entry for newcomers.

Referral Network for Workers

We propose a referral based system that deals with the following reputation and relevant work issues:

  • A problem with a system that prefers assigning tasks to workers with high reputation, is that new workers may find it difficult to find good HITs. Requesters would tend to allot jobs to the same set of highly rated users. Similarly, existing workers would tend to take up HITs from the same set of requesters with high ratings.
  • Workers rely on external forums (TurkerNation, Reddit etc) to find good HITs from reliable requesters. Clearly any future crowd-sourcing platform should reduce this burden on the worker by allowing dedicated workers to easily share good HITs and help fellow workers to find relevant work .


The employee referral network in the corporate world is known to be a cost and time effective method of recruitment that produces high quality candidates. 92% of the participants in the Global Employee Referral Index 2013 Survey stated that referrals were a top source of recruitment. We thought about adapting the referral system to our crowd sourcing platform design. This would enable workers to recommend or “refer” other workers (existing or new) for good HITs.

Implementation of a Referral Network for Crowdsourcing

We illustrate the use of the referral system through the following example.

1. Requester R has a HIT to post on the crowd-sourcing platform. R can view highly rated workers using Boomerang and make the HIT visible to them (or post the HIT publicly for all to see).

2. Worker W notices the HIT posted by R and has the following options:

  • Accept the HIT.
  • Share the HIT with fellow workers. In the referral based scheme, this can be done in the following ways:
1. W refers workers he or she knows to R.
2. W can publicly offer to refer anyone who is interested in HIT. This is similar to how referrals are shared across social networks like Facebook, Twitter. This referral sharing network can be integrated into the crowd-sourcing platform, removing the need for multiple external forums.
3 The quality of referrals is incentivized in a number of ways, such as including a measure of good referrals in the scores used to rank workers and requesters. A separate index could also be used for referrals or requesters may reward good referrals with bonus payments.

Benefits of Referral Network

A HIT initially becomes visible to highly rated workers who have worked with that particular requester in the past. The referral network enables sharing of HITs with other (existing and new) workers who would otherwise have been the last in line to see the HIT. This is especially useful when a requester posts a large number of HITs, which can be shared quickly among workers.

Assisting Requesters in Finding the Right Workers at the Right Time

An often cited challenge for workers is that high paying HITs are quickly completed, and if the worker is not available when HITs are posted, the worker may miss out on lucrative work. A worker panelist mentioned that instances such as, skipping lunch to work on a high paying HIT, are not uncommon. This issue is compounded by the fact that workers are available globally, and tasks posted globally.

To resolve this issue, we propose a simple feature that lets a requester post work at the optimal time to target the requester's most highly rated workers:

Although Boomerang attempts to resolve this issue by creating a time staggered access to work, based on reputation; as the number of workers grow, this may not be enough to resolve this issue.

Our system overcomes this by allowing workers to opt into a system where the platform keeps track of the time range when they are usually available and working on tasks. A requester is presented with a suggestion mentioning the optimal times to post the task on the platform and the availability percentage of workers the requester has rated highly.


Suggestion: 73% of workers you have rated as 'Good' are usually available between 9:00 am and 3:00 pm. 
Would you like us to post the task for you in those hours?

This has the benefit of letting requesters get the best workers (in addition to the reputation based access) and also allows highly rated workers to not miss out of lucrative work.


This would require workers to allow the platform to log usage timings which is a potential privacy issue. In addition, if most workers are from a certain timezone, it could skew the tasks in favor of one country.

Reputation of Newcomers

Allowing newcomers to quickly join a system, where the rating assigned by a requester can determine if a worker gets work, is of prime importance. Boomerang's approach to this is to assign a new worker with the global average rating and a time decay component that opens up tasks as time progresses. While this is better than a fixed value, it still may slow down a new worker's road to becoming a full fledged worker.

In addition to the referral scheme we suggested above, we propose a simple technique to allow a new worker's ability to be evaluated:

- Similar to how many freelance work websites offer skill tests, a requester who has posted tasks multiple times 
  will be asked to offer a snippet of  an old completed task to be used as a test.
- New workers seeking access to a task offered by that requester complete this sample task.
- New worker's answers are correlated with answers given by a worker that was judged to be good by the requester.
- If there is high correlation between the answers, the new worker may be allowed to perform the latest task.

This mechanism can be extended to multiple requesters, by making a new worker take a series of small task 'tests', which can be used to decide if a worker gets first access to a task, in lieu of requester's rating. This would serve as a better solution to the problem of newcomers having no requester assigned rating.


This will work only if the task is defined such that it is possible to evaluate similarity of answers by different people (eg. image labeling, reading text in an image). This system will work best only if the requester posts similar work (boomerang works on the assumption that a requester will prefer a worker he/she rated well earlier).

Milestone Contributors

@adityanadimpalli , @sreenihit