Customized Boomerang Team biubiubiu

From crowdresearch
Jump to: navigation, search


Brief introduction of the system

Customized Boomerang is a reputation system that extends original Boomerang to a broader version. Its core idea is to rank tasks according to the Total Points (TP) of a task (TP of even the same task can be different according to different boomerangs):

Core.png where


w (weight) reflects how important a worker thinks of an aspect of task feed, A shows how much a worker likes/dislikes the actual property of the task in that aspect and n is the number of aspects that a worker evaluates.

A is determined by two factors:

  • The explicit signals in the like/dislike lists given by a worker.
  • The implicit signals in the ratings of tasks given by a worker.

w is

  • determined by how a worker values this aspect/area. (most important, 2nd important, etc.)
  • (slightly) adjusted by the system after analyzing related user data.

How is the system solving critical problems

  • Current mainstream crowdsourcing platforms provide different task feed systems for workers. These systems, to some extent, satisfy workers with some benefits. However, none of them is able to account for various incentives of workers so that workers still have trouble in finding wanted tasks. We propose Customized Boomerang to meet diverse requirements of workers. In this system, workers have the ability to customize their own Boomerang to find wanted tasks easily. A worker can have several different customized Boomerangs in his account but can only have one active boomerang at a time. To help new workers get over the common "cold start", a new worker can preset his/her like/dislike lists of properties in the aspects that matter to him/her. In this case, even a new worker can have a satisfying task feed if he/she spends some time customizing his/her individual Boomerang. Since it is completely possible that a worker cannot decide clearly what he/she likes/dislikes or how much he/she likes/dislikes a certain property, every time when they rate a task workers can choose to allow (or not allow) the system to improve their like/dislike lists using data extracted from their ratings , which(whether to rate) totally depends on workers' willingness.
  • We expect the system to encourage multi-mode for workers, e.g. a worker can choose to focus on learning or making money using customized Boomerang according to the needs. Additionally, recommend systems used in mainstream crowdsourcing platforms can feed workers with tons of similar tasks, which drives workers bored about that quickly and nothing can be done to change that. Customized Boomerang solves this problem by providing multi-mode for workers to give workers the ability to change recommended task types.
  • Current task feed systems are not flexible to be improved since their mechanisms of recommendation are fixed at the beginning. Customized Boomerang is always open to new data that can be used to improve task feed for workers. This feature is shown in the unlimited aspects offered to workers for evaluation, thus we give this system unlimited potential to get better.. Once there's a new kind of data on the platform which some workers think it usable in refining their task feed, they can simply add that into the algorithm by deciding how important the data is to themselves so that when calculating the TP of a task, the system will take that new data into account.

Introducing features of the system

Below, we introduce the three main features to customize and refine a personal boomerang.

Feature 1: Customization - what are your main concerns?

Problem/Limitations

The core idea of Daemo's Boomerang reputation system is to use the ratings to directly influence the work quality you get (if you are a requester) and the ease of finding quality work (if you are a worker). So that requesters and workers can provide more genuine reviews and thus help reduce reputation inflation at online crowdsourcing platforms. However, the task feed in Daemo have some drawbacks: 1) the only rank reference is workers' ratings to the requester, so that the task feed cannot satisfy workers since there're other references like task types, payment or rejection rate, etc. 2) it is hard for new workers to get a satisfying task feed. 3) there will be more and more requesters having the same cumulative rating score. Workers will find their task feed becoming chaos that many requesters, regardless workers' past ratings to them, are at the same level. Hence, they will gradually lose trust on the platform for that the only feature Daemo has (or had) is gone and workers cannot get obviously differentiated task feed.

How to customize your personal boomerang?

At the first step, we attempt to acquire workers' preferences. First, we have workers choose and rank aspects that matter to them from the aspect pool:

Since ranking boils down (for workers) to 2 things: to find good requesters (on basis of payment, rejection, etc.) and to find good tasks (interesting, matches abilities, learning something, etc.) --- @shreygupta98 Shrey Gupta
We provide two main categories of aspects: Requester, which contains subjective evaluation of a requester(which is exactly what original Boomerang includes), payment,rejection rate, accept time, quality of communication etc. and Task, which contains task type, estimated working time, estimated hourly wage, etc. Workers can choose and rank subcategories of these two main categories.

There is no limitation of the number of aspects chosen. Workers can choose only one aspect or choose as many as they want. Then we use workers' ranked aspects to determine the value of w(weight).

Here we give an example about how to determine the value of w according to aspects' ranking.

A worker Lily chooses three aspects she thinks that matter to her and then ranks them, which are estimated hourly wage, subjective evaluation of a requester and rejection rate. We allocate different weight to these three aspects: w(payment)=8, w(evaluation)=5, w(rejection rate)=3 (according to the importance Lily puts in different aspects).

  • The problem that how to determine weight is still open to discussion.

(Optional) According to workers' choice, we have workers set their like/dislike lists of different aspects, which helps new workers since they don't have past ratings for the system to guess their preferences. And we use A to denote the preferences. Workers set different lists for aspects according their own choice. Here, we give a simple table to explain different aspects and their preferences.

Hourly Wage Like List >$6/h
Dislike List <$2/h
Subjective Evaluation White List Adam, Bob ....
Black List James, Kevin...
Rejection rate Like List <3%
Dislike List >10%

For example, regarding payment, Lily needs to set an expected estimated hourly wage, like higher than $6/ h. If the payment of a certain requester’s task doesn't meet the requirement, we determine that this A(wage)= -0.2. Otherwise, A(wage)=0.2. Regarding rejection rate, Lily sets less than 3%, however, this requester has a rejection rate of 8%, then the A(rejection rate) =0. Similarly, regarding subjective evaluation , this requester, we call him Adam, is in Lily’s whitelist (this happens either because Lily sets the requester in the like list or because Lily gave a good rating of the requester), then the A(evaluation) = 0.2.

  • There can be more levels of A. And it's still open to discussion how many levels should A be divided into.

Finally, we have the Total Point of this task. TP=(1-0.2)*8+(1+0.2)*5+(1+0)*3=15.4. As a result, we rank tasks fed to Lily according to TP of different tasks from high to low.

  • If you still have trouble understanding how this works, feel free to ask any kind of questions.

Feature 2: Refine your personal boomerang through ratings

We want to claim that whether to rate or not is totally up to users. But if you rate accurately, the system will provide better task feed to you.

Problem/Limitations

It's been proved that reputation inflation exists in traditional crowdsourcing platforms and original Boomerang on Daemo provides a solution to that: let users know that their past ratings will determine their future task feed. That is both an incentive and a warning to have users rate accurately and truly. We believe it's a really effective idea, though ratings in original Boomerang are limited only in Requesters/Workers. Here, the broader version of Boomerang, Customized Boomerang provides great freedom in ratings but still carries forward the idea behind the original Boomerang.

how ratings work in the task feed process

It's been mentioned that A is determined by two factors:1)The explicit signals in the like/dislike lists given by a worker (which is explained in Feature 1). 2)The implicit signals in the ratings of tasks given by a worker. Here's a rough view of different kinds of ratings in Customized Boomerang.

Overall Rating Use it to refine *1
Ignore this rating *2
Detailed Rating Use it to refine *3
Ignore this rating *4

Again, we uses worker Lily as an example in some cases. And here's what will happen in 4 kinds of cases: 1. An Overall Rating that helps refine personal Boomerang: When a worker finishes a task and feels comfortable about properties in every aspect he/she concerns, he/she can choose to give an overall rating and allow the system use this rating to refine his/her boomerang, the properties of the task in aspects which the worker chose and ranked will be extracted for use. If an extracted properties already exist in the worker's like/dislike list, the system upgrades/degrades that property; Otherwise, the property is added to like/dislike list according to the rating.

  • Example: Lily, whose boomerang is shown in the table in Feature 1, chooses to give an overall rating of 4/5 to the task she's done, which is a $5/h, 5min task posted by James with a rejection rate of 5%, and wants to use this rating to refine her task feed. The system extracts the needed properties: $5/h, James(requester's name), rejection rate of 5%. Since Lily's original like list of hourly wage is >$6/h and she gives a 4/5 to a $5/h task, the system adds >$5/h into her like list, but A(when wage >$5/h) < A(when wage >$6/h). Regarding subjective evaluation of Adam, since Adam already exists in the white list, the system will define A(posted by James) < A(posted by Kevin). As for Rejection rate, the system will add <5% into her like list but still, A(when rate<5%) < A(when rate<3%).

2. An Overall Rating that the worker tells the system to ignore: When a worker finishes a task and don't want to let this rating affect his/her future task, e.g. a worker got an excellent experience through working for a task whose type the worker isn't interested in at all, he/she just wants to express his/her feelings but still don't want to be fed by such tasks, the worker can choose to only give an overall rating that won't have effects on his/her future task feed.

  • Example: After finishing a task posted by Adam, Lily feels that this time the task is terribly authored. But she knows that Adam is a good requester and maybe this time something happens to him and she wants to work for him in the future. However, in original Boomerang, if she gives a bad rating, she'll not be fed with tasks posted by Adam first. Then she can choose to give an overall rating and tell the system to ignore since she just wants to express her feelings to this single task. In this case, the system will not use the rating for refinement.

3. Some detailed Ratings that helps refine personal Boomerang: When a worker is satisfied with some properties of the task, apathetic with some properties and annoyed by some properties, he/she can choose to give detailed ratings to express such mixed feelings.

  • Example: A worker Chan(male) sets his boomerang concerns in hourly wage, working time, requester's payment and subjective evaluation of a requester. And he completes a $20/h ,40min task posted by a requester Zhou(female). Chan feels it's good to earn $14 in 40min(hourly wage is good) but a 40min task is too tiring for him(working time is not fine). What's more, the requester's payment is generous but the task she posts is a bit unclear and has poor instructions. With such mixed feelings, Chan can choose to give detailed ratings and use that to help future task feed. Only give ratings to satisfying properties or only give ratings to annoying properties or give ratings to all properties in all aspects he concerns is all cool, it's up to him.

4. Some detailed Ratings that the worker tells the system to ignore: Such ratings happen when workers want to express their feelings of some specific properties but don't want this affect their future task feed.

  • Example: That worker Chan again completes a $50/h, 5min task posted by some requester. He feels that he met an angle just like you. So he has such strong desire to express his feelings to the requester's payment and to the hourly wage. However, Chan knows most tasks with such high payment are boring and require special skills which he doesn't have. Then if his like for such high hourly wage is used in future task feed, he'll have a bunch of tasks with high wage but he can't do. This is the case to give detailed ratings and don't use it for future task feed. Chan can give 10/5 (kidding) to requester's payment and the hourly wage to express his thanks and then go back to his normal tasks with the wish to meet some tasks like that again. Additionally, he can add that angle requester into his whitelist to "subscribe".

Feature 3: Multi-Mode for Workers

Problem/Limitations

Crowdsourcing platforms have various workers whose incentives and purposes can be distinctive. However, few platforms notice that and come up with proper ideas to account for such phenomenon. What's more, the working incentives of the same worker can vary according to the worker's mood, need or interest. Our Customized Boomerang provides multi-mode for workers to find wanted tasks whatever their purposes are.

Treat your Boomerang like a card set in Hearthstone

We're inspired by the system in the game Hearthstone, where users can have up to 9 card sets that can be edited/deleted to fight against different types of enemies. For the same worker, incentives can change among learning, making money and so on. Some task feed systems provide tasks according to workers' recent working history, which is effective if workers' purposes don't change. For example, a worker uses his skills in programming to make money while has great interest in learning how to take beautiful photos. Then most of the worker's working history will be related to programming types and very few are about photos (since programmers don't have spare time). In this case, it can be quite difficult for the worker to find tasks where he can learn to take photos.

Customized Boomerang gives this worker freedom to change between making money and learning. The only thing the worker needs to do is to customize two different personal boomerangs for making money and learning. The worker can set one of his boomerangs as a working-mode, where the worker's main concerns are task type, hourly wage, working time and something else. So this working-mode boomerang helps the worker find good programming tasks. After working for some time in programming, the worker can change his active boomerang from working-mode to learning-mode, where the worker's main concerns are task type, subjective evaluation of requesters (he can "subscribe" some famous photographer requesters when customizing learning-mode boomerang) and so on. This learning-mode boomerang helps the worker find tasks where he can learn about taking photos and let the worker forget about programming things since there will be no programming tasks on top of the task feed.

Improvements

Official boomerangs for workers

Since many users don't bother customizing their own boomerang and think it useless, Daemo can provide some official boomerangs for different purposes like making money, learning and etc. Through research, the platform itself decide concerned aspects but leave the like/dislike lists for users.

Customization only for high-level workers in the guild

To offer an extra incentive for workers to participate in the guild, only high-level workers have the authority to customize their boomerangs. For workers who haven't reached the level of customization but done several tasks, they can find customized boomerangs shared by high-level workers in the forum and make good use of them. For new workers, they can only choose to get tasks fed according to the official boomerangs.

Boomerang github

Since Daemo has a forum where various information can be found, the forum can include a forum where high-level workers share their own boomerang to other workers. Workers in qualified levels can view those shared boomerangs and vote for their favorites.

How the system works for a worker

We use an imagined worker to help you better understand the system.

  • A worker with coding skills wants to make money and learn some photography skills finds Daemo and begins his career on Daemo.
    1. 蛤蛤 begins his registration process before he completely understands mechanisms of the system. After completing his profile like he did in other crowdsourcing platforms, here's something he never met in other platforms:
      • 蛤蛤 is asked to choose what's his main incentive to work on the platform, options are: 1)making money. 2)learning. 3)for fun. Here, these three options are three kinds of official boomerangs whose concerned aspects (the n and w) are decided through research by the Crowd Research Collective. 蛤蛤 chooses 1)making money.
      • Assuming the making-money boomerang consists of Speciality, Hourly Wage and Evaluation for Requesters, 蛤蛤 selects "coding" tag among several skill tags offered by the platform for like/dislike list of Speciality, just like registering on Freelancer.
    2. At this point of time, 蛤蛤 just doesn't care about other stuff and begins to work without setting like/dislike lists of other two aspects. After finishing his first task, 蛤蛤 finds that:
      • There's a rating interface which has some words "Please express your true feelings about the task, and we can help you get more suitable tasks for you" and three options:1) Give an overall rating. 2) Give detailed ratings. 3) I don't want to rate.
        1. 蛤蛤 chooses 1) Give an overall rating. and gives a 4/5 rating on the task and select the choice "help me get better task feed using this rating". Then he finds that properties in the concerned aspects of the task: coding which belongs to Speciality, $20/h which belongs to Hourly Wage and 华莱士(requester's name) which belongs to the Evaluation for Requesters are extracted and added to 蛤蛤's like/dislike list. In the future task feed, A(coding), A(>=$20/h) and A(posted by 华莱士) are higher than other properties respectively in their aspects.
          • If 蛤蛤 doesn't select the choice "help me get better task feed using this rating". Then NOTHING changes for him.
        2. 蛤蛤 chooses 2) Give detailed ratings and gives 5/5 on Speciality, 4/5 on Hourly Wage and 5/5 on Evaluation for Requesters and select the choice "help me get better task feed using this rating". He'll find that A(coding)=A(posted by 华莱士)>A(>=$20/h), but of course the As have effect only in relative aspects.
          • If 蛤蛤 doesn't select the choice "help me get better task feed using this rating". Then NOTHING changes for him.
        3. If 蛤蛤 chooses 3) I don't want to rate, he can just start working on other tasks.
    3. After working for a while on the platform, 蛤蛤 is starting to understand how the system works. To refine his task feed and eliminate negative effects of wrong ratings, 蛤蛤 decides to customize his own like/dislike lists:
      • For Speciality, 蛤蛤 finds out he can do some simple photoshop and design tasks. So 蛤蛤 adds "design" to the like list of Speciality, but sets A(coding)>A(design) since he does better in coding. Once 蛤蛤 gave a 4/5 rating to a boring survey task just because he was well paid and since then he found many similarly boring survey tasks ranked relatively high, so he deletes "survey" in the like list of Speciality and adds it to dislike list. In the future, 蛤蛤 will find coding tasks first, design tasks second and then other types of tasks, followed by survey tasks.
      • For Evaluation for Requesters, 蛤蛤 deletes some Requesters' names because he no longer wants to work for them and adds some famous requesters(祖英,记者,长者) known from the Daemo forum to white list.
      • ...
    4. Then after quite a while, 蛤蛤 reaches a higher level in the guild and has the authority of using boomerangs shared by advanced workers in the forum (boomerang github section). Since shared boomerangs are categorized and ranked by votes, 蛤蛤 easily finds a customized making-money boomerang with over 500 votes, which fits 蛤蛤's concerns perfectly. Assuming the new boomerang 蛤蛤 uses has Speciality, Working Time, Hourly Wage, Evaluation for Requesters and Rejection Rate of Requesters, 蛤蛤 can use existing like/dislike lists of Speciality, Hourly Wage and Evaluation for Requesters and add new items to like/dislike lists of new aspects included manually or through ratings.
      • Besides, 蛤蛤 finds a boomerang which can feed him with photography-relating tasks. And now 蛤蛤 begins his learning career using the learning-boomerang in his spare time. The switch is really simple, 蛤蛤 just has to select one boomerang and sets it active so that TP of tasks are recalculated and reranked.
        • 蛤蛤 can also use the official learning boomerang at the beginning of his career on Daemo, but he didn't know this at that time.
    5. Through hard work and by seasoned skills, 蛤蛤 reaches a higher level in the guild where he has the authority to customize his own boomerangs. Happy Ending!Ending.jpg

Contributors

@xi: Xi Chen 陈曦
@juechi: Juechi Zhou 周珏池