Crowdresearch:WinterMilestone 3 Carpe Noctem - Reputation
To improve reputation system beyond Boomerang, we can look into some more creative ways of calculating and utilizing the reputation. A successful reputation system should more or less accurately reflect the quality of work and areas of expertise of each worker. It should reflect the trustworthiness and quality of requesters. Such reputation system should also be resistant to hackers who may try find shortcuts to rack up reputation.
First, let’s focus on reputation for workers. Due to a wide range of types of work, using a single reputation score may not accurately represent a worker’s work ethics, quality of work, areas of expertise, level of experiences etc.
Instead, we will break down the reputation into subcategories. The subcategories include whether work finishes tasks on time (timeliness), work quality, communication skill/style, work ethics, areas of expertise.
Each category has its own metrics. For example, > work quality would be based on the satisfaction ratings from requesters. Areas of expertise would be computed based on worker task history, task size, level of difficulty and so on. > if workers have been receiving good scores on web development consistently, the task, he would be rated high in the area of web development. > Finally, a comprehensive reputation score will be calculated based on [timeliness, work quality, communication skills, work ethics] as well as areas of expertise [web dev, machine learning, UX design, …].
This score is used as an overall measurement of worker’s reputation for humans but when matching requesters with workers, the computer will use the specific subcategories to recommend workers. The specifics of how each subcategory reputation is computed is too long and detailed to put it here but can use StackOverflow reputation as a reference.
Worker Home Page
- can search through search bar
- can scroll down to look at different categories eg. Popular HITs, Newest HITs etc.
- Worker experience ratings are given
- Average worker compensation is written in the description
- After they enter keywords in the search bar, they can sort by Highest Reward, Deadline (Latest, Earliest), Date Posted (Latest, Earliest), etc
- they can search for existing templates or scroll down to see popular templates, or templates that is voted with best user experience
- they can select good templates from “Suggested Templates” section or search on their own, and Copy the templates and change the description
Similar to worker reputation, requester reputation can be calculated based on a variety of criterias. They are [task quality, response time, payment timeliness, easy-to-work]. Like worker reputation, the comprehensive reputation is for humans, while the actually recommendation is based on subcategories.
All users start with one reputation point, and reputation can never drop below 1. To help new users jump start their journey, tutorials and mentors will be provided on demand. Reputation can go to negative if workers receive many low score (<4 on scale of 10). This would make sure newcomers are not at a natural disadvantage comparatively.
How Workers, Requesters Find Each Other
Subcategory scores will be used by computer algorithm to recommend workers to requesters and vice versa. Request
- Michelle Chan : @michellechan
- Manoj Pandey : @manojpandey
- Lucas Qiu : @lucasq
- Mengnan Wang : @mengnan