Difference between revisions of "Crowdresearch:WinterMilestone 3 Carpe Noctem - Reputation"

From crowdresearch
Jump to: navigation, search
(Worker Home Page)
Line 22: Line 22:
 
* After they enter keywords in the search bar, they can sort by Highest Reward, Deadline (Latest, Earliest), Date Posted (Latest, Earliest), etc
 
* After they enter keywords in the search bar, they can sort by Highest Reward, Deadline (Latest, Earliest), Date Posted (Latest, Earliest), etc
  
[[File:Worker_Homepage.jpg | thumb |800px |center | alt=MTurk Sandbox Dashboard | My MTurk Sandbox Dashboard]]
+
[[File:Worker_Homepage.jpg | thumb |400px |left | alt=Worker Homepage | Worker Homepage]]
 +
 
 +
[[File:Worker_HIT_Page.jpg | thumb |400px |right | alt=Worker HIT Page | Worker HIT Page]]
 +
 
 +
[[File:Worker_Search_Results.jpg | thumb |400px |center | alt=Worker Search Results | Worker Search Results]]
  
 
=== Requester’s Homepage ===
 
=== Requester’s Homepage ===

Revision as of 21:05, 31 January 2016

To improve reputation system beyond Boomerang, we can look into some more creative ways of calculating and utilizing the reputation. A successful reputation system should more or less accurately reflect the quality of work and areas of expertise of each worker. It should reflect the trustworthiness and quality of requesters. Such reputation system should also be resistant to hackers who may try find shortcuts to rack up reputation.

Worker Reputation

First, let’s focus on reputation for workers. Due to a wide range of types of work, using a single reputation score may not accurately represent a worker’s work ethics, quality of work, areas of expertise, level of experiences etc.

Instead, we will break down the reputation into subcategories. The subcategories include whether work finishes tasks on time (timeliness), work quality, communication skill/style, work ethics, areas of expertise.

Each category has its own metrics. For example, > work quality would be based on the satisfaction ratings from requesters. Areas of expertise would be computed based on worker task history, task size, level of difficulty and so on. > if workers have been receiving good scores on web development consistently, the task, he would be rated high in the area of web development. > Finally, a comprehensive reputation score will be calculated based on [timeliness, work quality, communication skills, work ethics] as well as areas of expertise [web dev, machine learning, UX design, …].

This score is used as an overall measurement of worker’s reputation for humans but when matching requesters with workers, the computer will use the specific subcategories to recommend workers. The specifics of how each subcategory reputation is computed is too long and detailed to put it here but can use StackOverflow reputation as a reference.

Wireframes

Worker Home Page

  • can search through search bar
  • can scroll down to look at different categories eg. Popular HITs, Newest HITs etc.
  • Worker experience ratings are given
  • Average worker compensation is written in the description
  • After they enter keywords in the search bar, they can sort by Highest Reward, Deadline (Latest, Earliest), Date Posted (Latest, Earliest), etc
Worker Homepage
Worker Homepage
Worker HIT Page
Worker HIT Page
Worker Search Results
Worker Search Results

Requester’s Homepage

  • they can search for existing templates or scroll down to see popular templates, or templates that is voted with best user experience
  • they can select good templates from “Suggested Templates” section or search on their own, and Copy the templates and change the description

Requester Reputation

Similar to worker reputation, requester reputation can be calculated based on a variety of criterias. They are [task quality, response time, payment timeliness, easy-to-work]. Like worker reputation, the comprehensive reputation is for humans, while the actually recommendation is based on subcategories.

For Newcomers

All users start with one reputation point, and reputation can never drop below 1. To help new users jump start their journey, tutorials and mentors will be provided on demand. Reputation can go to negative if workers receive many low score (<4 on scale of 10). This would make sure newcomers are not at a natural disadvantage comparatively.

How Workers, Requesters Find Each Other

Subcategory scores will be used by computer algorithm to recommend workers to requesters and vice versa. Request



Milestone Contributors

  • Michelle Chan : @michellechan
  • Manoj Pandey : @manojpandey
  • Lucas Qiu  : @lucasq
  • Mengnan Wang : @mengnan