Faceted Reputation System

From crowdresearch
Jump to: navigation, search

Current 'Flat' Approaches

The Reputation System is integral to supporting trust in and of Daemo by allowing workers to show off their successes and allowing Requesters to utilize a workers past history (among other metrics) in order to determine if this worker is a good choice for the task.

To date, most reputation systems distill to a single flat metric, 4 out of 5 stars, a rating of A,B,C or D. While in many instances this distilling is beneficial to making clear binary choices, it provides a number of challenges in the context of crowd working. 1. Inhibits workers from testing their skills in new areas for fear of compromising their reputation score. 2. Provides limited complexity to allow requesters to manage preferences on a per task basis. 3. The combined effect of the two above points limits the growth and greatest economic benefit of the system participants.

A Different Question

A faceted reputation system that is dynamically generated on a per task basis. The idea is to provide not answer the question "Is this worker good or bad?" but instead to answer "Can this worker do what I am asking them to do?"

Formula

The features are flexible (numerical, alphabetic, multivariate profile etc.) but the foundation is such that the 'score' will be weighted based on the workers history with this specific type and level of task. The formula would also include a baseline of overall reputation score across all work history as well as 'constant' of skill/level, traits etc. The coefficients of each variable will change depending on the workers history in this specific task area.

For example, if a worker has a long history of successful completion of photo shop tasks and they are applying for a photoshop assignment, then the 'score' will rely heavily on history within this area. If this same worker recently starting programming in python and was looking to take on a task in this area, the reputation would include the overall reputation score of the worker, since the worker would not have history in this task area, they would have the option to up the coefficient by taking a skills test. Leveling would replace reputation to solve cold-start issues. In this case workers would not be penalized in areas of exhibited expertise by venturing in to new domains. As the workers gains more repetitions in this task category the score would shift from skills to experience.

Future

The solution itself can scale to include more advanced machine learning, task matching, profile etc. The solution can grow in complexity.

The simple MVP solution though is to provide a dynamic and appropriate reputation metric that is truthful and supports expansion and professional development.