Crowdresearch:WinterMilestone 3 Carpe Noctem - Reputation

From crowdresearch
Jump to: navigation, search

We explore ways to building an enabling platform so that anyone with any experience level can start from here and build a interesting career instead of being just a task rabbit.

To improve reputation system beyond Boomerang, we look into some creative ways of calculating and utilizing the reputation, and its effect on pricing, future relationships and career [yes, workers can have a career]. A successful reputation system should more or less accurately reflect the quality of work and areas of expertise of each worker. It should reflect the trustworthiness and other qualities of requesters. Such reputation system should help both sides to develop trust and relationships and be resistant to reputation hacking.

Worker Reputation

Due to a wide range of types of work, using a single reputation score may not accurately represent a worker’s work ethics, quality of work, areas of expertise, level of experiences etc. Instead, we will break down the reputation into subcategory reputation. The are two main types of categories, general characteristic and areas of expertise. And each subcategory has its own metrics to compute the reputation score for that subcategory.

General characteristics consist of the universal qualities of a worker, e.g. whether worker finishes tasks on time (timeliness), general requester satisfaction, work ethics, communication skill/style, work ethics etc. For a specific subcategory, e.g. timeliness, we look at all the tasks completed by this worker and the percentage of them within deadlines.

Reputation for areas of expertise will be computed based on the number of tasks this worker completed in that area, the difficulty level of the tasks, the size/scope of the tasks, what role the user take and so on. These metrics will results in a reputation score for that particular task type. For example, if workers have been receiving good scores on web development consistently, he would be rated high in the area of web development.

Finally, a comprehensive reputation score can be provided based on general worker characteristics as well as areas of expertise. This score is used as an overall reputation for humans but when recommending requesters with workers, the computer will use the subcategory reputation in its algorithm.

Requester Reputation

Similar to worker reputation, requester reputation will be divided into subcategories like task quality, response time, payment timeliness, easy-to-work-with score etc. Like worker reputation, a comprehensive reputation is generated for humans, while the actually recommendation is based on subcategory reputation.

Peer Evaluation

According to studies [Shaw, Horton and Chen, CSCW ’11], the social pressure from others can make workers more cautious and motivated about his work quality. We thus inform workers that they will be evaluated by their fellow collaborators and requesters on the same tasks during the work process. The feedback will be provided (with anonymous option) to the workers during the process so they can adjust as they go instead of waiting to be corrected after mistakes have caused damage. Consistent bad feedback, i.e. no improvement after feedback is provided, will harm workers reputation score.

Newcomers: Mentorship Reputation

The Basics: all users start with one reputation point. Reputation can go to negative if workers receive many low score (<4 on scale of 10). This aims to avoid making newcomers with low reputation standing at a natural disadvantage.

To help new users jump start their journey, tutorials and mentors can be provided on demand. Experience workers can mentor new users to earn reputation. And new users will have a guided experience into the platform. This reputation reflects the helpfulness and impact of a worker outside his own work. We hope this will help build a virtuous cycle to onboard new workers.

How Workers, Requesters Find Each Other

Subcategory scores will be used by computer algorithm to recommend workers to requesters and vice versa. As described above, our recommendation system will look at the requirements of the requester AND worker to recommend the most likely candidate for each other.

Building on top of this system, we want to add a few social features into recommendation to facilitate relationship building between workers and requesters in the long run.

Requesters can directly recommend workers to other requesters. This is similar to off-line referral where companies find talents. Utilize social networks has been one of the most reliable ways to find good talents in the off-line world and we want to have that in this online marketplace as well.

Workers can also follow requesters. There are many times when you had a really good experience working with someone, you want to work with him again, regardless of his reputation scores. If worker follows a certain requester, the tasks from that requesters will be ranked high in the worker's task recommendation feed.

Impact on Pricing Model

A note on the impact of reputation on labor pricing. To mitigate or avoid the cases where things become a complete failure and requesters unwilling to pay, we need a good contact between both parties. The traditional fixed pre-set price which does not foresee the changes in the work process and reflect the changes and work quality accurately and fairly. Our dark horse idea explores the possibility to use real-time bidding for both workers and tasks. This has some consequences that resonates 'winner takes it all' situation.

The compromise we have is a happy middle ground, a negotiation model. Our algorithm will provide a suggested price range based on market supply and demand and similar tasks in history. The requester will then set a maximum budget he can pay for this task. The requester then can select workers from recommendation feed or other channels. Each time the requester contact a worker, a auto-generated market price for that worker will be generated and provided to both, mainly based on the task characteristics and worker’s subcategory reputations. Two parties can negotiate a range of payment. The final payment is dependant on several ratings, requester-to-worker rating, worker self-evaluation, worker peer evaluation(if applicable). If worker gets good ratings from all, which is the best case scenario, he will receive the maximum payment agreed beforehand. Otherwise, payment will be deductible based on ratings.

The reasons for using all three ratings are due to psychological reasons/biases, but the goal is to ensure a fair and accurate payment while making both requester and worker happy.

Education, Career Track, Meaning and Purpose

Most people want their work to be challenging but not over challenging, meaningful and also rewarding. We designed a couple paths for that.

For the beginner folks, our platform give instant feedback and positive reinforcement for every successful tasks. We incorporate gamification into task workflow and make workers feel rewarded for every small steps they take. The platform may even work with educational organizations to provide certifications. This helps them boost their confidence and motivation.

As workers work get more experienced, our recommendation system will recommend bigger, more complex projects to their experience level to keep them challenged, engaged and continue improving their skills instead of staggering doing similar tasks over and over again. The reputation can be used in the real world for job application like what Github, StackOverflow does currently.

The goal is to enable people from all experience levels, from all around the world, from any background and social class, to be able to start learning and rack up experience and build a career on the platform.

Team Members

  • Michelle Chan : @michellechan
  • Manoj Pandey : @manojpandey
  • Lucas Qiu  : @lucasq
  • Mengnan Wang : @mengnan