Crowdresearch:Open Governance Through Managed Services (Milestone 5)
The Challenge in building a reputation system
Developing reliable reputation system is always a open challenge that limit the effectiveness of reputations, a user enjoying a highly valued reputation would imply that the user has conducted good transactions in the past, however his present transactions are kept aside.
Reputation system are always based on past data, or the near past data, each day in human life is different where he encounters a series of events which can turn out to be the influencing agents to change behaviors, John did social work while he was in college but later on he performed a series of illegitimate tasks which has brought his reputation down.
Reputation has 3 variants - Bad / Good / No reputation(always neutral) / mixed reputation (perform illegitimate jobs, but still help the helpless)
How do you earn reputation in the real world?
Earning reputation is a continuous process and can be earned through delivering good services over a period of time.
The way we earn reputation in real world
- Helping others solve a complex task and earn reputation as a skilled worker
- Establish as a Business man & delivering social responsibility
- Doctors earn reputation by successfully executing a critical surgical operation
- Doing Illegal activity – adding fear factor brings negative reputation
- Being a research scholar
- By Working for reputed organizations
- Get political portfolio
- Graduates from reputed Institutes earn reputation by virtue of the stamp provided by the institute
- High Social class with income group
The reputation system may differ from demography to demography
How to improve reputation system beyond Boomerang?
IDEA 1 - Remove Reputation system - Provide equal Opportunity
Boomerang is trying to establish a relationship between a worker & requester using the following equation Low rated W = Low rated R Medium rated W = Medium rated R High rated W = High rated R
To set the context let's take an example, there are 10 workers and all the 10 workers are equally performing good, for a particular task, say task 1 if we disclose that the output of worker A is selected then for the next set of task the requester will run after A, where as in reality worker B is equally good as A and others. Similarly, let's say a requester had posted a task of designing a logo 100+ workers came up with different concept but the requester pick up one, but he is not making a public announcement that which logo he has picked up.
This in a way will provide equal opportunity to all workers & requester.
- Remove the concept of reputation system and have a neutral platform where workers come and work based on their choice. This reduces the human tendency to only look for task from requester with High rating, not everyone can have a high rating, what will happen to tasks which are from coming from requester having low rating, same applies to requester, a requester would always look for a reputed worker what will happen to other workers having low rating. Everybody wants a good boss, but if we don't get we don't quit jobs, we live with it get adjusted to it.
- However, there should be way to establish Open channels between requester & worker through a facilitator so that they can give feedback to each other and set each others expectation prior working on a task
IDEA 2 - Establish a career path with yearly review cycles
- Let the rating system be not on a individual task but a series of tasks performed by a worker over a period of time
- Conceptual model developed from real life - The platform should allow rewards to worker in the form of a promotion, Provide roles to workers, as happens in Industry, Jr. worker, Sr.worker, Lead worker based on past performance records, engagement with the system, task approval rate, minimal task rejections
- Peer to peer rating system throughout the year where each workers based on concrete reasoning & facts also get the chance to rate other workers on a quarterly basis.
- The first level reviewers can be workers/requester, basically a process of self-appraisal
- Second level - Requester
- Third level - Platform Owners
- Minimum engagement with Crowd sourcing platform : 1 year
- Task approval rate : 90% and above
- Task rejections: less than 10%
- Based on yearly review the system will Create groups – low performers, high performance workers/requester and over time the objective of the worker would be to reach the top of the ladder.
- For “short-lived” workers/requester who have conducted few transactions the ratings will not be valid
- Allow the workers/ requester to rate who has been in system for a certain period
- Enable Points based reputation
IDEA 3 - Body shopping revisited
Body shopping is the practice of consultancy firms recruiting information technology workers in order to contract their services out on a short-term basis.
Let the requester post the requirement to the consultancy firm, and then distribute the task to individual based on workers interest & skill
Forming independent body who would be dealing with workers issues relating to payment, unfair ratings, task rejections, fair wages and can settle issue by directly talking to requester, similarly they can also solve problems for requester in finding a reliable & skilled worker
IDEA 4 - Transitivity
Transitivity is a highly desired property of a trust metric.In situations where A trusts B and B trusts C, transitivity concerns the extent to which A trusts C. Without transitivity, trust metrics are unlikely to be used to reason about trust in more complex relationships.
The intuition behind transitivity follows everyday experience of 'friends of a friend' (FOAF), the foundation of social networks. However, the attempt to attribute exact formal semantics to transitivity reveals problems, related to the notion of a context. For example, defines conditions for the limited transitivity of trust, distinguishing between direct trust and referral trust. Similarly, shows that simple trust transitivity does not always hold, based on information on the Advogato model and, consequently, have proposed new trust metrics.
The simple, holistic approach to transitivity is characteristic to social networks (FOAF, Advogato). It follows everyday intuition and assumes that trust and trustworthiness apply to the whole person, regardless of the particular context. If one can be trusted as a friend, one can be also trusted to recommend or endorse another friend. Therefore, transitivity is semantically valid without any constraints. and is a natural consequence of this approach.
The more thorough approach distinguishes between different contexts of trust, and does not allow for transitivity between contexts that are semantically incompatible or inappropriate. Contextual approach may, for instance, distinguish between trust in a particular competence, trust in honesty, trust in the ability to formulate a valid opinion, or trust in the ability to consolidate another's opinions. Contextual approach is often used in trust-based service composition. The understanding that trust is contextual is a foundation of a collaborative filtering.