Summer Milestone 4 Submission - Reputation System Prototype Outline

From crowdresearch
Jump to: navigation, search


Reviewing the Reputation System presented by @neilthemathguy, we were inspired by the foundational need for the platform to have a fair/equitable and unique reputation system. Current platforms are vexed by reputation inflation, bi-lateral feedback systems that are untrustworthy and insensitive to cultural and gender bias and reputation systems that are unduly shaped by the social psychology of public/private feedback. In short there is plenty of opportunity to build a better mousetrap; provided the effort is shaped by the appropriate philosophical considerations. Hence, the take away from our most recent brainstorming session was: How do we create an environment where we need not be skeptical of qualitative feedback/opinion?

Based on the depth of literature and the guidance provided by Neil's model ( We seek to create trust in a reputation system that not only prompts self-awareness through accurate/representative metrics but is sensitive to the unique challenges of new workers, culture and gender bias. We specifically seek to minimize the biases and fraud of bi-lateral feedback and mitigate reciprocity induced reputation inflation through data and algorithmic matching.

Do we need to rapid prototype the system? It may not be a priority on the development schedule or the paper agenda, but it's development will require a cross cultural team of op-gov and technologists to develop and implement, a cross pollination that will further bind the community together. So, if rapid prototyping means testing the math, establishing the foundational methodology of feedback collection and weighting, sketching out the UI and testing the scalability of the platform into AI/Machine Learning, yes, rapid prototyping may be the ticket.

Foundational Impact

Foundational Effect Table

Workers Requesters
Trust "Better" at Cold Start and building the reputation of Newbies True/accurate representation of skill/expertise
Power Still accounts for feedback, but in a neutral/honest fashion. Balances or weights data that can be as transparent as needed. Using common data sets, the system can be positioned to encourage self awareness and learning that prompts empathy and improved communication. The power dynamic currently shaped by fear, could be replaced by influence and improvement.

Further Solutioning

So, to dig deeper into a reputation system for our platform, there are three silos (Data Sets/Collection, Methdology/Algorithm, Output/UI) that merit deeper investigation

Data Sets/Collection: Its not just what we collect but how we collect it and treat it; relative to the Algorithm and desired output. We will need to drill down in to the observable (Skils, education, certification, etc) and latent (expertise, abilities, timliness, etc) data sets upon which Skill Utility, Worker Quality and Marketplace Context rest. Are the fields required to fuel the algorithm(s) baked into the Data Model?

Methodology/Algorithm: A literature review has sourced three approaches, Hidden Markov Model (HMM), WorkerRank Model and Worker-Requestor Matching that address Skill Utility, Worker Quality and Marketplace Context. Thinking out loud, HMM and the WorkerRank model both aggregate multiple data points to create output/score/ranking. At first glance HMM can also generate output that can be used for a leveling/learning scorecard, while the WorkerRank approach shifts the burden from feedback to Milestone0. WorkerRank is very aware to the cold-start of new workers. Worker-Requestor Matching is the idea of @Acossette and is driven by the hypthoesis: Would workers/clients want to work with the other in the future based on current/past activity. In theory, the algorithms/methodologies could be used in parallel to generate output that spans the entirety of the trust/power relationship. Using this multi-variate approach is a clear differentiator in the market as is predicating the system on community/collective goals of trust and power.

Output/UI: How do we present the data to both requestors and workers alike



@acossette, @aditi.nath, @arichmondfuller, @betoavila, @claudiasaviaga, @jsilver, @m.kambal, @neilthemathguy, @niki_ab, @trygve