Summer Milestone 9 Reputation Systems research and exploration

From crowdresearch
Revision as of 08:09, 28 July 2015 by Aditimithal (Talk | contribs)

Jump to: navigation, search

Papers related to reputation systems and a summary of the pros and cons of the implementation they explore:


Reputation Inflation: Evidence from an Online Labor Market http://econweb.tamu.edu/common/files/workshops/Theory%20and%20Experimental%20Economics/2015_3_5_John_Horton.pdf William Dai and Nisha K.K. (Team Pumas)

Summary:

Pros:

Cons:



Hidden Factors and Hidden Topics: Understanding Rating Dimensions with Review Text [1] Please note that you can now get access to this article; Michael Bernstein has put a copy into our #reputation-sys channel on slack.

Juechi and @alfonsoxw

Summary:

Pros:

Cons:



Liquidity in Credit Networks: A Little Trust Goes a Long Way [2]

Tejas Sarma

Summary:

Pros:

Cons:



Inefficient Hiring in Entry-Level Labor Markets [3] @aricmondfuller

Summary:

Pros:

Cons:



Opinion Mining Using Econometrics: A Case Study on Reputation Systems [4] @rijul

Summary:

Pros:

Cons:


Rating Friends Without Making Enemies [5]Surabhi


Summary:

Pros:

Cons:


I rate you. You rate me. Should we do so publicly? [6] Nuwan@theekshana


Summary:

Pros:

Cons:


Reputation Transferability in Online Labor Markets https://slack-files.com/files-pri-safe/T03R8AQ66-F086A376G/mnsc_2e2015_2e2217.pdf?c=1438098109-ee4aa0e839b27e980c646a0d537f0be4a9fbc3a8 Claudia Flores Saviaga (@claudiasaviaga)


Summary: In online market places such as oDesk,AMTurk and TaskRabbit employers post tasks on which contractors work and deliver the product of their work online. In this places reputations systems play a important role in instilling trust and often are used by employers to predict the future performance of the worker. However, the tasks available in such places span across a variety of different categories, which leaves the employer with the issue of trying to guess how this reputation, in different task categories, is mapped to the category at hand. This paper analyzes how past, task specific reputation can be used to predict future performance on different types of task.

The paper explores the following questions:

1) Are reputations transferable across categories and predictive of future performance?

2) How can we estimate task affinity and use past information to best estimate expectations of future performance?

For answering the questions the authors use a set of real transactional oDesk data consisting of over a million real transactions across six different categories from the oDesk marketplace (Software Development, Web Development, Design and Multimedia, Writing, Administration, and Sales and Marketing). The data was collected between September 1 and September 21 of 2012.

It is important to remember that in the oDesk platform after a user completes the task, the employer supplies feedback scores integers between 0 and 5 in the following six fields:

- Availability

-Communication

-Cooperation

-Deadlines

-Quality

-Skills


The average of these scores divided by five represents the final rating.

The authors analyze the information using a binominal model and a multinomial model from the assumption that the latent qualities of the workers are static, in the sense that they assign equal weights to past ratings. Then they use a linear dynamical sytem (LDS), to take into account that as users complete more and more tasks, their more recent tasks are more predictive than their initial and older completed tasks. As expected due to its simplicity, the binomial model performs worse than the multinomial model, which in turn performs worse than the LDS. The paper concludes that reputation systems can be improved if the feedback scores of the participating users are adjusted to take into account the type of task that a worker is expected to complete (or has completed), as well as the user's past category-specific performance history.


Pros:The paper shows a clear approach for analizyng the correlations between different tasks categories and as a result,provide a more accurate estimate of a worker's performance in a new category. This can allow employers to make safer and better informed decisions about which workers to hire. The authors also suggest that their approach can also be used to recommend workers to apply for tasks that are seemingly out of their scope but for which these contractors are highly likely to provide successful outcomes.


Cons: It does not address the problem that new workers have when they first join this online markets and have no reputation at all.



A System for Scalable and Reliable Technical-Skill Testing in Online Labor Markets [7]- Vineet Sethia

Summary:

Pros:

Cons:


The Utility of Skills in Online Labor Markets [8] Surabhi @trygve

Summary:

Pros:

Cons:


On Assigning Implicit Reputation Scores in an Online Labor Marketplace [9] Jsilver


Summary:

Pros:

Cons:



Strategic Formation of Credit Networks [10] Jsilver

Summary:

Pros:

Cons:


Teapot - Trust Network (Stanford U. research project) [11] Aditi Mithal

Summary:

Teapot : "transactions and exchanges along paths of trust" This Platform was developed to enable trusted interactions on the web, as a Stanford research Project. Every aspect of transaction and communication involves trust, it is an important factor while buying and selling a product or assigning a task and getting the right result. The question is how can one analyse the trust factor existing between people .Teapot is a trust network which decides who to trust and who not to by analyzing the online interactions and social network of the users . Teapot finds a "Trust Path" between the users and establishes a trust network which determines how much a user is likely to trust the user on the far end. It captures the idea of transitivity of trust wherein it determines the level of connectivity that exists between the two users on the basis of mutual friends. This also determines the trust score. Having a "shared background" enables stronger trust relations because one is likely to trust someone having the same background Any references made by the users for building stronger trust relations are also taken into account by this platform.

Pros: - Everyone likes to trust someone they know, it works like a powerful recommendation system. Teapot uses this feature of connectivity and trust existing in relationships to make user's transaction and experience better.

- In a trust network, one would not likely indulge in fraudulent activities as it would disrupt all his/her connections. So, this eliminates unwanted users and online fraudulent practices.

- Teapot solves the cold-start problem by making trust portable across marketplaces. (paper ref.) [Also, Teapot provides access to its reputation system through simple, easy-to-integrate web-based APIs.]

- Transactions based on such a trust network reduces anxiety and boosts a larger trust network.


Cons: - There is a possibility that weaker intermediate relations are found between the users which is not enough to establish healthy trust relations.

- Social Network: Not everyone is on a social networking site( like FB ), the platform takes into consideration the social circle existing on facebook only, which eliminates the possibility of more existing connections .


Others??

Summary:

Pros:

Cons:


Identify what is working and what is failing on current crowdsourcing platforms. Find inequality between how requesters and workers are treated by the system. Please feel free to sign up to review as many of these platforms as you'd like.

Amazon Mechanical Turk


Upwork/oDesk


Freelancer @arichmondfuller


Taskrabbit @arichmondfuller


PeoplePerHour


any others??