Milestone 4 Task Ranking by Crowdgeist

From crowdresearch
Revision as of 02:33, 8 February 2016 by Angelarichmondfuller (Talk | contribs) (What we propose)

Jump to: navigation, search

Outline of current system

Crowdsourcing markets currently offer ranking for matching those who want their tasks completed to those who do the work. Workers want to get work suited to their own needs, skills and possibly personalized criteria. Current reputation systems do not adequately differentiate the quality of the tasks completed by workers: requesters complain that they cannot get good results from some workers and cannot be guaranteed a high quality of results, and would like some KPI to evaluate the reputation of a worker (@seku).

Trust and power are broken because traditional approaches to ranking does not differentiate in in ways that fully meet the needs of workers or requesters. Moreover, ranking has one dimension, while our idea is this dimension flattens workers attitudes and possibilities. The quest to achieve 5 star mono-dimensional recognition, if many cases, leads to reputation inflation (Horton 2015) or threatened retaliation and has also lead to the emergence of add-ons such as Turkopticon to enable workers to filter out requesters who are notorious for not paying. Ranking workers and requesters may results in cases of (endless?) retaliation, over-competition, gaming and, at times, inaccurate matching of tasks from requesters to workers and in cases are deemed unfair (Horton 2010). In synthesis, current rating systems do not take into account all of the possible criteria and datasets that could make the matching of task to worker more efficient and effective (Horton 2012 - Matchmaking).

What we propose

We propose multi-dimensional ranking system based on diversity and not on a mono-dimensional, final, numeric system.

We will test vectorization algorithms.

To recognize possible alternative systems, we propose to analyze:


task: listen to supergood music

personality: my preferences give me more pleasure than the top 10


task: write articles about all known by mankind

personality: my skills and preferences allow me to write an article whenever I want and on the subject I want, rather than following a list of priorities and needs in an encyclopaedia

OkCupid / Tinder

Matching systems from dating sites like OKCupid are very detailed in matching people with each other. ( Visual Matching in Tinder can also be interesting. You only see each other… if you match each other.

Kindle suggestion system (“Customers who bought this also bought” and “Customers who viewed this also viewed” sections.

Task: offers suggestions for books one might be interested in reading but keeps the kid-related stuff separate.

Current Stanford Crowd Research Collective - Daemo

We could also insert Daemo in our list if any ranking system is tested.

Time banking / Timerepublic

Hours exchange.


  • Specific details to follow*

The Results

Multidimensional Percentage Ranks should result in possible accurate Matching with no retaliation drawbacks and more identification by both workers and requesters.







References and Notes

Psychological Questionnaires ask many (also very private but entertaining questions) in order to match you quite accurately through a percentage number from 1 to 100. r The company seem to put put a lot of research in how to make recommendations. For collaborative filtering there is a cold start problem where the algorithms are based on historical usage data. We'd like to be able to recommend new workers to tasks even when there is no usage data to analyze. Predictive preferences with deep learning. Task recommended to workers based on signals based a comprehensive set of parameters including ability and availability (skills, timescale, etc). To determine the worker and requesters preferences based on historical usage data. An alternative to the ranking system based on roles, diversity, compatibility.

(@admp idea) We believe there are requesters and workers with different strengths and through multi dimensional matching we can make them happier. A key would be to have a algorithm which is hidden and figures out information about workers and users from various dimension as (value of efficiency of getting work done accurate or in time, etc. )

@amdp suggests: a simple comparison methodology a system with ranking and one without for sure the one without does not exist in the online labour market

Information on what’s required of us for this milestone:

Examples to look at: Example1: Gupta A, Thies W, Cutrell E, et al. mClerk: enabling mobile crowdsourcing in developing regions. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2012: 1843-1852.

Example2: Narula P, Gutheim P, Rolnitzky D, et al. MobileWorks: A Mobile Crowdsourcing Platform for Workers at the Bottom of the Pyramid. Human Computation, 2011, 11: 11.

Example3: Vaish R, Wyngarden K, Chen J, et al. Twitch crowdsourcing: crowd contributions in short bursts of time. Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 2014: 3645-3654.

A team that does research and matching and is always working on improving the matching algorithms. Questionnaires about personal working preferences that are fun to answer

We need this sort of differentiation when we put specific skills into the task scenario. What I need and want as a worker when doing Japanese translation is very different from what I need and want consulting on UX, for example. Forum discussing the above circles.


okcupid… multiranking ( questionairies/very detailed matches … /)

very detailed information, and very precise matches, everyone gives also very closed information ,

vs. tinder…

anonymous superficial (location based) matches… once you have them you can dive (tinder is not the best example as it is location based and matching very superficial but Human and intuitive on pictures).

Peer-to-peer exchange is when two parties trade goods or services directly to each other. Exchange services such as Uber, Lyft, AirBnB, etc. offer information systems to support the matching process of supplier and demander. Such exchange services exemplify rapid growth and demonstrate the power of the so-called shared economy. In this manner, a lot of different services have risen, from simple ride sharing, car sharing (, parking lot sharing (, workspace sharing (, temporary overnight sharing ( to even textbook sharing ( The most common form of exchange is a good or service for money. (Caroll 2014) Since these exchange services are rather handled privately instead of as a business, most information systems provide a rating mechanism to build up credibility. A typical rating system is a 5 point Likert scale resembled in stars (Uber, AirB’n’B). To stay above a certain rating, the work might involve a high amount of “emotional work” (Hall & Krueger 2015, Rogers 2015). Depending on the matching system of the service provider and the amounts of ratings, a single negative or not 100% review can cause severe consequences for the worker him/herself. Furthermore, money as a trading currency is discriminating towards people being incapable of having or earning money, such as the handicapped, sick, children or unskilled ones. An alternative for this is time sharing. hOurWorld is such a timebanking provider. Users can post and requests tasks, offering a service or accepting an offer. The currency is time, therefore, every service has the same value. For example one hour of a blue collar job equals the same as one hours of white collar job. Both parties have to agree for the actual exchange of the time value (approving credit). A rating system as described above does not exist. The exchange is purely based on offers and requests from people. In real life, however, the timebank lives from the offers of people (Belotti et al. 2014). Further research shows that engagement and helping out in a local community has a psychological benefits to an individual. According to , among other benefits volunteering shows increased personal well-being, reduction of depression and increased self-esteem (Thoits 2001, Post 2005).

Caroll, J. (2014). Co-Production Issues in Time Banking and Peer-to-Peer Exchange.

Bellotti et al. (2014). Towards Community-Centered Support for Peer-to-Peer Service Exchange: Rethinking the Timebanking Metaphor

Thoits, P. A. & Hewitt, L. N. Volunteer work and wellbeing. Journal of Health and Social Behavior 42, (2001), 115–131

Hall, J. & Krueger, A. (2015). An Analysis of the Labor Market for Uber’s Driver-Partners in the United States.

Rogers, B. (2015). The Social Cost of Uber.

Post, S. Altruism, happiness, and health: It’s good to be good. International Journal of Behavioral Medicine 12, 2 (2005), 66-77.