Milestone 6 RATH - The IDA Platform

From crowdresearch
Jump to: navigation, search

Title

The IDA Platform: Information Equality for a Trustworthy and Empowered Crowdsourcing System

Abstract

To date, crowdsourcing platforms have consistently faced challenges with regard to equal access to information for decision making purposes. Whether it is a requester unsure if the worker assigned to the task is capable of providing a quality result, of a worker unsure if s/he will have their work fairly critiqued, current systems are riddled with “blind spots”. Platforms to date have not been designed with a depth of data transparency and availability in mind. This profound asymmetry of information access creates a power differential, where a party is forced to make decisions based on the unknown. This in turn creates a lack of trust and empathy and potential for exploitation that undermines the sustainability and integrity of the system. We seek to create a new platform based on full access to Information, Data, and Analytics (IDA) that allows for individuals to easily access the metrics they find meaningful to insure trust in the system and its participants. IDA is the fulcrum on which the system stays in dynamic and corrective balance.

Motivation

The problem we are attempting to solve is a one of an imbalance in access to information. This perceived asymmetrical alignment has eroded trust between workers and requestors which has lead to an imbalanced powering sharing relationship. Controlling the access to information and the subsequent inequities it causes are historical as well as contemporary problems. Systems like M-Turk merely ported over the problems with them as they were created by consumers of labor that were looking to activate underutilized/marginaled workers. While it would be jaded to suggested that the intention was exploitative, the perception amongst Mturkers and close observers appears to be just that. The motivation is to bring balance to the relationship between workers and requestors through information, data and analytical tools available to both sides of the equation. Ineffective rating systems.

Related Work

Researchers have to date investigated such options as ranking as a way to bridge this information gap. (insert link) However,each worker and requester has a unique checklist for who/what s/he may deem trustworthy and this may not be reflected in popular opinion. Herein lies the fault in ranking systems. Ranking systems are based on a default, or the opinions of others and do not directly speak to variables that may be important to any given individual. They are in some ways a popularity contest and not a match-making tool. The prom-king/queen may not be the best match for you.

Insight

We are seeking to create a platform that utilizes Information (historical), Data (real-time) and Analytics (predictive) (IDA), to set symmetrical access to decision making tools. This symmetrical access creates a new opportunity to mitigate the power differential, and expose exploitative practices through transparent metrics based dashboards. We believe this transparency will allow individuals to investigate and satisfy their individualized trust points to whatever extent is needed to deem the platform and participants "trustworthy".

The IDA platform by solidifying trust creates a domino effect of positive movement throughout the system. Workers are able to be more efficient as they no longer need to "sniff around" on boards to find reliable requesters based on word of mouth. This increased efficiency results in higher total compensation as less worker time is spent getting work and more time is spent doing work. In turn this resulting higher hourly pay rate can buoy the worker to increased motivation and satisfaction. This increased productivity will also increase the total number of completed HITS and this increase in repetitions can improve quality of work submitted. Additionally, requesters will be have their HITS better matched with workers decreasing turn around time on HITS. Requesters will have more readily available feedback metrics for quality of HIT and appropriateness of compensation which will increase their ROI - less time spent "guessing". Not to be overlooked is the impact of IDA not only on empathy but on self-reflection. With ease of access analytics, both parties will be able to check to see if their perceptions of the work being done matches up with reality.

System

The heart of the system is found in the modular dashboards available to participants. The modular library will provide both a width and depth of quantitative insight into any/all aspects of the system's functionality. We recognize the challenge of the volume of modular functionality required to provide near omniscience from data mining however this is critical to solving the asymmetrical information problem.

We seek to offer historical information, real-time data and predictive analytics in a variety of visualizations and decision making tools. Modules discussed to date include:

  • Pricing Engine
    • Milestone_5_RATH_Mockup:_Compensation_Suggestion_Tool - Offers an historical histogram of completed transactions based upon HIT variables. Could allow for rules engine based on standard deviation to set minimum compensation threshold
    • Real-time Current Price Charting Tool - Provides real-time "stock-price" like graphics of HIT compensation based on numerous available variables
  • Performance Profile
    • Worker History - Number of HITS, acceptance rates, time to completion, breadth of task types
    • Requester History - HIT Volume, payment turnaround time, HIT clarity/acceptance rates, Communication profile
  • HIT Metrics
    • Current Workers - How many workers with a certain profile are currently active or have been within a given variable of time.
    • Current HITS - Metrics of current HIT volume - variables could include HIT type and other parameters
    • HIT FEED/Worker FEED - Provides a real-time feed of likely matches of HITs/Workers (view dependent on whether worker or requester)


Technology as envisioned here in a series of modularized tools cannot prevent poor behavior, but it can create a level of transactional transparency and access to systemic information that encourages altruism, delayed gratification and prevents such malfeasance as much as it encourages a system run by enlightened self interest and market forces. Both of which when executed efficiently require access to all the data required to make informed rational decisions. In a metrics driven market, worker expertise is not who your are, but how you do it (well, timely) and requestor reputation is founded not on anecdotes, but on an index of measurable criteria.

Evaluation

The results we strive for are tied to changed behaviors and capturing the tenants of transformation. As a dynamic model driven by market forces, the corrective/responsive nature of IDA will capture changes in attitudes and approaches of both workers and requestors as well as deviations in price, productivity, HIT rejection, scope and scale of the system and other variables. As with the tools designed to help with the decision making process, the same a la carte approach will serve the determination of success, which like beauty, is in the eyes of the beholder.

References

  1. Trust-based fusion of untrustworthy information in crowdsourcing applications; Matteo Venanzi, Alex Rogers, Nicholas R. Jennings
  2. The Philosophy of Information; Floridi, Luciano
  3. Rational Choice; Elster, Jon
  4. Crowdsourcing information systems : a systems theory perspective;Geiger, David, Rosemann, Michael, & Fielt, Erwin (2011) Crowdsourcing information systems : a systems theory perspective. In Proceedings of the 22nd Australasian Conference on Information Systems (ACIS 2011), Sydney, Australia.
  5. Han Yu, Zhiqi Shen, Chunyan Miao, and Bo An. 2012. Challenges and Opportunities for Trust Management in Crowdsourcing. In Proceedings of the The 2012 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technology - Volume 02 (WI-IAT '12), Vol. 2. IEEE Computer Society, Washington, DC, USA, 486-493. DOI=10.1109/WI-IAT.2012.104 http://dx.doi.org/10.1109/WI-IAT.2012.104
  6. Han Yu, Zhiqi Shen, Chunyan Miao, and Bo An. 2013. A reputation-aware decision-making approach for improving the efficiency of crowdsourcing systems. In Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems (AAMAS '13). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 1315-1316.