Milestone 4 Triple Clicks Transparency: Implicit Metrics to Build Quality Estimation Models

From crowdresearch
Jump to: navigation, search

Problem

How to quantify activity geared towards quality on the site to be able to provide evidence of activities and actions on the platform?

Inspiration

Borrowing from usability testing (timing delays, backspaces, delete, clicks, scrolling, etc), Article on “Judging Quality Implicitly” [Rzeszotarski and Kittur, UIST ‘12]

Concept

Since it takes time to develop models based on different tasks (and that there will always inevitably be variables), this concept is likely for the future. Developing a mechanism in the background to collect data on workers behaviors/patterns in interacting with a task’s instructions or specific HITs will help to develop a rudimentary model (or at least metrics) that help to gauge individual workers workflow, comprehension of the task, implied skills, and confidence in their work. Similarly, we could track specific requestors and how their HIT batches or tasks fare over time. This could have the benefit of incentivising both workers and requesters by allowing them to look over their past performance and progression, much as one might with an activity monitor.

Limitations

  • Balancing transparency (who can see the metrics and data) with privacy. Whose interests are we protecting or supporting?
  • Validating models. There may be a considerable variation across individual tasks that a single model might be too rigid or too simple.
  • Identifying key metrics and performance indicators to inform models.
  • Validating the assumption that insights from usability testing are linked to concepts like reputation, motivation, performance, trust, etc.