WinterMilestone 3 DavidThompson ReputationIdea: BaselineTrustCreation

From crowdresearch
Jump to: navigation, search

Goal - To establish a baseline level of trust amongst all participants in the Daemo platform.

The key question is how to build trust in an unknown context? At present workers are penalized for being unknown or new. With this approach I suggest the creation of a trust baseline for all Daemo participants. This would happen in the following way:

1. Upon registration each user (worker and requester) would perform an initial screening against a battery of 'gold standard' tasks that had been curated according to the difficulty required to complete them. In addition all participants would be evaluated against a validated scale to assess participant conscientiousness (a psychological trait, and a component of the Big-5). These scores get signaled through your profile, and do not change, ever. As such your completion of these baseline tasks is to be taken seriously.

Gold Standard Task Curation

Daemo administrators should curate an ever growing library of tasks to be used for the evaluation of new registrant performance. These tasks should be coded according to a difficulty or cognitive load and the new registrant should be able to progress up a ladder of difficulty of tasks once each earlier level is complete. For each completed level the new registrant earns a level badge that is displayed on their profile. For the battery of tasks completed during each level, new registrants are given an average 'time of completion' and 'accuracy' score.

Currently people tolerate a 1-2 day registration process on mTurk; the process is currently opaque. With this suggestion every registrant is an active participant in an initial evaluation against curated gold standard tasks, resulting in the establishment of an even playing field amongst all workers and requesters alike.

I believe there is some value in having requesters similarly having to build a trust profile, and experience the same process as workers.

Trust profile determining your view on the available work

Given that the gold standard work used to evaluate the initial trust profile is curated according to its difficulty, you could imagine holding work back from registrants who hadn't reached a certain level (by completing the gold standard tasks at that level). One could imagine all 'Level 5' (and above work) being invisible to participants who hadn't completed the Level 5 screening. In this way, each participant would view a unique slice of work available to them, without seeing the total universe of possible work available.

This suggested design ensures that all participants in the platform have experienced the same process in joining and that that performance is signaled honestly, through the platform, in an objective and unambiguous way. Such transparent signaling should ensure even new workers are trusted from the get go, or at least not distrusted from the outset.

Milestone Contributors

Slack usernames of all who helped create this wiki page submission: @dcthompson