Leveling Study

From crowdresearch
Revision as of 06:31, 5 April 2016 by Aarongilbee (Talk | contribs) (GUILD LEVELING STRUCTURE)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

SUMMARY: This page is dedicated to the topic of guild-based leveling.


Leveling arose as a method to address the cold start process of Daemo, or any new software as a matter of fact, when new users entered the system. The idea being that the user needs, initially, a framework to accurately understand the software and its functionality, and eventually the user would become more involved in software organizational actions, such as managing portions of a guild system. As of current discussions, users benefit by increased payment levels, greater information access, increased ability to influence guilds and a growing variety of tasks tied with increased ranks.


Crowdsourcing platforms are subject to reputation inflation due to repercussions from social pressures. As a result worker-requester relationships can be mired by false impressions of the other's professional profile as presented from a platform. Guild based leveling presents a solution to this issue by creating a system whereby a worker's peers-identify through review the quality of another person's work product. With higher review ratings, the worker achieves a higher level within the guild as a publicly demonstrable award of task-performance competence. In this study, researchers utilized a mixed-method research study using a daily activity/thought journal, scripted tasks, and a primitive micro-world simulation. Researchers identified that leveling mechanism indeed did resulted in better work products as a function of leveling and identified patterns in behavior that changed with increased rank.


Level Significant Change
1 Learns the system and performs any of a number of tasks with ground truthes
2 After a demonstrated competency level, the worker is able to review others with a .5 sigma increase in pay, ability to respond to posts in a forum, introduce standard survey tasks
3 Worker receives ability to review micro-task results within a gaming framework (speeds up review and increases pay levels) with a 1 sigma increase in pay, start posts in Daemo forum, introduce standard tasks
4 Worker receives voting ability within the Daemo forum and a 1.5 sigma increase in pay. Introduce tasks with real world requesters.
5 Fictional level to help constrain the other levels. This is the guild administrator who is played by a confederate team for the purposes of the study.


Researchers recruited 20 participants from MTurk to perform this study. Over a three week period, participants were given tasks and increased benefits according to the levels. Participants engaged with researchers on a daily basis by receiving and completing tasks through emails and direct access to the Daemo Forum as a function of their leveling. Each completed a daily diary detailing their activities along with quantitative questions tied to the platform and mood. After 3 weeks, the diaries were analyzed using Qualitative Comparison Data Analysis and custom codings that were identified, discussed, and review by a team of researchers using RDQA. Extreme and unique cases were identified for follow up interviews.

Wizard of Oz Method

Tasks were assigned to groups as they emerged through reviews by a sequential Graeco-Latin square design. The task classification scheme is as follows:

  1. Tasks with known answers
  2. Standard Surveys
  3. Standard Tasks
  4. Tasks from Real World Requesters

Each task will be given on the respective days to each group.

StudyDesign Leveling 4-5-2016.jpg

Quantitative analysis was performed to identify behavior changes associated with worker level.