Difference between revisions of "Leveling Study"
Aarongilbee (Talk | contribs) (→METHOD) |
Aarongilbee (Talk | contribs) (→GUILD LEVELING STRUCTURE) |
||
| (6 intermediate revisions by the same user not shown) | |||
| Line 13: | Line 13: | ||
|- | |- | ||
|1 | |1 | ||
| − | |Learns the system and performs any of a number of | + | |Learns the system and performs any of a number of tasks with ground truthes |
|- | |- | ||
|2 | |2 | ||
| − | |After a demonstrated competency level, the worker is able to review others with a .5 sigma increase in pay, ability to respond to posts in a forum | + | |After a demonstrated competency level, the worker is able to review others with a .5 sigma increase in pay, ability to respond to posts in a forum, introduce standard survey tasks |
|- | |- | ||
|3 | |3 | ||
| − | |Worker receives ability to review micro-task results within a gaming framework (speeds up review and increases pay levels) with a 1 sigma increase in pay, start posts in Daemo forum | + | |Worker receives ability to review micro-task results within a gaming framework (speeds up review and increases pay levels) with a 1 sigma increase in pay, start posts in Daemo forum, introduce standard tasks |
|- | |- | ||
|4 | |4 | ||
| − | |Worker receives voting ability within the Daemo forum and a 1.5 sigma increase in pay. | + | |Worker receives voting ability within the Daemo forum and a 1.5 sigma increase in pay. Introduce tasks with real world requesters. |
|- | |- | ||
|5 | |5 | ||
|Fictional level to help constrain the other levels. This is the guild administrator who is played by a confederate team for the purposes of the study. | |Fictional level to help constrain the other levels. This is the guild administrator who is played by a confederate team for the purposes of the study. | ||
|} | |} | ||
| + | |||
=METHOD= | =METHOD= | ||
Researchers recruited 20 participants from MTurk to perform this study. Over a three week period, participants were given tasks and increased benefits according to the levels. Participants engaged with researchers on a daily basis by receiving and completing tasks through emails and direct access to the Daemo Forum as a function of their leveling. Each completed a daily diary detailing their activities along with quantitative questions tied to the platform and mood. After 3 weeks, the diaries were analyzed using Qualitative Comparison Data Analysis and custom codings that were identified, discussed, and review by a team of researchers using RDQA. Extreme and unique cases were identified for follow up interviews. | Researchers recruited 20 participants from MTurk to perform this study. Over a three week period, participants were given tasks and increased benefits according to the levels. Participants engaged with researchers on a daily basis by receiving and completing tasks through emails and direct access to the Daemo Forum as a function of their leveling. Each completed a daily diary detailing their activities along with quantitative questions tied to the platform and mood. After 3 weeks, the diaries were analyzed using Qualitative Comparison Data Analysis and custom codings that were identified, discussed, and review by a team of researchers using RDQA. Extreme and unique cases were identified for follow up interviews. | ||
| − | Tasks were assigned to groups as they emerged through reviews by a Graeco-Latin square design. | + | ==Wizard of Oz Method== |
| + | |||
| + | Tasks were assigned to groups as they emerged through reviews by a sequential Graeco-Latin square design. The task classification scheme is as follows: | ||
| + | #Tasks with known answers | ||
| + | #Standard Surveys | ||
| + | #Standard Tasks | ||
| + | #Tasks from Real World Requesters | ||
| + | |||
| + | Each task will be given on the respective days to each group. | ||
[[File:StudyDesign_Leveling_4-5-2016.jpg|500px]] | [[File:StudyDesign_Leveling_4-5-2016.jpg|500px]] | ||
Latest revision as of 07:31, 5 April 2016
SUMMARY: This page is dedicated to the topic of guild-based leveling.
Contents
VISION
Leveling arose as a method to address the cold start process of Daemo, or any new software as a matter of fact, when new users entered the system. The idea being that the user needs, initially, a framework to accurately understand the software and its functionality, and eventually the user would become more involved in software organizational actions, such as managing portions of a guild system. As of current discussions, users benefit by increased payment levels, greater information access, increased ability to influence guilds and a growing variety of tasks tied with increased ranks.
EXAMPLE PAPER ABSTRACT
Crowdsourcing platforms are subject to reputation inflation due to repercussions from social pressures. As a result worker-requester relationships can be mired by false impressions of the other's professional profile as presented from a platform. Guild based leveling presents a solution to this issue by creating a system whereby a worker's peers-identify through review the quality of another person's work product. With higher review ratings, the worker achieves a higher level within the guild as a publicly demonstrable award of task-performance competence. In this study, researchers utilized a mixed-method research study using a daily activity/thought journal, scripted tasks, and a primitive micro-world simulation. Researchers identified that leveling mechanism indeed did resulted in better work products as a function of leveling and identified patterns in behavior that changed with increased rank.
GUILD LEVELING STRUCTURE
| Level | Significant Change |
|---|---|
| 1 | Learns the system and performs any of a number of tasks with ground truthes |
| 2 | After a demonstrated competency level, the worker is able to review others with a .5 sigma increase in pay, ability to respond to posts in a forum, introduce standard survey tasks |
| 3 | Worker receives ability to review micro-task results within a gaming framework (speeds up review and increases pay levels) with a 1 sigma increase in pay, start posts in Daemo forum, introduce standard tasks |
| 4 | Worker receives voting ability within the Daemo forum and a 1.5 sigma increase in pay. Introduce tasks with real world requesters. |
| 5 | Fictional level to help constrain the other levels. This is the guild administrator who is played by a confederate team for the purposes of the study. |
METHOD
Researchers recruited 20 participants from MTurk to perform this study. Over a three week period, participants were given tasks and increased benefits according to the levels. Participants engaged with researchers on a daily basis by receiving and completing tasks through emails and direct access to the Daemo Forum as a function of their leveling. Each completed a daily diary detailing their activities along with quantitative questions tied to the platform and mood. After 3 weeks, the diaries were analyzed using Qualitative Comparison Data Analysis and custom codings that were identified, discussed, and review by a team of researchers using RDQA. Extreme and unique cases were identified for follow up interviews.
Wizard of Oz Method
Tasks were assigned to groups as they emerged through reviews by a sequential Graeco-Latin square design. The task classification scheme is as follows:
- Tasks with known answers
- Standard Surveys
- Standard Tasks
- Tasks from Real World Requesters
Each task will be given on the respective days to each group.
Quantitative analysis was performed to identify behavior changes associated with worker level.
