Guild leveling - review and quasi-mentorship ARichmondFuller

From crowdresearch
Jump to: navigation, search

Overview

Leveling, the process by which the platform defines skill competency and expertise, is a key process within Guilds on Daemo. Leveling directly impacts pricing, reputation, and although we are not yet addressing training and learning, there are also significant impacts in that area. We need to clearly outline the process, methodology, algorithm and scalablity of leveling.

Some of this is based on our work on leveling for Daemo Guilds from 2015: http://crowdresearch.stanford.edu/w/index.php?title=Leveling_Brainstorm by @acossette, @arichmondfuller, @dmorina, @james, @neilthemathguy, @Trygve, @niki_ab @ucerron @claudia @dilrukshi


Observation is a powerful way to gain knowledge. When workers are given a clear rubric outlining what is expected at each "level" with clear examples or work, they learn what is the requirement for each level. Giving workers the opportunity to review other workers' completed work at a variety of levels(bands/grades) allows workers to see examples of good practice, bad practice and provides benchmarks against which they can complete self-assessments. This is in fact, a form of quasi-mentorship where workers engage with others' work to enhance their own skills, knowledge and work performance.

Assessing work to determine if a worker should level up requires time and resources.

A proposal to engage workers in review and quasi-mentorship:
1. engage in training tasks to ensure understanding of levels
2. self-evaluate
3. peer-evaluation

Workers who want to leveled-up to use a MOOC-inspired peer review model where they first of all engage in assessing some training tasks. Once they accurately assess those task, they do a self-evaluation then finally, they take part in assessing the work of other workers based on a clear rubric. The training tasks will be created by the Guild and must be completed first in order to ensure the worker understands the requirements at each level. The number of these will be set by the Guild - 5-7 are typical numbers in MOOCs. A worker's work will be assessed by 5 other workers - the top and bottom marks can be dropped and the middle three marks averaged to determine if the worker can level up or not. This model is scalable and sustainable. There can be the option of workers having to pay a fee to put themselves forward for level-up. Another option might be for workers to contribute a certain number of other workers' assessments to unlock the opportunity of getting leveled-up themselves.

Considerations need to be given to privacy and requester confidentiality for this. Offer requesters the option of option out of having their tasks available for task review. For leveling in Guilds we need to identify:

  • how many levels are there per skill
  • if we want a standardized template for each skill-specific Guild
  • what competencies are required for each level with the Guild/s and how we will measure this
  • how competency is determined and validated. Exams, interviews, outside world experience
  • how workers move from level to level
  • what the bands/grades/levels will be such as Beginner, Knowledgeable, Expert or will there be numbers such as 1, 2, 3, 4 (1=beginner to 4 = expert)
  • how we apply the standards across different cultural/educational systems
  • how dynamic the assessment of skills should be


Future work:

Key areas to consider for building skills and knowledge through training and education:

  • When there is a need for upskilling, will we build in mechanisms to alert the worker?
  • Will the Guild identify learning resources to further develop skills?
  • If a worker acquires new skills, education and techniques, how can they use this to advance through the system?
  • Do we create a learning library from open source/free content? Partnership with Coursera?
  • How important is mentoring, real time reviews, peer assessments in validating skill?