Guild leveling - self-assessment + peer-assessment + Meta-reviews ARichmondFuller

From crowdresearch
Jump to: navigation, search

Overview

Workers who produce quality work want to be able to make more money per hour on crowdsourcing marketplaces. Requesters want a quick and easy option of choosing to pay a bit more for guaranteed quality output. How much more (and over time) is still to be determined.

Daemo offers Guilds for workers which offer leveling as a guide for quality. The workers' output is assess and they are leveled up then they demonstrate continued quality to a certain level. The leveling system aims to assess worker quality. Guilds will allow the crowd to identify who is excellent, good, etc.

Leveling directly impacts pricing. The higher the level, the more the worker earns per hour. Workers who produce higher quality work will progress through the levels. Our initial research indicates that the more tasks a worker has completed, the lower quality he/she believes of reviewed tasks. This is system which is do-able in the first instance and also scalable and sustainable for the future.

Although it's been decided that initially Daemo will offer workers the opportunity to join 1 Guild, the question remains as to how we can assure quality for the various areas of work including but not limited to processing photos and videos, data cleaning and verification, information collecting, and data processing. If a worker levels up in based on the output in one area, does it mean that they will produce high quality results in another area? Also, as the platform further develops, there may be a move to include macro-type work in addition to the micro-task work. More Guilds will need to be formed as a result.

Guilds' Roles and Responsibilities:

  • outline how many levels are required within their particular Guild
  • produce and display benchmark tasks for each level (this also acts as a training feature. People improve their own work when seeing examples of good practice. Here we may also want to include some examples of bad practice with variations on completed real tasks)
  • clearly outline the process and procedures for workers to level-up with the Guild
  • produce and display a clear rubric of what is expected at each level (band/grade - not sure what terminology we should use)
  • identify whether or not to include an appeals process

Proposed System:

  • workers used the benchmark tasks to identify which level they are ready to apply for.
  • workers complete a self-assessment - this should be based on a rubric and could be in the form of comparing their work to a benchmark task
  • workers engage in a training reviews. When 3 are identified correctly in a row, they stop the training and go on to do the peer reviews (5-10).
  • once workers have received n reviews on their own tasks (from last 20% of competed work) they can submit their "pack/portfolio/tasks" to be reviewed by Meta-reviewers
  • Meta-reviewers review the set of peer-reviewed tasks and then either level-up the worker or identify what they need to do in order to improve, ideally with suggestions on tools/resources/training that would help them to improve.


Key take-aways:

  • This system allows for ample observation which is a powerful way to gain knowledge.
  • When workers are given a clear rubric outlining what is expected at each "level" with clear examples or work, they know what is the requirement for each level.
  • Reviewing other workers' completed work at a variety of levels(bands/grades) disseminates of good practice, bad practice and provides benchmarks against which they can complete self-assessments.
  • This is in fact, a form of quasi-mentorship where workers engage with others' work to enhance their own skills, knowledge and work performance.
Privacy and requester confidentiality: Provide a tick box on the Terms and Conditions or task authoring page interface where the requester can opt out of
  allowing the worker to use his/her task in their work portfolio for the purpose of leveling up. Use wording such as *Workers show their portfolio of work in 
  order to level-up. If you do not want other workers to review the workers quality of task completion then please untick this box*  Or *I give my consent for 
  workers to be assessed on how well they complete my task.*


For leveling in Guilds we still need to identify:

  • how many levels are there per skill area
  • if we want a standardized template for each skill-specific Guild
  • what competencies are required for each level with the Guild/s and how we will measure this
  • how competency is determined and validated. Rubric for tasks.
  • what the bands/grades/levels will be such as Beginner, Knowledgeable, Expert or will there be numbers such as 1, 2, 3, 4 (1=beginner to 4 = expert)


Future work on key areas to consider for building skills and knowledge through training and education:

  • When there is a need for up-skilling, what kind of feedback is needed and how will the details be delivered to the worker. Can will we build in mechanisms to alert the worker?
  • Will the Guild identify learning resources to further develop skills?
  • If a worker acquires new skills, education and techniques, how can they use this to advance through the system?
  • Do we create a learning library from open source/free content? Partnership with MOOCs such as Coursera, EdX, Iversity, NovoEd, FutureLearn etc.
  • Do workers want more formal opportunities for mentoring? If so, what form could this take?