Research Questions - AngelaRichmondFuller

From crowdresearch
Jump to: navigation, search
  1. Discuss the questions you are interested in
  2. TO DO: Summarize previous research you read
  3. TO DO Create short summaries of articles
  4. TO DO: Discuss how this papers applies to your research idea within Daemo

Research Topics and Questions

Learning & Skills:

  • Do workers and requesters within the Daemo marketplace want to have learning and skills formally recongnised?
  • What is the most effective and least time- and resource-consuming way to certify learning and skills on Daemo?
  • How might Daemo identify when a worker needs some skills development in a particular area?
  • Would workers find learning cohorts useful for job satisfaction and career progression?
  • Will mentors naturally emerge from learning cohorts to support other workers? If so, how can Daemo compensate these workers?

Reputation System:

  • How will requesters be provided with information on a workers skill level and quality through the boomerang reputation system?
  • Would the provision and completion of benchmark tasks help to identify worker skill levels and provide valid and useful information to feed into the reputation system so that matching workers' skills and knowledge to the projects/tasks is more efficient and accurate?
  • Would workers want to be provided with opportunities to join specific skill groups for learning and collaborative work opportunities? If so, would these learning cohorts/groups want to create/identify skill-specific benchmark tasks to be completed by new workers to the site to help ensure quality within Daemo? If so, would they want some sort of recognition/payment for this work in the form of reputation points, badges, bonuses, prestige, and free learning opportunities.
  • Are there mechanisms we can use to have the system trigger notification to workers (from the feedback received from the requesters) and to requesters (from the feedback received from the workers) should their skills in a certain area need to be improved along with suggested learning materials?

Previous Daemo crowd ideas on Faceted Reputation System [1]

A Different Question

A faceted reputation system that is dynamically generated on a per task basis. The idea is to provide not answer the question "Is this worker good or bad?" but instead to answer "Can this worker do what I am asking them to do?"


The features are flexible (numerical, alphabetic, multivariate profile etc.) but the foundation is such that the 'score' will be weighted based on the workers history with this specific type and level of task. The formula would also include a baseline of overall reputation score across all work history as well as 'constant' of skill/level, traits etc. The coefficients of each variable will change depending on the workers history in this specific task area.

For example, if a worker has a long history of successful completion of photo shop tasks and they are applying for a photoshop assignment, then the 'score' will rely heavily on history within this area. If this same worker recently starting programming in python and was looking to take on a task in this area, the reputation would include the overall reputation score of the worker, since the worker would not have history in this task area, they would have the option to up the coefficient by taking a skills test. Leveling would replace reputation to solve cold-start issues. In this case workers would not be penalized in areas of exhibited expertise by venturing in to new domains. As the workers gains more repetitions in this task category the score would shift from skills to experience.

Future The solution itself can scale to include more advanced machine learning, task matching, profile etc. The solution can grow in complexity.

The simple MVP solution though is to provide a dynamic and appropriate reputation metric that is truthful and supports expansion and professional development.

Previous work on Reputation System [2]

Leveling/Ranking of skill levels:

  • Would it be beneficial to the Daemo marketplace to identify and provide information on good practice by way of providing examples of tasks that are 'working toward', 'performing at level' and 'commendable' - or something to that effect.
  • What levels/ranks for each specific skill be useful/appropriate?
  • Would Daemo benefit from having a bank of skills test questions created by the workers for using to level new workers by way of skills tests to identify their level of expertise on any given topic? If yes, then would we want these skills tests to carry some reputation points?
  • Should workers' and requesters' reputation ratings link directly to each skill area rather than an aggregate rating for all the tasks in a Project (incorporating various skills all bundled into one). Would requesters want to know the quality of work a worker does on logo work vs python coding, for example?
  • Would identifying workers' knowledge and skills (including levels) be useful to Daemo? If so, which of these methods would provide the best results for our purposes?

indent *Workers self-certify their level (novice, veteran, expert) for each skill for which they want to accept work.

indent *Workers are set some tasks (determined and developed by set of workers (paid) with those skills) to complete in order to demonstrate their skills OR submit a portfolio of previous work for review?

How the badges would work:

  • Workers given badges to identify skill mastery. If the worker doesn't complete the task to an acceptable level they could choose to hide the results on their profile page.

Badges are linked directly to skills - when the skill evolves, the worker will get a notification that it's time to upskill in order to keep the badge. I envisage that we'd eventually builds some sort of skills development advisor algorithms into the platform which will link through to appropriate, free, relevant and timely resources for upskilling. Would self-certification lead to fraudulent behaviour? How would the reviewers of the tasks completed be paid? If we trust that workers will be honest about their skill level will that lead

*Previous ideas and work done on levelling. [3] Overview

Leveling, the process by which the platform defines skill competency and expertise, is very much the keystone process for the platform. Unto itself, the leveling process can be a challenging undertaking, but when you also take into account the impact those decisions have on pricing, reputation, and training/learning, the enormity can be overwhelming. Hence the need to establish the process, methodology, algorithm and scalablity of the leveling process before moving into development. In the equation of trust and power, leveling is more than a peer to peer dialogue, it impacts the clients and the platforms ability to deliver on micro and macro tasks. So when discussing leveling one must do so through these optics:


  • How many levels are there per skill?
  • Should there be a standardized template for all skills?
  • What competencies are required for each stepping stone?
  • How is competency determined and validated? Exams, interviews, outside world experience
  • How does one move from level to level?
  • Are there grade's within a level? (Beginner, Knowledgeable, Expert)
  • How do we apply the standards across different cultural/educational systems?
  • How dynamic should the assessment of skills be?


  • How does one acquire the skills, education and technique to advance through the system?
  • Do we create a learning library from open source/free content? Partnership with Coursera?
  • How important is mentoring, real time reviews, peer assessments in validating skill?


  • How do we set a price based on skill and task?
  • Does the individual or the platform set the price?
  • How dynamic is the pricing engine to demand, region and requestor?


Onboarding New Researchers:

  • What support mechanisms ensure newcomers stay with the project?
  • How can we structure the onboarding process so as not to overwhelm newcomers?

Suggestions by @arichmondfuller on @rcomptons Onboarding Newcomers Slideshow [4]

1. To help people catch up on missed meetings and to move things forward quickly, spend the last 3-5 minutes of each meeting summarising (and agreeing) the main points verbally and identifying "next steps" and actionable items along with who will take on what and items where other volunteers are needed. To spread the work-load, someone other than the DRI could volunteer to write those up to post in #announcements-discuss.

2. DRIs list who contributed to each milestone in #announcements at the end of each week helped to show who was doing what. The main problem @arichmondfuller found with this system previously is that you can't tell how much (or the quality of work, for that matter) that each person contributed. ​*Suggestions:*​ DRIs list contributors for each milestone in the order of percentage of contribution - first being the top contributor and the last place as having contributed the least.

Onboarding workers and requesters:

Would an incubator space, interview or skills assessment be an effective way to select skilled workers to work on Daemo or should anyone be allowed to start working asap without a skills assessment?

Claudia and I previously identified a way of onboarding newbies by offering them work on tasks for charities and NGOs reduced pay to build a reputation score [5]). I would also add another element in that the worker's true ability of the listed qualifications/certifications will also be assessed through the output. There could be 360 reviews by co-workers and mentors which will determine if the worker will be invited to join the site or not. This could be outlined as being the "interview stage". People in the real world aren't paid for preparing for and attending interviews so why should workers online? Daemo doesn't have to accept just any old worker. Another option if we want to pay people for this it to make this a separate arm of Daemo or separate not-for-profit company that the profitable arm of Daemo funds. If the worker doesn't get accepted after the incubation stage, they will be told why, what skills they need to improve and what they need to do in order to develop those skills through learning pathways, for example, take a course, read an article, do an assignment etc. Also, the worker would then be told when they can re-apply.


examples of other forums for workers [6]

Previous Work

Here is the link for my previous ideas on Reputation Systems:

Milestone 11 Skills-based Reputation System idea by AngelaRichmondFuller [7]

One idea on Daemo Reputation System by AngelaRichmondFuller [8]

Open Honest Crowdsourcing by AngelaRichmondFuller [9]