Milestone 4 TuringMachine: Feedback Mechanism with Motivation Leading to Better Results
Influence and Related work
Past research has shown that the lack of skills, misunderstanding, and irresponsible behaviors are some of the main causes of low quality submissions. Various techniques can be used to increase the quality of work. Figure below highlights the some of these techniques. The paper Shepherding the crowd yields better work, Dow et.al. 2012 proposed an idea of Rubric that can be helpful for quality measurement.
In What makes things fun to learn? Study of Intrinsically motivating computer games, 1980, professor Malone discusses the effect of Challenge, Fantasy, and Curiosity on the learning process. Based on the past research, we propose review system that is supported by Gamification & Visualization of the Real Time Task Progress. The objective of the system is achieve high quality results using feedback, competition, motivation, and incentives.
Iterative Review Process
Reasons for bad quality submissions
Trust Circle Design Process: Requestor - Expert Worker - Supervisors - Workers
- Figure 3.0 highlights the detailed review process
- Select Expert Workers using automated algorithms
- Select supervisors from the set of Expert Workers. Expert workers are paid higher and selected from pool of high accomplished individuals. We have also designed the ranking mechanism that can be integrated with the system to motivate workers to perform well and get into the class of Expert Workers. Motivation for being an Expert:
- Intrinsic motivation: Bad quality submission or cheating behavior affects entire crowdsourcing community. Most of the workers want to stop the bad guys and volunteer their time. However, the current system does not have any mechanism that will involve workers in filtering out bad submissions. Pool of experts is a motivated group of individuals who want to maximize the social welfare.
- Extrinsic motivation: The expert workers are paid higher for the experience & managerial skills they bring in. In addition, the proposed ranking mechanism & leaderboard system encourages the workers to do well and move into the class of socially recognized experts.
The task flow
- Figure 3.1 shows the task claimed by 6 workers. This number can be larger.
Feedback & Gamification
- Figure below highlights the worker A's dashboard. Please read the diagram from #0 to #5 i.e. from the bottom to top
- The worker A receives real time feedback, motivational messages.
- The worker A can see the live task statistics and performance of his colleagues can motivate him to participate and do well in the task.
Over the period of time the system can build a network graph of workers and requestors who work well together. This can be further extended to build the teams that can work together on highly complex tasks.
Limitations & Challenges
- Initial training & coordination overhead for workers and experts