Milestone 4 TuringMachine: Recommendation System & Task Design Dashboard

From crowdresearch
Revision as of 05:37, 12 July 2015 by Avrimmit (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search


Neil Gaikwad

Influence and Related work

The current MTurk task design templates are limited in functionality and flexibility. It has often seen that the ambitious instructions and confusing task flow affects workers productivity. The Future of Work, Kittur 2013, authors highlight one of the complains: “Too often the job itself is badly designed or is messed up and there is a degree of misunderstanding between the worker and the job engineer.” Furthermore, the paper elaborates 12 different key aspects of crowd-computing design. One of them Crowd Work Process focuses on How might we design the workflow and control the quality of work?

Future of Crowd Work

Templates can be one of the possible solutions to design clear instructions. The PixelPerfect team has proposed an excellent idea of new template framework. In addition, the paper Shepherding the crowd yields better work, Dow 2012 proposes idea of Rubric that can be helpful for quality measurement.


Crowdsourcing the Template Design Using the Task Design Dashboard on Web-browser and iPad

In what follows, we highlight our vision about the task design system based on following elements:

  • Task Design Dashboard with Drag and Drop controls
  • Standardized template library based on Recommendation System Algorithm
  • Reward based Crowdsourced template creation process

Motivation for Voting & Sharing:

  • Intrinsic motivation: Workers want to do well at the job and avoid rejections, whereas requestors want to achieve high quality submissions. Both know that good templates can help reduce the misunderstanding and make system more efficient. Therefore, they would like promote the high quality templets and maximize the social welfare .
  • Extrinsic motivation: I proposed a ranking & leaderboard system that encourages the requestor to share the templets with the community. TemplateRanking is one of the parameters that determines the requestor's global reputation. The parameter has positive weights if maximum number of requestors use the requestor's template. This process is very similar to citing the research paper. In addition, the system can give badges to requestors/workers who reviewed and voted for the templates.



Design the task using a web browser or iPad: Drag & Drop Controls with the Task Design Board

  • Standardized dashboard will provide the drag and drop controls required to design a wide range of crowdsourcing tasks in a fraction of time.
  • The requestor doesn't need to know any programming; however, technical users can access the programming board to design the task.
  • The requestor can design the task from the scratch or using various templates from the existing library. Templates are categorized by type of the tasks.
  • Requestor can design the task profile and rubric to measure the quality of the task. He can also customize the standard rubric templates from the library. .
  • Finally, the requestor can share the templates & rubrics with the community, earn rewards, and further enhance his reputation. Other requestors and workers can vote on the high quality templates. An automated algorithm can track the features of the best templates and turn it into a best recommended practices/guidelines.
  • The interface can be extended to mobile devices as well.
  • Figure 1 highlights the task design dashboard for the requestors.


Context: Writing Task

  • Figure 2 highlights the design of the Writing Review Task for IPhone 6.
  • The requestor drags and drops the components and rubric highlighted in GREEN
  • Depending on Task Context, the system recommends the top templates from the library

Automated Task Categorization with Recommendation System

We enhance our vision and idea from milestone 3 to automated task categorization and recommendation process. We had propose the system that will:

  • Maintain the Master Profile Map of Tasks according to functionalities. The figure below highlights various problems crowdsourcing systems can solve.
  • Maintain Master Profile Map of Workers' skills, experience, salary history, salary expectations, and interests.
  • Execute Clustering/Classification algorithm to maintain categorize of profiles.

Execution: Recommendation Systems


Execution Details: Combining Global and Local Effects


Algorithm: Latent Factor Model for Workers to Task Recommendation


Example: Categorized Task Recommendation to the Worker


Limitations & Challenges

  • Template Design: Privacy issue with sharing templates
  • Recommender Engine: Initial phase challenge to obtain the ratings data for training the algorithm