Difference between revisions of "Milestone 5 TuringMachine Storyboard: CrowdSourced HITs creation using Task Design Dashboard & Recommendation System"

From crowdresearch
Jump to: navigation, search
(Influence and Related work)
(Influence and Related work)
Line 6: Line 6:
 
[[File:Rubricneil.png|400px|center|Class]]
 
[[File:Rubricneil.png|400px|center|Class]]
  
Sky's [http://crowdresearch.stanford.edu/w/index.php?title=Milestone_3_Sky_TrustIdea_1:_Multilevel_Result_Delivery_system_to_requesters Multilevel Result Delivery system to requesters] idea propose Pre-defined Filters to increase work's quality. We have Integrated Some type of these filters as drag-able widgets to dashboard. We propose Validator template including these filters for quality control and making result filtering more efficient. Furthermore, by using these filters we can implement multilevel delivery results but its out of this submission's scope.
+
Sky's [http://crowdresearch.stanford.edu/w/index.php?title=Milestone_3_Sky_TrustIdea_1:_Multilevel_Result_Delivery_system_to_requesters Multilevel Result Delivery system to requesters] idea propose Pre-defined Filters to increase work's quality. We have Integrated Some type of these filters as drag-able widgets in the dashboard. We propose Validator template including these filters for quality control and making result filtering more efficient. Furthermore, by using these filters we can implement multilevel delivery results but its out of this submission's scope.
  
 
==<u>Scenario 1: Task Design using Dashboard </u>==
 
==<u>Scenario 1: Task Design using Dashboard </u>==

Revision as of 02:28, 2 April 2015

Influence and Related work

MTurk task design templates are limited in functionality and flexibility. It has often seen that the ambitious instructions and confusing task flow affects workers productivity. The Future of Work, Kittur et.al. 2013, authors highlight one of the complains: “Too often the job itself is badly designed or is messed up and there is a degree of misunderstanding between the worker and the job engineer.” Furthermore, the paper elaborates 12 different key aspects of crowd-computing design. One of them Crowd Work Process focuses on How might we design the workflow and control the quality of work?

Class
Future of Crowd Work

Templates can be one of the possible solutions to design clear instructions. The PixelPerfect team has proposed an excellent idea of new template framework. In addition, the paper Shepherding the crowd yields better work, Dow et.al. 2012 proposes idea of Rubric that can be helpful for quality measurement.

Class

Sky's Multilevel Result Delivery system to requesters idea propose Pre-defined Filters to increase work's quality. We have Integrated Some type of these filters as drag-able widgets in the dashboard. We propose Validator template including these filters for quality control and making result filtering more efficient. Furthermore, by using these filters we can implement multilevel delivery results but its out of this submission's scope.

Scenario 1: Task Design using Dashboard

Problem Design, Setting, & Environment

problemDef

Story

Dashboard

Task Design Dashboard: Task Categorization & Selection

Dashboard Dashboard Dashboard Class Class Class Class


Scenarios 2: Automated Verification

Requestor's Story

Dashboard

Worker's Story Scenarios 2: Automated Verification

Dashboard


Scenarios 3: Validator Scripts

alt text
alt text
alt text

Scenarios 4: Golden Questions

alt text
alt text

Scenarios 5: Arbitrator Questions

alt text
alt text

Scenarios 6: Condition validator

alt text
alt text

Scenarios 7: Connector Widgets

alt text
alt text

Scenarios 8: Validator Template

alt text
alt text

Scenarios 9: Widget Wizard Studio

alt text

Scenarios 10: Assign moderators

alt text

Scenarios 11: Select dispatch strategy

alt text


Behind the Interface: Algorithm NLP automatic verifier widget

Based on the RUBRIC idea in paper : RUBRIC: A flexible tool for automated checking of conformance to requirement boilerplates, we propose that by using a NLP verifier boilerplate, based on Rupp's requirement, we can develop a NLP module, based on set of predefined words selected by the HIT designer to automate the verification process. The idea is to let the HIT designer activate an automatic verifier widget, and by that, he/she can select 6 words for each part of the boilerplate from a "bag of words" or vocabulary set, and then submits the questions. The requester has to creat a couple of possible answers to his question. The questions are embedded in the HIT. When the worker is doing the HIT, is prompted by the verifiable questions, and then, for the answer, the same NLP processor boilerplate comes into play, and selects the answer from a set of limited words for each 6 blanks of the boilerplate, and creates an answer. The verifier now easily can match the requester possible answers to the worker answers and verifies the answers. Please see, Arora et al. "RUBRIC: A flexible tool for automated checking of conformance to requirement boilerplates." Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. ACM, 2013