View source for Milestone 5 TuringMachine Storyboard: CrowdSourced HITs creation using Task Design Dashboard & Recommendation System

==Influence and Related work ==
MTurk task design templates are limited in functionality and flexibility. It has often seen that the ambitious instructions and confusing task flow affects workers productivity.  [http://crowdresearch.stanford.edu/w/img_auth.php/7/7f/The_future_of_crowd_work_%28private%29.pdf The Future of Work, Kittur et.al. 2013], authors highlight one of the complains: ''“Too often the job itself is badly designed or is messed up and there is a degree of misunderstanding between the worker and the job engineer.”''  Furthermore, the paper elaborates 12 different key aspects of crowd-computing design. One of them Crowd Work Process focuses on ''How might we design the workflow and control the quality of work?'' 
[[File:Neil23.png|400px|left|Class]] [[File:future.png|625px|center|Future of Crowd Work]]

Past research has shown that the lack of skills, misunderstanding, and irresponsible behaviors are some of the main causes of low quality submissions. Various techniques can be used to increase the quality of work. Templates can be one of the possible solutions to design clear instructions. The [http://crowdresearch.stanford.edu/w/index.php?title=Milestone_3_PixelPerfect_PowerIdea_2:_Task_Templates  PixelPerfect team]  has proposed an excellent idea of new template framework.  Figure below highlights the some of these techniques. The paper [http://crowdresearch.stanford.edu/w/img_auth.php/f/f2/Shepherding_the_crowd_%28private%29.pdf ''Shepherding the crowd yields better work, Dow et.al. 2012'' ] proposed an idea of Rubric that can be helpful for quality measurement. 
 


Sky's [http://crowdresearch.stanford.edu/w/index.php?title=Milestone_3_Sky_TrustIdea_1:_Multilevel_Result_Delivery_system_to_requesters Multilevel Result Delivery system to requesters] idea propose Pre-defined Filters to increase work's quality. We have Integrated Some type of these filters as drag-able widgets in the dashboard. We propose Validator template including these filters for quality control and making result filtering more efficient. Furthermore, by using these filters we can implement multilevel delivery results but its out of this submission's scope.

==<u>Link to the Mockups </u>==
[http://crowdresearch.stanford.edu/w/index.php?title=Milestone_5_TuringMachine_Mockup:_CrowdSourced_HITs_creation_using_Task_Design_Dashboard_%26_Recommendation_System CrowdSourced HITs creation using Task Design Dashboard & Recommendation System]

==<u>Scenario 1: Task Design using Dashboard </u>==

<u>  Problem Design, Setting, & Environment </u>
[[File:Neilgaikwad1ProblemDef.png|800px|center|problemDef]]

<u>  Story </u>
[[File:neilgaikwadIdea1.2.png|1300px|center|Dashboard]]

<u>[http://crowdresearch.stanford.edu/w/index.php?title=Milestone_5_TuringMachine_Mockup:_CrowdSourced_HITs_creation_using_Task_Design_Dashboard_%26_Recommendation_System Task Design Dashboard: Task Categorization & Selection]</u> 

[[File:TaskDesignBordneilGaikwad1.png|200px|Dashboard]] [[File:NeilGaikwadTaskCat.png|200px|Dashboard]] [[File:Gaikwad122.png|200px|Dashboard]] [[File:NeilgaikwadCode.png|200px|Class]]  [[File:NeilgaikwadEWF.png|200px|Class]] [[File:NeilgaikwadAnalytics.png|200px|Class]]  [[File:Fig_2.2_dashboardGaikwad.png|200px|Class]]

[[File:Vishnu3.png|1300px|center|Task-Categorization]]

==<u>Scenarios 2: Automated Verification </u> ==
<u> Requestor's Story </u>

[[File:BitaScenario2.png|1400px|center|Dashboard]]

<u> Worker's Story</u>
<u>Scenarios 2: Automated Verification </u> 
[[File:Bita2.png|1400px|center|Dashboard]]


== <u>Scenarios 3: Validator Scripts  </u> ==

[[File:Problem_sett_script.png|center|800px|alt text]]

[[File: Programmer_script_all.png|center|1200px|alt text]]

[[File:validatorScript.png|center|800px|alt text]]

== <u>Scenarios 4: Golden Questions </u>==

[[File:Golden_question_all_2.png|center|1400px|alt text]]

[[File:Golden_question.png|center|800px|alt text]]

== <u>Scenarios 5: Arbitrator  Questions </u>==

[[File:Arbirator_questions_all.png|center|1200px|alt text]]

[[File:ArbitratorQ.png|center|500px|alt text]]

== <u>Scenarios 6: Condition validator </u>==

[[File:Condition_val_all.png|center|1200pxpx|alt text]]

[[File:ConditionValidator.png|center|500px|alt text]]

== <u>Scenarios 7: Connector Widgets </u>==

[[File:Widget_connector_all.png|center|1200px|alt text]]

[[File:Connector_widget.png|center|700px|alt text]]

== <u>Scenarios 8: Validator Template </u>==

[[File:Widget_wizard_template_all.png|center|1400pxpx|alt text]]

[[File:ValidatorTemp.png|center|800px|alt text]]

== <u>Scenarios 9: Widget Wizard Studio </u> ==

[[File:WidgetWizardStudio.png|center|800px|alt text]]

== <u>Scenarios 10: Assign moderators </u> ==

[[File:Assign_moderator.png|800px|center|alt text]]

== <u>Scenarios 11: Select dispatch strategy </u>==

[[File:Dispatch_sterategy.png|700px|center|alt text]]


==<u>Behind the Interface: Algorithm NLP automatic verifier widget</u> ==
Based on the RUBRIC idea in paper : RUBRIC: A flexible tool for automated checking of conformance to requirement boilerplates, we propose that by using a NLP verifier boilerplate, based on Rupp's requirement, we can develop a NLP module, based on set of predefined words selected by the HIT designer to automate the verification process. The idea is to let the HIT designer activate an automatic verifier widget, and by that, he/she can select 6 words for each part of the boilerplate from a "bag of words" or vocabulary set, and then submits the questions. The requester has to creat a couple of possible answers to his question. The questions are embedded in the HIT. When the worker is doing the HIT, is prompted by the verifiable questions, and then, for the answer, the same NLP processor boilerplate comes into play, and selects the answer from a set of limited words for each 6 blanks of the boilerplate, and creates an answer. The verifier now easily can match the requester possible answers to the worker answers and verifies the answers. ''Please see, Arora et al. "RUBRIC: A flexible tool for automated checking of conformance to requirement boilerplates." Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. ACM, 2013''