Milestone 4 taskforce taskclarity automatically

From crowdresearch
Jump to: navigation, search

Improving the clarity of Microtasks Automatically

While involving workers and requesters in the process of task design might prove to be fruitful in order to ensure that requesters learn to design tasks in a worker-friendly manner, this may give rise to several limitations. Some of these are listed below: Time Overhead. Requesters need to wait until their task has been screened by workers, before the task can be deployed for gathering responses from the crowd. Workers need to wait before a potentially interesting task is made accessible for their participation. Costs Overhead. Crowd workers or mediators who are responsible for screening tasks need to be compensated in some way. Assuming, compensation is typically a monetary means, this leads to additional expenditure.

In order to overcome the aforementioned limitations, we propose an automatic approach to improve the clarity of crowdsourced microtasks. The main components in our prescribed method are : (i) a simplifier, and (ii) a task validator , and (iii) a task classifier.

(i) Automatic Task Description Simplifier:

In the light of the fact that several crowdsourced tasks are not restricted to particular geographic boundaries, we argue that proficiency in the English language (or other languages) of various crowd workers varies greatly. Due to this reason, crowd workers may not be adept at interpreting tasks accurately despite the presence of reasonably clear instructions. In order to avoid the potential bias generated through misinterpretations of a given task, we propose to use the Task Description Simplifier (TDS). The TDS relies on dictionaries such as WordNet in order to simplify words by replacing difficult words with more comprehensible synonyms. Difficult words are determined through NLP methods and heuristics. We can ensure that the reformulated task description does not lose its integrity in terms of fluency, and thereby the message conveyed, by automatically checking for breaches in the grammar. In the absence of difficult words in the task description, no changes are made.

(ii) Automatic Task Validator:

In the next step, the reformulated task descriptions are validated in order to ensure that the task design is not malformed and is fair with respect to the workers, i,e, with respect to the time allocated for task completion, and so forth.

(iii) Automatic Task Classifier:

The automatic task classifier is a supervised model that can classify a given task based on its task description and settings into one of the pre-determined classes of tasks. By doing so and annotation the task itself with a particular class , workers can easily and quickly find a task that is of their interest. This is important due to the following reasons: Workers who do not find tasks that they prefer to work on or are good at, nevertheless attempt and complete other microtasks. This prevents better qualified workers to work on the tasks, resulting in a potential loss w.r.t. to the time required for task completion and/or the quality of the results produced.