Milestone 3 ZSpace PowerIdea 1: Result Submission

From crowdresearch
Jump to: navigation, search

Current Situation

At present the result to a HIT is submitted to the requester directly who then has to sieve out the bad ones. This requires a lot of time and patience. It would be really helpful if the submission were directed through some system that partially verifies the result before it reaches the requester, saving his time and energy.

Idea

Automated

When the results are submitted, automated system can check for the correct results and rate them. This shall be really effective in case of HITs that require a definitive response. However this can be expanded to incorporate the HITs that require open-ended responses, by looking for certain keyword in the results.

Possible Challenges

This cannot be used for the HITs that focus on creativity or designing.

Recursive

Same task can be assigned to more than one worker and a task is approved if same result is produced by majority of the workers assigned with the task. This system again works well with the HITs requiring well-defined answers like captcha recognition, mathematical expressions.

Possible Challenges

This approach assumes the democratic principle that the majority tends to be right. However this may not be the situation always. For example: In HITs requiring the Turker to translate a paragraph, the majority tends to use the less reliable but faster way of using Google Translate thus giving an answer which is wrong even though it is the majority one.

Iterative

In iterative solution, the results of one Turker are assigned as task to another Turker who further improvises the task or rates the previously done task. This way ensures that the final result obtained by the requester has better quality. This process can work well where the above mentioned processes fail.

Possible Challenges

The problem occurs when a less experienced Turker is entrusted with task of improving more experienced Turker's task. There is a chances of the task being deteriorated rather than being improvised.

Possible Benefits

  • The end result has a higher probability of have been attempted correctly.Thus, the requester has to reassign the HIT less often.
  • The time in accessing the results is reduced for the requester.