Milestone 3 Sky TrustIdea 1: Multilevel Result Delivery system to requesters
Multilevel Result Delivery system
Abstract and implementation description
We are going to keep the worker's output until both requester and worker are satisfied by the service. We can have two step for delivering the result to the requesters.
- First step is Filtering stage which they can filter workers based on their criteria. This step is preview of complete data which enable requesters to eliminate obvious low-quality works. This enforces requesters to acquire a firm reason to eliminate workers. list a few of these filters.
- Filter based on pre-defined threshold or characteristic
- Filter based on pre-defined test and pre-defined questions (Predefined The boolean function which can verify that worker answered carefully or randomly )
- Filter based on Minimum completion Time
- We are going to use predefined validator questions and part of training questions (randomly select part of training data that are valuable for requester )
All of these are based on the fact that requesters should define a set of filters prior to running the HIT. And they can not add any additional filter for completed HIT attempts.
NOTE : In these step we are not going to show all of workers output until requester push the next button to go next delivery stage.
NOTE : If requester decided to go next step. They are not going see rejected workers output anymore .
- Second is exporting the result. In exporting sage we just allow the requester to export output of approved workers. Requester might export part of data manually in Filtering stage but its not as easy as before and also its not all . So requesters for reducing overhead tend to just reject those that are not really useful.
NOTE : Export file just contains output of approved workers and can not reject them anymore.
- Reducing rejection rate by requesters
- Improving quality of work by filtering
- Automating Filtering
- Firm reason for rejection
- Balance power among requester and workers