Micro Prototype AMT
Had to work around cause unable to find how to create qualification tasks (they are present, the interface was very intuitive on how to build one) Also my web programming abilities is rather limited, so I first am working with the built in tools from Amazon Mechanical Turk and can try to implement this system if need be through the developer API.
Milestone 0's main goal is to ensure the requestor that the worker has the ability to complete the work well enough to continue working. This proposes an issue for micro tasks that cannot be broken up into smaller tasks (I'm making that assumption for this prototype). Two milestone 0 options have been proposed and this system will take advantage of both:
- Create a task with the truth already known and can compare that to current answer
- Create a task where the information is not known and had to manually check if the variability was within a good range
One issue is where are these task answers coming from if they are all micro tasks. This is an issue and can be dealt with the requestor perhaps completing some of the work in order to get examples, or just gather some work from workers and then begin the comparisons
Evaluation is a process that may I am hoping to automate more so with the developer API, but I have been having some issues implementing that. Automation can be used for evaluating the work that can be compared to the known answers. Inter reliability is a little more complex, but it may also be possible to implement giving a threshold of a given variability between the current work and previous work.