Milestone 3 TestSet TrustIdea 2: NoRejectionPossible
Rejection is always bad, it hurts! But should someone be rejected even if the person worked sincerely? How is the work quality measured?
Goal of the design: Respect the cognitive effort and the time spent
One of the most popular complaints about amazon mechanical turk is that workers need to be respected. But in labor markets like this the key problem happens to be the fact that the ratio of workers to requesters is extremely high and this results in the requesters making the call about how much to pay, who all to pay (i.e which tasks get approved) and in many cases when to pay. (There are often complaints about late payments). Now since this ratio of workers to requesters is not going to change anytime soon because of the inherent design of the market it is essentially key to be able to respect the workers.
The key idea governing that is "No work should be rejected". The rationale behind such a decision is that an individual spent his cognitive ability and time to accomplish a task and in most cases, even if the monetary gain is minimal, lots of workers depend on the earnings from Amazon Mechanical Turk. This will lead to more effective trust on from the part of the workers to the requesters. Workers should be able to concentrate on the work more and not be worried about payment policies.
How does this address the Goal? Is this unfair on the requesters?
A lot of Turkers in general took great satisfaction in doing jobs that involved something to do with research. In effect, they felt partly compensated for doing something that we think can be considered "worthwhile".
The requesters are not affected by the above category.
Now an obvious challenge to this is how can one ensure that the quality of the work is maintained? There are many ways to approach the problem. Amazon can have a clear policy of what work is rejected. Suppose the task was to recognize human faces in images. An outlier or a bad worker can be easily distinguished from the others if the person's selection lies very far away from the average responses that have been received. Essentially the people whose answers lie away from 3 standard deviations from the mean can be rejected and that too would require a very clear explanation by the requester.
Requesters should be penalized for unfair rejections and should a requester reject a work, a group of dedicated individuals would look into the issue and if at all the work output of the turker can be considered valid the requesters should bear the consequences severely. This can include bad reputation and thus less power and privilege related to posting work in the future. These methods will be a step towards gaining workers trust and can lead to an environment where workers get rewarded for time and cognitive effort generously. This would make the workers feel that the whole process was worthwhile for the individual.