WinterMilestone 3 Xi Chen on: Crowdsourcing rejection appeal
When an rejection happens, if the worker feels that the submitted work doesn't deserve this, he can appeal. This is common in some crowdsourcing marketplaces. But there are various ways of handling such appeal. My idea is to post such appeal as a task to workers&requesters with good reputation and within the same category(or cluster) as the rejected work. And as a crowdsourced task, workers&requesters who make judgements about whether the rejection is reasonable should receive payment. If the final result suggests that this rejection is not reasonable, the requester who rejected the work should pay for this appeal task. And if the worker fails at appeal, he/she should pay for the appeal task.
Advantage of The System
- Block spammers' unreasonable appeal:
- For micro-tasks like MTurk, workers mostly receive little money when finishing one HIT. So if a spammer gets rejected, he/she would not take a risk at appealing since if fails, that can costs more than he/she could earn.
- Don't take much time and energy of either side
- If a worker feels getting rejected unfairly, he/she could simply begin a appeal process. And the following process is taken care of other workers&requesters. That worker can get final results without paying attention to the process. Same for the requester who makes the rejection
- Decrease unreasonable/careless/random rejection
- As shown above, if a requester rejects a worker unfairly and that worker successfully appeals, the requester would pay extra fee to make up for the rejection. So requesters would be more careful when rating submitted work.
Disadvantage of The System
- Takes time to finish an appeal
- Since there may not be enough available workers&requesters of the same cluster with good reputation, it can take some time to gather enough judgements and make a final judgement.
@xi.chen: Xi Chen