Milestone 3 TestSet PowerIdea 1: Moderator

From crowdresearch
Jump to: navigation, search

This is a possible solution for trust issues that exist on most crowd sourcing platforms. A moderator is someone who performs the same function that a moderator performs on a forum. Most forums allow those posters who post more often or are upvoted more often to have more power in the forum. This idea is related to the "Platform Operator" idea in the post "Design notes for a future crowd work market" on Medium.

Need For Promotion

Turkers on Amazon Mechanical Turk perform multiple jobs to get any decent sum of money as payment. There are many workers on AMT who spend a considerable amount of time slowly accumulating reputation on the site by doing jobs. However, beyond whatever respect these people get from the specific Requesters they help out, there is no concept of 'promotion' in Amazon Mechanical Turk. A natural response for anyone in a system like this is to desire to be given greater respect depending on the time they have spent on the platform.

Peer Review

In Science, academics regularly churn out papers which then have to undergo stringent checks by their colleagues before the papers are published. Frequently, papers are sent back to the scientists for things to be added/changed. The idea here, though slightly outdated at this point, and under regular debate, is that a scientist's peers evaluates their ideas and judges whether they are worth their salt.

There are multiple ways this occurs, usually, the reviewer is unknown to the author. However, of late, open source journals have been making public reviewers to make them more accountable to the task. There are papers have have slipped under the radar and are propagating bad ideas in the world, and this is not at all useful. Depending on trial runs on the crowdsourcing platform, we could choose between these ideas.

Peers as Moderators

The playing field in AMT is not as equal (or maybe more equal?) as science is expected to be. Some requesters and Turkers will always be more dedicated than others (though the cheaters are regularly filtered out). Effectively this makes it a good idea to have them act as moderators, possible within the respective job areas they are most accomplished in.

Setting a filter on the experience shouldn't be hard, it would be purely dependent on the amount of time people spend on the platform. Assuming the amount of time spent on AMT follows a distribution with heavy tails, one could set the limit on who can be a moderator at people who have spent time on the medium more than 80% of the rest of the group doing tasks in that field.

As a final thought, hopefully this won't happen.