Milestone 7 Altamira
Welcome! This is a first draft of the proposal due to time constraints...
A Dynamic Reputation profiler
The system proposed here aims to deal with the aspects of trust between the workers and the requesters through a system of reputation profiles. The motivation behind such a system is the sense of frustration with current crowdsourcing platforms such as mTurk, due to lack of knowledge of the skills or abilities of the other party
Based on our observations, there is a need to recognise the skills of various workers in order for them to be utilised in the best way. Also, the need for a profiling system arises from the fact that the workers and requesters remain largely anonymous and do not know each other well enough for more conducive work.
Tasks are categorised based on skills required for it. Tougher tasks which pay better are at a higher level of expertise. Workers have a profile where their expertise or experience in various skill categories is displayed based on a reputation-based set up. The allotment of tasks is dependent on the task skill requirement and the worker's skills
In order to address the need to improve the trust for better quality in the work produce, the designed features are:
Likes: Allows workers to subscribe to notifications of the work types the worker prefers. Also, it allows requesters to understand the worker in a better way. The categorization of work in various types allows this feature
Task Experience: The worker’s experience and expertise in the a particular category is a crucial aspect of the profile which can help others understand the abilities of the person much better
Ranking of Expertise: The workers with high HIT rates are given ranks as a motivation factor, only when they are in the top half of the group so that the lower ranked ones are not demotivated being ranked low.
In order to evaluate the effectiveness of the workers continuously, a dynamic system of review is adopted as described below: • The tasks are categorized based on the skills required. Each skill category has certain hierarchy which will build the reputation of the worker in it. The worker can move up the levels to get tasks which are more complex and pay well accordingly. The proficiency can be measured through multiple levels for a more complete classification. One such level in one skill category is illustrated in the figure through the first stage.
• In order for a worker to move up the ranks, he/she will undergo a series of testing and evaluation, and based on the performance, the person moves up or down the levels. The movement considers both the accuracy of task completion and the number of similar tasks done before. The evaluation phase has a minimum amount of pay.
• Here, a newbie, who can do the required tasks well, will be able to move up the ranks rather quickly. He should be able to build up the reputation in an easy way once he gets through the evaluation phases.
• Further, with the evaluation being done continuously, even an established pro may move down the ranks if his quality drops steadily over a sustained period of time. This will ensure a merit based system to gauge performance.
• However, if the experienced person fails to meet the minimum level of competency, he will be lead through a short training set with feedback about his work. This will help them correct their mistakes and give them a chance to move up more easily. Such a system is not done for newcomers until they get the experience. So, some amount of credit is given to the past experience.
Testimonials: Reviews for workers or requesters are provided if they have worked together before. In addition, the system provides a “like” option for people if they really like work done to motivate the worker.
The reference section is where you cite prior work that you build upon. If you are aware of existing related research papers, list them here. We also encourage you to borrow ideas from the past submissions (see the meteor links above). Please list the links of the ideas you used to create this proposal (there's no restriction in terms of number of ideas or whether its yours or others'). You can use the following template:
- [Foundation Idea] Link...
- [Feature Idea] Link...
- [Foundation Idea] Link...so on, and so forth...