Milestone 3 PixelPerfect

From crowdresearch
Jump to: navigation, search

Team Pixel Perfect's submission for Milestone 3.

Initial Brainstorm

We started with the need cloud for workers and requesters and worked from there. Picture12.png
In the end, we ended with an accumulation of ideas for a system. Our brainstorm slides for a system can be seen here File:PixelPerfectMileStone 3.pdf

Trust-related Ideas

How do requesters trust workers and vice versa?


Picture5.png
  1. Strict identification system : Each worker or requester is requested to submit valid ID and proofs at time of application, to verify their genuineness. This may be a combination of Social Security Number, Voting Card, Driving License, Ration Card, Aadhaar card (in India) etc. Worker details are by default, private and disclosed only if the worker wishes to. Each application is reviewed before users can join the platform.
  2. Worker-Requester Social Graph : We use the Facebook social graph model to conceptualize worker-requester relations and compatibility. We develop a crowdworker graph (bipartite for now i.e no relations within workers or requesters) whose nodes are requester and workers, and compatibility between them modeled by the edge weights.
Compatibility develops through a two-way feedback system between workers and requesters after completion of a task.
It is centralized around a simple Yes/No question :
Would you like to do a task from this requester again? [For workers]
Would you delegate a task to this worker again? [For requesters]
Compatibility plays a role in how visible a given requester’s HIT is, to various workers. Workers having higher compatibility with the requester are more likely to see the HIT and therefore take it up.


How do requesters trust good results on their tasks?


  1. Demographics check : The platform shall have a mechanism where only workers with appropriate demographic are eligible for the task. These might be helpful in certain HITs which are demographic specific (eg. Surveys involving only native American women). Such tasks can be filled anonymously, with the system performing the demographic check based on the identification provided. If the tasks require disclosure of details, workers can only take up such tasks if they willing to forgo anonymity. This preserves the choice to be anonymous.
  2. Task templates : Good task design can go a long way in reducing spam in submissions. These can be done by including questions which require correct answering before completing rest of the survey/task. Correct answering forces spammers to read and appreciate the content, spending more of their time, and hence discouraging bad replies. Therefore, pre-designed task templates for different task classes are available for requester use, thus standardizing task quality. Some templates may have scripts/tools to help with evaluation. This helps requesters develop HITs quickly and get more trustworthy results. Workers also benefit from consistent clarity of standardized tasks.

How do requesters trust they will get their results on time?


  1. HIT visibility algorithm : We model HITs analogous to posts on Facebook, liking or commenting a post is analogous to accepting a HIT by a worker. On Facebook, the posts visible to users are determined by the EdgeRank algorithm, which is a function of affinity, weight and time decay.
On the platform we develop, the HIT visibility is determined by :
  • Worker-requester compatibility determined from the Social Graph (Affinity)
  • Importance of task to worker determined from skillset, task preference and time/money payoff (Weight)
  • Tasks whose deadlines are approaching given more preference (Time Decay)
Finally a randomized algorithm scrambles and sorts these HITs and chooses which ones are visible to the worker.
Thus, requesters can have confidence in receiving their results on time.

How do requesters trust the skillset and expertise of workers?


Picture13.png
  1. Skill levels : Workers are free to register for certain skills and fields of expertise, although a machine learning algorithm can automatically do the same, by analyzing the tasks undertaken. Each worker will have a radar graph associated with him or her, which shows at a glance, expertise levels in the registered skillsets. Skill levels grow with successful and accurate completion of tasks in that skill. Requesters can adjust lower and upper bounds on these skill levels for their tasks, to filter qualified people.

How do workers trust they are not being exploited?


  1. Report exploitation system : A common issue is work theft, where requesters do not acknowledge work submitted, however incorporate it in the results. This happens because requesters enjoy intellectual property rights over the work submitted and not the workers. A workaround might be to allow workers to submit “Exploitation Forms” to the requester with appropriate evidence attached. These might be analysed by moderators who may decrease the rating of the requester if found guilty. Workers may be penalized if found to submit false claims. This legal facility may be included as part of the terms and conditions of the platform.
  2. Penalty system : Workers would be compensated for the delay in the payment as quite a large percentage of people rely on these platforms for their daily living. The penalty amount would be a function of delay time and task amount.

How do workers trust they will be gainfully employed at all times?


  1. Rating system : Rating is assigned to each worker or requester and is a function of several factors. Rating is indicative of quality of work done/given and credibility.
  • Workers earn rating for accurate and fast completion of tasks, skill level and knowledge, time spent on platform, compatibility with requesters.
  • Requesters ratings are similar to that on Turkopticon – generosity, promptness, professionalism, fairness and communicability, task clarity and feedback history.
A common issue on current crowdsourcing platforms is, certain weeks generate good payoff for workers while other weeks don’t. Auto-generated tasks whose solutions are known can be made available at all times. During idle scenarios (low task demand), these can be taken up for work. This enables workers to improve their experience and ratings during downtime, which in turn makes them eligible for tougher and more intricate tasks later on.

How do new workers get started quickly and earn the trust of their counterparts?


  1. Tutorial Mode : Consider workers just starting out with the platform and having little experience. Lot of tasks might not be available to them at this point, since they are relatively unknown. Therefore, computer generated tasks whose solutions are known, will be visible to them – we can gauge their skillset, accuracy and efficiency. This acts as an initial seed for rating/experience.

Power-related Ideas

What powers do workers have to feel respected and voice their opinion?


  1. Two-Stage Feedback : There will be a two-stage feedback for a particular HIT and requester – one during completion of task and the other, after its completion.
    • First stage of feedback for workers involves them rating the clarity of the instructions, if the task was appropriately priced and whether optimum time was given for the task. Also a single question 'Would you like to work for this requester again?' signifying the overall experience is asked.
    • Second stage involves requesters being rated on the promptness of their payment and their professionalism in general (mass rejections, feedback given to workers)
    Moreover, workers can send "exploitation forms" as mentioned above in case they feel they have been wronged.
  2. Comment Feature : Each HIT will have a comment section, where workers can suggest improvement or provide tips on task completion, or even publicly declare their overall experience with that HIT and requester. Requesters may use this feature to positively critique task design and other aspects.

What powers do workers have to find HITs quickly?


  1. Requester and Worker Profiles : Requester and workers could have a profile page for purposes of communication and knowledge. This is similar to the concept of a wall on Facebook.
    • Requester profiles may carry information about rating, tier, level and experience points, typical allotted work, contact information (email/github/IRC) and perhaps a button to IM. An option to subscribe to this requester may be allowed – to increase visibility of tasks from this requester.
    • Worker profile pages carry information about rating, tier, level, skillset and task preferences. Registering for more skills, can increase task visibility.
  2. Dashboard sorting  : Workers can sort HITs on the dashboard according to the level of complexity, skills required, estimated time taken to complete a HIT, nature of task etc. This can help increase productivity. Sorting by payoff and requester is a debatable issue.

What powers do users of the platform gain with time?


  1. Level system : Experience points are earned through worker or requester specific actions. Earning a set amount of experience allows a worker or requester to level up. Levels help unlock higher privileges and productivity tools. Higher levels are indicative of higher quantity of work done/given.
  2. Platform moderators : Top-level worker/requesters have moderator like privileges. Moderators have authority to flag comments and users and resolve exploitation disputes. They are compensated for moderation duties as well.
    Picture15.png
  3. Task Classification : We planned to classify tasks first broadly like information retrieval, social media, surveys, OCR tasks. These could be further classified into simple and complex tasks and so on, to form a ontology tree. The ontology tree can be arranged/organised based on nature of task (broad categories listed above), skillset (mathematical/visual/logical) and complexity. Moderators can change the ontology tree and classify existing tasks with keywords.
  4. Template creation : In order to write good HITs quickly and easily, templates/wizards may be made available for use. There will be a set of official templates which are added by expert users, subject to formal testing; and a set of templates added by other users. This official/unofficial template model is similar to the homebrew/third party system of apps on a mobile phone.
  5. Advanced Tasks and Tools : High-skilled workers can access tasks having higher complexity and payscale. Further, they get access to tools and scripts designed to increase their productivity - for instance completion of one task loads the next predicted task automatically, without the task selection screen. Prediction is expected to be accurate for long-term users and hence saves time in the long run.
  6. Role in system development : Chosen experts of the platform will be invited to work closely with platform developers, to improve the system based on their experience.

Dive Deeper into Specific Ideas

We present some of our best ideas in depth here :

Trust-related Ideas

Power-related Ideas

Dark Horse idea

Presenting our dark horse idea :-