Milestone 3 TuringMachine
Background Research and Foundation for Brainstorming
In this assignment we have conducted the brainstorming activity based on learnings from the milestone 1 and Milestone 2. Furthermore, we build our work on The Future of Work, Kittur etal 2013, a suggested background reading and Genomes of Collective Intelligence Framework, Malone etal 2010.
The Future of Crowd Work, paper highlights 12 different key aspects of crowd-computing design. These aspects are divided into three major categories:
- Crowd Computation: How might we design the crowd-computing platforms using Artificial Intelligence and machine learning?
- Workers: How might we motivate and reward the workers? How might we provide them credentials and reputation?
- Crowd Work Process: How might we design the workflow and control the quality of work?
Genomes of Collective Intelligence The framework provides essential building blocks require to design the crowd computing systems based on following elements:
- Goal: What are the main objective of the system?
- Staffing: Who are the workers and requestors? Where do they come from?
- Incentives: What motivates people to participate and contribute?
- Process: How is it being done? How can machine intelligence and mathematics help establish the task workflow and harness wisdom of crowd?
We break down the brainstormed ideas
Quality of Work
- Distribution of Work according to interests of workers: Previous research has shown that individuals are more committed to work if the tasks they are performing fall under their area of interests. We propose intelligent system to 1) Maintain Master Profile of Tasks according to functionalities. 2) Maintain Master Profile of Workers' skills, experience, Money, and interests. 3) Develop Clustering/Classification algorithm to maintain categorize of profiles. 4) A perfect matching/recommender algorithm to match workers profile with tasks
- Supervisors New category of workers for quality assurance: 1) Introduce concept or virtual managers i.e. supervisors who will monitor the quality and progress of the work. 2) Requestor will delegate the responsibility to supervisors. This will work similar to AKKA actor based concurrent model where supervisor can monitor the pipelines and provide on demand feedback to workers.
- Communication, task design, & training for Workers and Requestors based on ONION RING strategy It has been seen that unclear instructions from the requestors are also responsible for the low quality submissions. To overcome this challenge: 1) An automated tutor system can be used to train requestors to design the workflows 2) Imagine requestors at the center layer of the ONION, the core ring. While designing the task, they can recruit the fraction of supervisors from the next layer as a beta users 3) Based on supervisors feedback requestor can redesign the task during this beta phase and make sure that instructions and guidelines are clear. 4) Supervisors can then reach out to further layers in the onion to get the task done 5) Dashboard and visualization will highlight the progress of the task and failure of the nodes (workers) 5) If some workers (nodes) fail then system can use Error Kernal strategy to mitigate the risk. 6) Workers can communicate with supervisors or with each others by posting the questions on Task Dashboard. 7) Workers will vote on the top questions and concerns 8) Requestors/Supervisors can answer the questions as Teaching Assistants do at the office hours.
- Proof-of-work Protocol (POW) It is challenging for requestors to find out who is doing the actual work. The Requestors can set up Golden Questions to traceback the workers. However, this approach is not efficient. Instead, the system can generate a digital signature protocol or authorize users depending on the their facebook/twitter ids or SNN Tax Identification Numbers. In such cases workers are less likely to share this private information with others and delegate the work to someone else.
Reputation, Transparency and Incentive
- Leaderboard profile for Requestors How might we increase the reputation of requestors? 1) Motivate Requestors to be transparent and attract quality workers. 2) Create Requestors' profile highlighting their track-records. 3) Workers can vote for top requestors who are providing clear instructions about the tasks and fairness. 4) Workers can trust the requestors who are high ranked and provide good value for their time
- Leaderboard profile for Workers How might we increase the reputation of workers? 1) Motivate workers by being recognized in the community 2) Provide incentive to reach to top using hierarchy 3) Provide value to their commitment and ability to get things done 4) Requestors can trust the worker's profile based on making payments or recruiting them
- Social Network, Reference based recruitment: 1) Requestors can recommend workers to other requestors 2) Supervisors can recommend workers to other requestors
- Worker's/Requestor's background check 1) System can leverage APIs provided by background checking industry to run automated checks to ensure the crowd worker/requestor doesn't come from any criminal. This will be an important component for sustaining stable and trustful crowdsourcing community.
- Payment Timeline A system where the requestor should initially pay for the work could be introduced. The amount is stored as a buffer till work is completed. If the requestor is satisfied with the work then he gives an approval and then the money is transferred from the buffer to the worker's account. Otherwise it is transferred back to the requestor's account.
Trust as a function of time for Interdependent and Complex Tasks
- Swift Trust Meyerson etal 1996 proposed the theory of Swift Trust. According to theory team members assume such trust initially and update their beliefs over the period of time. These team members have limited history of working together under strict time constraints. There are various parameters responsible for establishing the swift trust. Theory suggests that team working under time pressure make greater use of category-driven rather than evidence-driven information processing. Asynchronous communication channels such as forums can help store, retrieve, share, and use the task specific knowledge whenever required. This is a vital part for developing the trust. An automated AI based algorithm can use theoretical matrix and worker's profile to recommend combinations of personalities that have potential for developing such trust.
- Reserve Price Payment, Legal Protection 1)Determine minimum hourly wage or reserve price for the tasks 2) According to legal systems around the world establish the minimum payment benchmark for corresponding geographic location.
- Guaranteed Payment if Review is Delayed 1) Requestor posts the task profile on the Task Board, which includes Task completion deadline, Review deadline, Payment deadline, Review Clock Start Time, Payment. 2) A worker views the Task Profile and decide to work on it; completes and submits the work. 3) Both agrees on the deadlines mentioned in the Task Profiles. 4) If requestor missed the review deadline, money gets transferred to the worker.
- Appeal Against Rejection Over the period of time as the Ranking system described above evolves, we can create a committee of reviewers from the top Requestors and Workers. The committee will be assigned for specific category of tasks. 2) Members serving in the committee will use the Profile and Quality parameter matrix associated with the task to review the submission. Each worker could appeal 5-10 reviews per month. If committed calls against worker, he looses the review and his ranking goes down, whereas if committee calls against requestors his ranking and reputation goes down.
- MOOC Structure for Knowledge Sharing Knowledge is Power for Workers. They can use in build forum to discuss and share ideas about particular task. 2) Automated NLP algorithm can parse through their comments and search for cheating/spamming incidences. 3) Workers can monitor the forum and upvote the good posts.
- Collaboration & Equilibrium In complex creative tasks, workers and requestors can collaborate to design the solution for the task. They can adhere the task profile guidelines and work together synchronously. Automated algorithm can calculate the cost of positive and negative externalities and recommend the collaborations that are profitable for participating agents.
- Second Price Auction for fair Pricing Most of the times requestors controls the prices. However, reserve price will help establish minimum payment rule. Furthermore, workers can participate in the second price auction and provide true valuation for the task. Automated algorithm can collect the price inputs i.e. bids from different workers and approximated prices from the requesters. Then the algorithm can find trade off between demand and supply to determine prices zero, one, or two standard deviation away from the expected value.
- Access to Requestor's payment profile The historical data about the base prices provided by the requestor should be associated with his profile. This will reduce the monopoly of requestors and allow workers to evaluate the tasks based on amount of money they expect to make.
- Hall of Shame, Reporting Abuse Automated system to block and list workers/requestors who are breaking the system protocols. This will lower the rank of worker/requestor and prevent them from participating in high paid tasks.
- Requestors hiring workers Requestors can collaborate with each other and form a team. They can select workers based on employment profile and pay them at predefined salary for a longer period of time. Depending on how well the worker has done jobs given beforehand, the requestors can decide whether to hire the worker or not. This will provide workers with a stable payment stream and requestors with a diverse task force.
- Pairwise stability, Conflict Resolution, Nash Equilibrium Over the period of time requestors and workers will establish professional relationships. Incentives or payment structure can be design such that these relationships flourish and no agent (worker/requestor) gains high rewards from severing the relationship when projects are in progress. This will help preventing potential chaos in the marketplace. Furthermore, the committee of top requestors and workers can help resolve the conflicts and establish the protocols.
- Bayesian Learning, Random Action Markov Processes. Who has influence? Understanding the logs data Network and Computer Science theory can help identify the influential workers or requestors in the eco-system. AI/ML agent can observe various communications patterns between workers-workers and requestors-workers. Recording logs of activities (without violating privacy measures) will help understand if there is any Homophily between various agents. For instance, to understand the patterns in the Turkopticon ratings (requestors, comm, pay, fair, fast, reviews), we have retrieved Turkopticon data using the API. The figure shows the sample screenshot of the dataset. Below we show the distribution of Reviews. The Review data follow the heavy tail right skewed distribution, which is very similar to most of social networks datasets. We can see the slow decay as reviews increase. Now, interesting question is : Why some requestors got so many reviews and other got few? Were those requestors good or bad? Answers to such questions can help us understand how community structure evolves over the period of time.
Dive Deeper into Specific Ideas
- Milestone 3 TuringMachine TrustIdea 1: Distribution of work according to workers' interests - http://crowdresearch.meteor.com/posts/TaNbEkmeoSSiv5vDB
- Milestone 3 TuringMachine TrustIdea 2: The leaderboard system based on performance - http://crowdresearch.meteor.com/posts/ShEsGS3tx2Gdq9noJ
- Milestone 3 TuringMachine PowerIdea 1: Guaranteed Payment if Review is Delayed - http://crowdresearch.meteor.com/posts/HEiG2GyJWLFPzThYc
- Milestone 3 TuringMachine PowerIdea 2: Fair Price Mechanism - http://crowdresearch.meteor.com/posts/qsv2bcRNjSL3PC8sC