Milestone 7 Mustang

From crowdresearch
Jump to: navigation, search

Methodologies to improve current crowdsourcing systems by adding functionality for better collaboration and fair compensation.

Authors: Achal Varma, Vignesh Vishwanathan


This paper looks into improving the current crowdsourcing systems by utilizing some of the features that were suggested during Milestone 2 and 3. The major criteria for our selection is the amount of impact the feature creates when added to the new crowdsourcing model that we are collectively in the process of creating. Two ideas that we think has a lot of promise are the Kickstarter model of pricing the work with a resubmission feature and a Github style “Issue” tracker with adaptable messaging features. While these ideas are not novel, they are something that we feel would positively impact the crowdsourcing model. We aim to accurately predict the use cases of our users but ideally we will learn more about our users during the testing phase to fine tune our current ideas.


All current crowdsourcing systems follow the same rough model - requesters post tasks and depending on past experience, workers are allowed to take on said tasks for monetary compensation. Most problems that arise in this system are due to lack of trust between both parties and an imbalance of power. We feel that better communication and a fair compensation model will allow the parties to overcome these shortcomings.


  • Kickstarter

An issue most supply-demand powered platforms invariably run into is an imbalance between supply and demand. In the case of crowdsourcing marketplaces, requesters find it difficult to get work submissions quickly, and with the current average crowdsourcing marketplace model, it’s a challenge to give workers more incentive to complete a task quickly and accurately. We came up with a probable solution to this issue. We thought we'd apply the Kickstarter model to new HITs that are posted. The operator of a KCST campaign may give backers a pledge reward in return for their monetary ‘pledge’ towards their campaign. Pledge rewards are almost always a limited edition item or perk, given to a limited number of backers, almost always at a first-come, first-serve basis – thereby acting as an incentive for backers to pitch in and support the project. Pledge rewards may decrease in lucrativeness as the minimum pledge amount decreases. Similarly in our implementation, for a limited amount of time, workers are offered an extra monetary incentive to work on a HIT. As time goes on, the added monetary incentive gradually decreases, further incentivizing the HIT opportunity. This is also similar to Uber’s surge pricing model, but flipped on its head – Instead of increasing the price of the service as long as high demand lasts, we're trying to create high demand and increase incentive for workers by offering an increased compensation amount for a limited amount of time. 
 We also looked into looking at how the system to tackle revisions of work completed if there were any errors. The pay would be equal to the percentage of work that was correctly completed, with the remainder being paid when the resubmission is verified as correct and complete.

  • Github Style “Issue” Tracker for every HIT + Reputation + Private Messaging

Workers and Requesters often aren’t on the same page when it comes to basic communication. Simple questions often go unanswered, because of which workers are unable to complete tasks to the requester’s satisfaction. This leads of obvious unrest between both parties. To limit similar occurrences, both workers and requesters ask for help on 3rd party forums and platforms. On industry-standard platforms like AMT, there is no way for Ws and Rs to communicate. We thought we’d improve upon this by implementing a system such that each HIT has its own “support” page or “issue” tracker, where Ws can report bugs, ask questions, suggest edits and participate in discussions with other Ws and Rs. We think that lifting this very fundamental communication barrier will be instrumental in instilling trust in both parties. A messaging feature between the requestor and worker has been an oft requested feature. But letting workers message requestors for all issues is no feasible. We suggest using the Github Style issue tracker, but when a question goes unanswered for 24-48hrs the worker has the option of messaging the requestor. Some jobs that require constant feedback could have the option of toggling on a private messaging line.


^ Image 1: Github Style issue tracker


^ Image 2: Private messaging


Once implemented, both these features would present an interesting challenge when tested for effectiveness. A gradual rollout of these two features would be key, as a platform-wide rollout of these features might be more detrimental than effective. We would start by identifying the Workers and Requesters who would benefit from these two features the most. For the Kickstarter model, we use existing usage data to find requesters who have a good past track record – they pay competitively, they pay on time, post engaging tasks – yet do not find enough workers to complete their task, or, alternatively, the workers they do find don’t complete tasks to their satisfaction. We let them/ask them to use and test the surge pricing feature. We will know that the feature has had a positive impact if engagement and successful submissions for their tasks increase – not only when they opt to use the surge pricing feature, but also without it. For the issue tracker, we take a similar approach. We have two main targets as far as testing beds go – complicated HITs that have a lot of workers, and HITs that have a lot of new workers attempting to finish them. The ideal testing ground would be a HIT that has a lot of workers confused, but that’s hard to gauge since most workers look to third parties/third party forums for help to begin with. The success of this feature can only be measured by measuring engagement between requesters and workers, and by getting direct feedback from users.


[1] Feature Idea - The Kickstarter Model/Surge Pricing “”

[2] Foundation Idea - Dynamic Pricing Model for co-ordinated sales and operations “”

[3] Feature Idea - Github style issues system “”

[4] Foundation Idea - Social Networking meets Software Development ""

[5] Feature Idea - Got issues? Who cares about it? A large scale investigation of issue trackers from GitHub ""

[6] An analysis of requirements evolution in open source projects: recommendations for issue trackers ""