Milestone 7 sanjosespartans

From crowdresearch
Revision as of 22:33, 15 April 2015 by Jsilver (Talk | contribs) (Abstract)

Jump to: navigation, search

Creating a Reliable Pricing Mechanism for the Future Crowdsourcing Platform


1. Effective Pricing and Allocation of tasks needs good tools.

2. Framework needed to design mechanism with provable guarantees in an online jobs platform (OJP) or crowdsourcing market. The framework enables the automation of pricing and allocation process for employers in an OJP where employers encounter budget constraints and task completion deadlines.

3. Competitive Incentive Compatible Mechanism for maximum number of tasks under budget and minimum payment given to a fixed number of tasks to be completed.

4. Create a platform that enables applying pricing mechanism in OJPs.


1. Advancement of Internet leads to new labor markets called OJPs where cognitive work distribution is done by hundreds of thousands of geographically-disparate workers.

2. Employers usually outsource large quantities of simple tasks to anonymous workers. Typically, the work includes image labeling, sentiment analysis, content generation, listing verification, and those types of things that are too difficult or expensive to automate are usually crowdsourced.

3. There are many crowdsourcing platforms that provide workers with non-monetary incentives like entertainment, educational opportunities, information, and altruism for their efforts. Despite platform success, it's tough to engineer a non-monetary incentive scheme for tedious and repetitive tasks.

4. Overwhelming majority of crowdsourcing tasks are done in exchange of payments. Here, implementing a campaign successfully needs pricing and allocation of tasks effectively.

5. Designing effective pricing and allocation mechanisms present challenging problems due to employers' constraints and realities of markets.

6. Employers face task completion deadlines and budget constraints and must account for dramatic elasticity in workforce supply. Also, there is large variance in effort needed to complete different tasks that depend on skills and background of workers based in multiple geographical locations.

7. There are limited tools present in OJPs for pricing tasks effectively. Develop theoretical framework and design mechanism that work in practice and have a provable guarantee.

8. We describe a platform that enables employers to automate the process of pricing in an OJP using a mechanism we have along with other pricing schemes. Framework designed for tasks where quality of worker's performance doesn't yield additional utility to employers above a certain threshold.

9. Workers receive payments after employer's approval in the OJP. It's assumed that employers have access to verification schemes and focus on effective pricing and allocating tasks independent of quality.

10. We take mechanism of design approach to pricing problem and enable workers to bid on work by expressing their cost for performing tasks and number of tasks they wish to perform, although most OJPs don't provide workers such level of expressions, existing APIs make feature easy to integrate into most platform.

11. Mechanism present here for two major objectives -

a) Maximize number of tasks performed under a budget and minimize the payments for given number of tasks. We consider employers that impose deadline for task completion and workers who arrive according to some known distribution and can strategically misreport their cost or number of tasks that they are performing.

b) 'Incentive Compatible Mechanism' designed to ensure allocation and pricing such that it's in everyone's interest to bid truthfully.

Related Work

1. Problem of designing mechanism for pricing tasks in OJPs has been done like bargaining between employer and workers to minimize work and using 'Bandit Algorithm' to maximize tasks.

2. While both are natural approximations, they leave room for frameworks that allow better theoretical guarantees. The approach here is 'Incentive Control' problem. Another approach is to develop mode for worker's efforts and learning its parameters from data. Problem of designing mechanism for procurement has been extensively studied by the Algorithm Game Theory community over the past decade.

3. Recently, budget feasibility framework has been initiated, where the goal is to design incentive compatible mechanism that maximizes employer's objective under budget.

4. In the model, we account for online arrival of workers that raises significant challenge. There is subset literature on online mechanism design where workers arrive according to given distribution. We consider the mechanism for buying items (rather than selling) from strategic agents that need different machinery.



We also crawled TopCoder website

Price was taken from the price section. Useful ‘price drivers’ categories constructed from data:

1) Development Type (DEV). e.g. new components or updates to existing components, what programming language will be used.

2) Quality of Input (QLY). e.g. the review score and related design statistics.

3) Input Complexity (CPX). e.g. implementation “difficulty” drivers.

4) Previous Phase Decision (PRE). e.g. the price decision of previous design phase.

16 such price drivers were constructed.

Other Models studied in the Research Papers are as follows :

1. Multiple Linear Regression Model

PRICE = β1TECH + β2DEPE + β3REQU + β4COMP + β5SEQU + β6SCOR + β7AWRD + β8EFRT+ β9SUML + β10WRAT + β11REGI + β12SUBM + β13ISUP + β14ISJA + β15ISCS + β16SIZE +β0 +ε

2. Three decision tree based learners (C4.5, CART, QUEST)

3. Two instance-based learners (KNN-1, KNN-k∈[3,7])

4. One multinomial Logistic regression method (Logistic), One Neural Network learner (NNet) and One Support Vector Machines for Regression learner (SVMR)


Pricing Tasks to Encourage Finishing on Time :

1. To prevent task starvation or tasks that stay unattended by workers, tasks should be priced right (not underpriced), good enough to be taken up by workers.

2. Using survival analysis model to create an optimal task price. Survival analysis can determine the right price based on historical market data. The disadvantage is, survival analysis does not provide any idea about how the market works, and how workers decide to perform tasks.

3. Worker arrivals can be modeled with a non-homogenous Poisson Process (NHPP) based on quantitative data.Building a proper model for worker needs a model of how workers decide to take their tasks and finish their work. Workers select tasks from a task pool according to their desire.

4. Observation shows that workers often have preferences for the types of tasks they like to accept. Concept used to develop a model for a better pricing policy and scheduling for the tasks. In cases where complete, or even partial, information of the market is available, an employer can optimize his task attributes to increase the likelihood of workers accepting the task.

5. Discrete choice models can provide a framework to optimize the attributes of a task and therefore increase its desirability to the user. One convenient aspect of discrete choice models is that this change in desirability can be captured, quantified and used for attribute optimization.


1. Aim is to design a mechanism that performs well, a mechanism that decides how many tasks each worker performs and how they are paid.

2. As workers may report false costs, we seek 'Incentive Compatible' Mechanism for which reporting true cost is the dominant strategy.

3. As Incentive Compatible Mechanism guarantees that bids are truthful, its performance over bids can be compared against theoretically-optimal algorithms that knows workers true values.