Winter Milestone 4 PierreF

From crowdresearch
Jump to: navigation, search

All contributions : PierreF_Space

Exploration of CrowSourcing Dynamics with Social Experimentations

CrowdSourcing platforms represent a new dimension in the workspace. While initially deemed as a way to hide the human component in automated workflows, this position has quickly shown its limits. Human relationships have rapidly resurfaced to play an essential role for the global efficiency of existing systems.

Finding a middle-ground between task providers or requesters and workers appear a necessary objective to maintain a lively market where both population can satisfactorily fulfill their respective objectives.

Current playground is heavily unbalanced in the advantages of requesters. Existing platforms put them in control of the dynamics of the major economic factors, namely prices and reputations. This situation is heavily resented by the worker population. A possible evolution leads to ultimate long-term demise of this kind of platforms due to loss of quality in the average task completion resulting from the departure of the more motivated and skilled workforce.

The background of crowdsourcing in game theory terms has already been set up by other authors:

  • Game Theory and Incentives in Human Computation Systems, Arpita Ghosh. Book chapter. Handbook of Human Computation, Springer 2013. [1]
  • Behavioral Mechanism Design: Optimal Crowdsourcing Contracts and Prospect Theory. David Easley, Arpita Ghosh. Proc. 16th ACM Conference on Economics and Computation (EC), 2015. [2]

Large scale social experiments have been conducted to discover best ways to manage financial incentives to optimally engage a crowd of participants.

  • Galen Pickard, Wei Pan, Iyad Rahwan, Manuel Cebrian, Riley Crane, Anmol Madan, and Alex Pentland. Time-critical social mobilization. Science, 334:509–512, 2011. [3]

This study intends to describe practical explorations of game theory concepts in crowdsourcing through social experimentations. These experiments involve the introduction of new mechanisms in the environment of an experimental crowdsourcing platforms. We intend to explore the feasibility of satisfactory Nash equilibria due to the introduction of such mechanisms leading to a more harmonious field of play.

Experimentations address in particular the introduction of guild organization concept and giving the capacity to workers to adopt different forms of self-governance using in-platform tools. Of particular importance is the repetition of interactions involving the same population to understanding the dynamics and the reactions of requesters and players in the long run. The workers and requesters are briefed to adopt stylized behaviour. For requesters:

  • Lenient
  • Fair and communicative
  • Judgemental

For workers:

  • Competent and efficient
  • Motivated but unskilled
  • Slow
  • Lazy and negligent

On top of that, guilds of different sizes will be tested as well a control group of people staying alone.

Three major metrics will be tracked to understand the evolution in interactions:

  • engagement in the platform (participants can decide to quit and not participate anymore)
  • price evolution on a range of task types
  • reputation evolution according to worker attitude and requester attitude.

Through the tracking of these metrics, we want to analyze the emergence of stable situations meeting preferred properties.

Q&A

  • You mention repetition and patterned behavior. Are you proposing a longitudinal study?

If I understand the concept of longitudinal study, the objective is to understand how people evolve in their opinions. That is not what I have in mind. I need repetition in the study, because I think we need to capture evolution and strategies developed by users. I foresee users reflecting from the results of a first batch of interactions and then changing their interactions in the next iteration (we need to keep the same people in the same role from one session to the next one). Our focus is to detect useful stable situations that should be favoured : for example, natural eviction of spammers and unfair requesters, promotion of the reputation of skilled workers, correct pricing range, etc...

  • The criteria you want to measure....Are there established norms for them. I mention this because for all the importance of trust in these systems, it awfully difficult to measure and experiment against. Determining satisfaction or systemic equilibrium may be more achievable...

You are right to stress that there is coupling at work here. But my feeling is that we cannot measure something like trust. I see it the other way around : we ask someone to act in a predefined way (basically we know before hand if people should trust him) and look at observables in our system to see if we have a match between inner knowledge and observables.

  • Are you proposing to drop your experiment into the Mturk system or one in which negotitation is possible?

Now we need to be in control of the production of observables for experimenting. So we need an experimental dedicated platform: Daemo.

  • If the later, I'd like to get more details on how the negotiations would work. Would Guild Leadership manage each engagement, would the Guild set a price, would individuals engage individually but go to leadership if they felt there were 'issues'

This remains to be defined for experimentation. I have of course ideas on the subject :

    • Guilds mandate representative able to "contract" for the whole guild. Individual workers remain free to contract for themselves. An hidden point here is the way we manage identity in the platform.
    • Guilds take in charge a full batch for a lump sum. The way this amount is shared between members is discretionary in the guild (Some regulation may be required there)
    • Prices result from a bid-ask market-oriented process (we must be able to segment types of similar batches).
  • How would guilds form and grow? Recruitment, Marketing/Advertising, Referrals? Curious about the barriers to entry.

Every way they find effective. In game environment, the best way to recruit is by one to one interaction: you play with a guy, find him ok and join his guild if he proposes you to do it. Here we do not have this kind of one-to-one first contact. So we need to give guilds to have leeway to differentiate one from the other in their offering to workers. The communication tools should take into account this need of promotion from guilds.

  • How do you envision requesters embracing guilds (if at all)....What would be the process of convincing requesters that there would be a value add working with guilds?

I see two main advantages for requesters in the existence of guilds:

    • Requesters get more assurance quality as guilds would be concerned by the performance of their members (guild reputation) and peer-regulation could be pretty effective
    • I remember a requester complaining about the time involved in managing his crowd. The guild can help in representing a single point of contact for a complete project.

Additionally

    • For new requester, a guild can take more risk in engaging to do business with him. We have seen that lone workers find too risky to work in large project provided by an unknown requester.

As a finale thought, I cannot see why guilds would not be able to develop original marketing policies to attract requesters.

  • I would focus a bit more on solving one particular problem....it will help focus your argument, make experimentation easier to manage and lend itself to an easier paper writing experience on the back end ;)

You are right. The point is that I fear that observables are heavily tangled one with the other. I am afraid that we miss something important by narrowing the scope. This said we must certainly limit the complexity of analyzed mechanisms and construct something more like a model than fully functional features. Then the focus should be to find properties that lead to the favoured Nash equilibrium. However we can limit the scope of definition for this Nash equilibrium.

  • Looking for ward to the evolution of this idea

I certainly do :-)

Contributors

@PierreF

This contribution has benefited a lot from the discussions in the hangout organized by @trygve So thank you to : @trygve @dilrukshi @m.kambal