Why Guilds?

From crowdresearch
Jump to: navigation, search

Why Guilds

When discussing Guilds or any organizational design solution for a crowdsourcing platform, one's mental model is influenced by the asymmetry in power, information access, pricing and reputation of Mturk. It is pervasive and the impulse is to correct the injustice; for us it has proven to be a nice starting point for discussion. Yet, for as unsavory the imbalance may seem, Mturk is still operating and functioning. If you were to create a new crowdsourcing platform how would you inject symmetry into the platform. We look more to systemic balance than social fairness in illiciting the idea of symmetry, in the crowdsourcing world, the invisible hand of the market is more opaque. So, we looked to history, distant and a bit more recent, for inspiration. Historically, guilds have represented craftsmen and projected an image of quality. This association is critical. In Mturk, workers are seen as low cost labor, in a balanced system, they are skilled craftsmen. So a mechanism for symmetry is to allow workers to associate within the system to share information, share training and leverage for better pricing. We see such worker impulses in the form of Turker Nation, Turkopticon and the like, yet they exist outside the system. They create a sense of community as it relates to training and development, yet we are looking at Guilds for the balancing agents they bring to the system as economic entities that impact Time, Cost, Quality. You cannot separate Benefits from Savings nor can you separate the social from the training and economic dimensions of the guild.

So, why collectives or guilds to address this problem, why not deploy a system-wide computational compatibility model? We feel that the intimacy of clustering workers around a task or a skill ultimately provides community. A cross cultural, cross generational collection of like minded individuals that share common professional/techncial interests, professional development goals and a sense of reasonableness that allows individuals to act as a network, to leverage collective intelligence, to collaborate and conspire. Bonds, that when scaled can create universal skills/compatability norms and rational dynamic pricing. Again, the need for these community elements at the bottom of the digital pyramid are arguably not as critical as in macro task environments, nevertheless, such cultural components are critical to the effeciency of the Guild.

In a guild individual workers aren't exposed to the randomness of requester reputation whims, they are protected behind a veil, by the collective reputation of the guild. This Protection allows for a freer exchange of information, for quality and learning to be ensured behind a veil and for edge costs to be absorbed in the process

Why would a requester choose a guild over an individual?

The initial engagement of requesters and guilds is a cold start challenge in its own right. Are individual requesters more inclined to trust an individual or an organization? Through mechanisms of managed risk, requesters are probably more disposed to using an organization to complete tasks over an individual. Guilds will be able to leverage their community, there individual approaches to vetting and ensuring quality…perhaps in the form of guarantees or rebating. However, one we move beyond that initial engagement, it will be up to how a guild positions itself in the market. This is a risk and a capacity question. Scale, aggregating work. Internal mechanisms (Rules, Standards and Best Practices) guarantee quality, time and cost savings. Guild members are selected based on a computational compatability model.

Origin Story of the Guild

So how would guilds form, if they weren't mandated by the system or systemically created. If the system was open enough to allow for worker organization within the system, two impulses would come into the formation of guilds. Two impulses would drive/lead to Guild creation: Entraprenuralship and Homophany. "Sphere of Confidence" (Pierre) Groups created by entrapreneurs who can share their reputation with others.

Rules, Best Practices

[PierreF] Historical functions of guilds

  • Formation through apprenticeship and its regulation
  • Quality standard settings
  • Hours of work regulation
  • Diffusion of market information
  • Diffusion of best practices
  • Price fixing
  • Access to credit and financial support

How Do Guilds Work: Candidate Attraction

a Computational Compatability tool, which they will use to identify new members, thereby validating the model and establishing the internal skills hierarchy.

“Facebook” pages : guilds are entitled to publish on a dedicated page entitled to them by the system Chat channels : the chat allocates two channels per guild : guild-open shared by all, guild-restricted

[PierreF] An autonomous has access to : a general chat, discussion between platform users a market chat, basically a channel where guilds are promoting themselves and requesters can also promote and annonce future job offers. Such a worker can select opened jobs/tasks to work on it and post job interests, but most importantly he could also apply to an existing guilds using direct chat contact with advertised guild officials.

How Do Guilds Work: Requester Attraction

A requester will have the ability to request a guild to work with or if they so desire, they can work with an individual. As this is a guild conversation, the requester, because of the size of the task (say image processing 25,000 photos) they come to a guild. Instead of having to give instructions, answer emails from a multitude of individuals, they have 1 point of contact. In exchange for trusting the guild, the convenience of the transaction they pay a premiuim, though moving forward we’d like to establish relationships with requesters that lead to retained services pricing structures or subscription based services. Once terms and conditions have been established, the task is released into the guild.

How Do Guilds Work: Communication and Governance

[PierreF]

The guild can have set standards for applying. Basically, an application means that a worker declare to meet these standards. In a guild, three ranks organize relationships between members: Apprentice : new member Journeyman : confirmed member, contribute by mentoring apprentices, can monitor batch results Master : senior member, takes part to decision, have admin access to guild tools, monitor journeymen activities, can contract for the guild. (we could change that to a more fancy context if necessary)

A journeyman can and must delegate some tasks to his apprentice and is in charge of controlling the produced results. As a side-note, apprentices benefit from the performance record of the journeyman: they have access to task that could have been not accessible due to restriction of application defined by the requester. Journeyman supervision is paid by a fee collected from the amount due for task completion. Supervision is a mandatory step for achieving mastership. Journeyman management of apprentices is reviewed by a master.

Skills are managed by tree-like structures from general to more specific (cf yoni description). Regarding promotion and skills levelling, attribution to apprentice results from proposition issued by the journeyman mentor. A Journeyman can present his own case to Masters for acquiring new level of skills while masters manage their own skill acquisition on a peer to peer basis.

How Do Guilds Work: Quality Assurance

While a skill may not be necessary to perform micro tasks, many attributes are required to relaibly complete tasks. Attention to detail, timeliness, task focus, immunity to repetition disorder and pride in work(maybe :) are all elements that can be heuristically measured from resumes/CV, Git Hub, MOOCS, a guild application, prior experience as a crowd worker, etc. Utilizing graph databases we’d extract data/attributes so as to define compatability proximity to a task. The challenge may be that,when implementing in a microtask only platform, willl there be enough discernable variation among workers and tasks to cluster meaningfully and/or have a statistically significant distance metric. Without this meaningful variation it is less likely to impact the task feed in a positive way. Only through development of the algorithm and underlying data model (graph vs. relational) will we be able to find the necessary variance thresholds required to determine effectiveness and appropriate implementaiton of this design. Until a certain density is achieved in the collective, human intervention is required to validate core competencies, until the system has achieved enough learning to automate. Spot checking will occur. Not only does computational compatibility give proximity to tasks, near misses can also be identified and converted into learning opportunities (associations with MOOCS or internal training) for workers

Minimizing the risk and enhancing the responsability for returning the task on time in the manner requested is tethered to 1. the appropriate matching of the work to the worker going in and 2. QC checking as the product is completed and delievered. This duality of task in, task out, is a dynamic method of training and adding measurable moments to a workers profile. It gives cold starters/newbies an opportunity to review work and/or participate in live work with the safety net. In a high frequency, low latency environment this double blind process is standard practice.

EXPERIMENTS

We think the areas to experiment on and validate why a requester would use a Guild over an individual is bore out best in evaluating the worker or Guild/requster relationship through an economic lens of Time, Cost and Quality. What gets tricky is the extent to which the organizational dynamics and structure of the guilds impacts the output/measurable. If we need to outline the internal mechanisms of a guild, can that be compressed into a 10 page paper? Or can we suggest the guild is the sum of these experiments?

Time

Time Experiment

Cost

Cost estimation is a skill, such as any other business skill. Each field has its own way of estimating costs and professionals who vary in the task cost estimation. A guild or an expert system can serve as a way of transferring industry business specific methods to tasks to avoid overpaying and underpaying. Possibly, guilds instead of people estimate costs for the same job to further minimize overbidding and underbidding.

Cost Experiment

We do an A/B test. A being 1 to N and B N to N. The objective would be to measure the edge costs associated with executing micro tasks in two different network models. The hypothesis is that when both workers and requesters interact with a single node the maintenance cost (administrative time associated with getting work completed) is less than when maintaining edges with mutliple worker/requester nodes. Further works could determine the effects and effectiveness of this central node when manifested as an individual (work broker/agent), a collective (guild structure) or algortihm(purely technical central node). .

Quality

Instead of looking a reputation we seek to assess and evaluate quality.

Quality Experiment

(@anotherhuman) gauge r and r study The specific dynamics of quality that matter are reproducibility and repeatability Is the outcome reproducab Is the process repeatable Quality in this situation is about eliminating variability

Contributors

  • @teomoura
  • @pierref
  • @trygve
  • @lucasbamidele
  • @yashovardhan
  • @gbayomi
  • @horefice
  • @scorpione
  • @markwhiting
  • @dilrukshi
  • @acossette
  • @m.kambal
  • @yoni.dayan
  • @rcompton
  • @arichmondfuller
  • @anotherhuman