Guilds and Computational Compatability Extension

From crowdresearch
Jump to: navigation, search

Disclaimer: This document is where the ideas for our main submission (http://crowdresearch.stanford.edu/w/index.php?title=Guilds_and_Computational_Compatability) took place. We're posting it so a newcomer can get a glimpse on how we came to our proposals on the main submission. However, it is highly recommended to read the main submission first, since it clearly and concisely describes what we have here.

Brief introduction of the system DaemoGuilds is an online Community/Organization that effeciently aligns workers and requesters based on a computational assessment of compatability (worker to task). A collective scheme allows its members to be grouped into specialized skill guilds and sub-guilds; while allowing requesters to have a single point of contact when it comes to large/repetitive micro tasks. This single point of contact can also assist with task authorship and helping requesters know their work. This type of organization can leverage the dynamic of collective intelligence for the full range of tasks (micro to project) while also maturing with the task sophistication of individual requesters.

Reputation systems can be gamed, to the point they are rendered irrelevant while testing and other standardized tools for assessing one’s competencies can also be gamed. Hence, the need for a matching mechanism that is both accurate and scalable. These assessments/this heuristic will be applied to both workers and requesters. These attributes position a worker closer to some tasks over others. Task authorship will also be distilled into a series of attributes as well.

How is the system solving critical problems

The first issue that an introduction to our system should address is: Why use guilds in a crowdsourcing platform?

It is known that current state-of-art crowdsourcing platforms allow its workers to address micro-tasks with a 1-to-1 relation, meaning that they only connect with one requester at any given moment even if they are performing various tasks at a time (since they're always performing one task a specific requester introduced into the platform). What we envision by providing a guild structure is to change that paradigm, by allowing the requesters to address not only specific workers but also letting them target guilds to perform more complex tasks, introducing the concept of collective intelligence into our system. By providing an environment where guilds are formed automatically over time accordingly to which kind of tasks are being requested, we're able to build dynamic communities and sub-communities(guilds that are part of other guilds) inside our platform, making 1-to-N and ideally N-to-N connections possible in our crowdsourcing space. An important point to notice before we address the modules inside our application is that the definition of what a guild is might be a topic of discussion when it comes to different areas of specialty in Computer Science. In online gaming, guilds could be defined as a group of gamers who help each other so that they all are able to achieve a specific objective that they wouldn't be able otherwise. However, even if we slightly change the style of the game in consideration the way guilds work in each of them may vary wildly. In some cases, the only thing in common between them is that they are groups of users who gather in an online community to achieve goals that they couldn't prior to that, but internal politics, structure and objectives may be very different from one another. From this point and beyond, we will provide our own definition of what a guild in a crowdsourcing is, so that we're able to provide a common ground for understanding. A guild in a crowdsourcing platform is defined by a group of workers who are put together inside a sub-community of a main community. The parameters that we suggest for creating these subsets are the following : the history of a specific worker W to perform a determined amount of "N" tasks, inside a scope of requests S, within a reasonable amount of time T for each task, with a certain acceptance rate Q, that cross a minimal threshold stablished by the admins of the platform. For instance, if we specify that a guild is formed each time a worker reaches 98% accepted tasks within the "#ImageProcessing" scope after doing 100 tasks - each worker that qualifies for all of these parameters would be automatically added to the same sub-community within our platform.

Introducing modules of the system

Module 1: Guilds

Problem/Limitations In the current crowdsourcing platforms such as MTurk, tasks are displayed to every worker and there is no reliable way for a requester to ensure that one specific worker has worked on that kind of task before or that he will actually be able to perform the task in hand. Therefore, several problems arise: How do we increase the likelihood that a worker will be able to perform one given task with an acceptable rate of quality? Is grouping workers by skill-based guilds a good solution to this problem? If so, should we have one general guild on the platform or several guilds?

Additionaly, if we can find a more advanced and detailed rating system, we can provide requesters with critical information regarding the capacity of the crowdsourcing population to service more advanced tasks. This gradually must enable them to create job opportunities with more added value. Budget allocation will potentially raise in proportion in a more diverse market. This will enable various types of workers to find adequate deals according to their skills and expectations.

This solution must scale to accommodate large population of workers and requesters. We thus have to demonstrate the feasibility of automation of task allocation in such a diverse universe. It is also necessary to describe the way guilds can operate, that in particular in order to capture and validate member skill description.


Module preview Two possibilites come to mind when one thinks about implementing guilds into crowdsourcing platforms. The first one is to have one general guild in which every worker is placed whenever he/she enters the platform. On the other hand, we could have several guilds divided by a certain parameter. We believe that having multiple guilds spread across the platform is the best way to ensure that our approach solves the reputation problem faced by crowdsourcing services nowadays because it allows the platform to provide a very well specified structure of growth for the community over time. It doesn't matter if a user is a worker who only do tasks in his/her spare time to earn a few dollars or if he/she works full time. If we implement guilds we're always able to provide an appropriate environment for them to find and perform tasks of any type of complexity . We do so by asking each requester to tag his request with one general tag (such as "#ImageProcessing" or "#WebDevelopment" for example) that while looks like a simple hashtag for the requester, it actually means a guild to the system itself.

System details As soon as the SysAdmin sets up the crowdsourcing software, we ask him to provide the number of tasks requested under a certain tag so that that tag becomes a guild (for example, if the sysadmin sets up "100" as this parameter, each time a tag such as #specificTag reach 100 tasks requested, it becomes a guild), and also the number of tasks a worker needs to properly perform under a specific tag to become part of that guild (if set up to "200", for example, than after 100 tasks had been successfully completed under #specificArea so that it becomes a guild and the worker performed 200 tasks under #specificArea, than that worker automatically becomes part of that guild, and a badge shows up on his profile page). A similar computation should also be performed in order to level up a group from a regular guild to a major guild, which represents a major area on the crowdsourcing platform such as the aforementioned #imageProcessing and #WebDevelopment. We suggest this calculation to be done in respect to the percentage of tasks under a certain guild in relation to the other ones. So, if #WebDevelopment is present on 20% of all tasks on the systems, for example, it should probably be suggested to requesters as a general tag. This way, we guarantee that there will always be major groups within our system, allowing us to implement the module described next, sub-guilds.

Description of a possible matching algorithm

Skill Forest definition: The system provides a skill ontology (big word to say a logical organization for skills from the more general to the particular).

E.g. Computer Science -> Software Development -> Python

This classification is not a tree, but a forrest:

  • E.g. Computer Science -> Web Development -> Python exists
  • but also Computer Science -> Software Development -> Functional Programming -> Python

“Computer Science” is defined as a root skill (there is no more global skill that encompasses Computer Science in the ontology.

Skills are expressed as a proficiency level on a node. We expect than the deeper the node the easier it is to assess a proficiency level. It may be for example quite difficult to assess the level of someone in software development, but clearly it is simple to test someone on its knowledge in Python (task completions, tests, etc.).

So a skill profile can mention a level 3 Python capacity for example. When a level is assessed, it is also required to select a path in the ontology from a root skill associated with this assessment. Let consider this person has mainly used Python in the context of web development, the associated path with the level 3 capacity is then (Computer Science -> Web Development -> Python). We see that the level is not really rating a node, but a path in the ontology.

When a requester publishes a set of tasks, he defines a list of constraints. These constraints are again paths from root skills in the ontology with an associated minimal level for each path. What is a match ? There is a match if all the constraints are sub-paths of existing skills at the required level.

Example : a requester asks for a (Computer Science -> Web Development) @ Level >= 2

The worker is (Computer Science -> Web Development -> Python) @ Level = 3

We have a match as (Computer Science -> Web Development) is a subpath of (Computer Science -> Web Development -> Python) and Level is superior to 2 (3 in this case)

If the request had been (Computer Science -> Web Development -> Python -> Django) @ Level >= 2, this worker would not have been a match

Then, still considering the first constraint, if another worker is (Computer Science -> Software Development -> Python) @ Level = 3, again we do not have a match.


Wait a minute, the level for the worker results from a guild assessment while the constraint is defined by the requester. How can we be sure they share the same idea of a proficiency? We cannot. It is part of the competitive policy of the guild to tune its attribution of level to meet requester expectations. Of course, if we have only one guild, this will work the other way around, requesters will learn to adapt their expectation to the guild rating scale.

How do we introduce new skills? We must allow for evolution in order to follow the demand. The skill ontology can thus be extended at will by requesters. Of course, it will take some time for this evolution of demand to be reflected by an adaptation of the offer. As a footnote, I am more cautious regarding the capacity of workers to do the same, as it may open some unexpected capacity to game the system.

How the guild plays its part in this system? To work efficiently, the described system need to have a moderated system for skill assessment. With the explosion of possible paths, we consider not economical to centralize moderation. The guild structure provides a peer environment to control the correctness of worker declaration as it is in the interest of the community that requesters can select the adequate candidates. The satisfaction of requesters in term of work quality is a condition for their repeat submissions.

Historically, the guild structures have ensured customers of the proficiency level of their members. The apprenticeship concept played an important role in this respect. It is certainly something that can be adapted to help validation at some higher levels. At the same time, being a validating mentor can be a mandatory experience to reach the highest levels in a domain.

Guild Reputation: In a competing world of guilds, we may be back to the basic question of evaluating guilds themselves. One can consider this a lesser problem as the volume of guilds is one to two orders of magnitude less numerous than workers, but this can be a loophole due to their role in validating their members. This may be an opportunity to introduce Boomerang.


Module 2: Sub-Guilds

Problem/Limitations[edit] Having major guilds allows our requesters to target N workers (1-to-N relation), which is already an enhancement from the current 1-to-1. However, how do we provide a requester the best-suited workers for a specific task? We cannot rely on tests, since workers could game the system by sharing answers with each other. Additionally, there is the problem of selecting the workers whose work history is most relevant to the experience required by a given task.

(Workers new to Daemo need opportunities to be matched to requesters’ tasks in the first instance. These new workers will be matched to requesters’ tasks based on self-declared capabilities. To assess quality of results, randomised dummy tasks from Daemo will be interspersed with the tasks from real requesters. We will build into the pricing structure a system whereby the worker/s in these subguilds are paid to assess the quality of these dummy tasks.???)

Such a system makes sure that new workers are not starved of tasks and have an opportunity to gain acceptance in the guild system. Managing identities of all workers then becomes a critical feature, so as to forbid them from repeatedly utilising this system to increase their reputation by trying to appear/behave like a “new” worker. (how? By ensuring everyone signs up using their real name?)

Module preview We establish the conditions in which sub-communities are formed within a main community (aka how sub-guilds form within guilds) and how we use that structure to determine which workers are the best suited to perform a certain task taking in consideration their reputation and their actual ability to manipulate certain pieces of software or other specific tools to be determined by each sub-guild.

System details After we ask for a general tag from the requester, we also ask him to follow up with more specific tags that reflects the nature of that task (for example, for a "#ImageProcessing" task further tags could be "#OpenCV" and "#C++" which would mean that anybody in the #ImageProcessing guild would be suitable to handle the task, but it would be optimal to have a worker who is also specialised using the OpenCV library written in C++ to work on that task). What we envision with this proposal is to allow sub-communities to develop within major ones, with the purpose of allowing us to keep the track-record of a specific worker, considering his reputation to be a consequence of the major and sub-groups he chooses to join within the community. Since guilds and sub-guilds always have a certain threshold of entry in terms of number of tasks concluded and the quality of those tasks, we believe that the likelihood of a worker being able to complete a certain task increases with the number of levels he matches with the task in hand. It might sound as a confusing idea, so to clear it up consider the following scenario: A requester posts a request with the tags #PhotoEditing (major one), #PhotoShop and #ContrastAndSharpeness. We consider that the likelihood of someone being the most appropriate person to complete that task increases with the number of guilds he matches with the task. So, a worker who belongs to the #PhotoEditing guild is probably a good one. However, if he also belongs to the #PhotoShop sub-guild, it is even more likely that he is suited to that task, and the same goes for someone who also happens to be in the #ContrastAndSharpeness sub-guild. Therefore, we show that task first for someone within those communities. By setting up such an environment, we're able not only to determine that a specific worker has a solid history of working with photography (aka his reputation is high) but also to increase the likelihood of him actually being able to perform that specific task, since he is proven to be a specialist using the tools asked by the requester.

Module 3: Collective Intelligence Tasks

Problem/Limitations

We have suggested solutions to the problems of a requester who posts micro-tasks only. However, how do we prepare our system to handle complex tasks? Ones that not only involve performing a simple procedure but also some level of interdependence with other tasks to accomplish a certain goal?

Module preview We allow the requester to link tasks and select the guilds to whom those tasks should be addressed to, therefore automating the process of completing a complex task. As an example, if a requester needs to have a photo converted to black and white, printed and delivered somewhere, all he has to do is to create 3 tasks and select the guilds that would handle them.

System details The requester creates a Collective Intelligence Task, which is composed by several other micro-tasks. He specifies what those requests are and address them to guilds either of his choosing or suggested by the system. The system than uses the structure of sub-guilds to quickly find who are the best workers to show those tasks to. If the Collective Intelligence Task has components that rely on each other (such as the example given above), the following tasks are only shown to the workers after the previous step has been successfully complete. After all steps are done, the Collective Intelligence Task is considered complete. A problem that may arise is how to determine if the requester should approve each step or only the final result. As of now, it is suggested that each step should be validated by the requester, but it is possible that in the future automated mechanisms using machine learning techniques will be suited to handle that. This module allows us to build complex tasks, but only within a certain context (do x y z to a photo, or w q s to a song).

Module 4: Complex Collective Intelligence Tasks

[this one is optional to the system, but would be very, very cool to see happening]

Problem/Limitations Having the ability to complete tasks that are considered complex to nowadays standards, how do we use guilds to unleash the full power of crowdsourcing communities? And how do we coordinate the process of dealing with several different Collective Intelligence Tasks at the same time?

Module preview We allow requesters to create multiple Complex Collective Tasks and group them together to create intricate crowdsourced workflows. An example would be a software project with millions of lines of code going directly to workers in a #Design(major tag), #iOS, #flat, #sketch3 guild so that they enhance the look and feel of the application and then delivering that to a #Deployment(major tag), #RubyOnRails, #Capistrano, #AWS worker to deploy the application. If well specified, a complex project could take from some days to a few hours to complete, which is far better than the current standards.

System details The requester creates a Workflow composed by two or more Complex Collective Intelligence Tasks or regular tasks connected to each other by relations of cause/consequence. Therefore, one Complex Collective Task will only begin after the previous ones have finished, in the case that they are dependent on one another. For each step of the workflow, the requester approve or not the results. This module allows requesters to perform very complex workflows by using guilds and sub-guilds, therefore making the crowdsourcing platform a major solution to people who only need simple micro-tasks done but also for people who need to build huge projects within a budget.

Conclusion

Having layed out a scheme of how a collective-based crowdsourcing platform (from this point on I'll be using the words "guild" and "collective" interchangeably) could work, it is a good idea to reinforce the importance of having workers group together into collectives. At this moment in history, we're living a time when most of the work done on crowdsourcing initiatives (such as Uber and Mturk) is fairly simple, consisting of tasks that are perfomed an amounted number of times by a worker. That layout includes no space for interaction between tasks and more than that, leaves no possibility for complex workflow-based tasks to emerge. That happens because workers are not used to adding work to tasks previously handled by others in addition to the fact that the platforms themselves don't even provide them with the chance to do so. By allowing workers to group themselves in guilds, we allow them to make contact with like-minded people and to work on tasks that require a whole new level of complexity, in which it isn't enough anymore to just take a data-based input and provide a processed output to the requester. It is fairly common knowledge that worldwide economy is shifting towards a model where wealth generation happens not only by the trade between group of people who manufacture a product that are consumed by another group of people, but also happens when common people are able to deliver services that they weren't previously empowered to that are consumed by people who before the internet couldn't get access to such services. (Global Production Networks….Like it) Therefore it is fundamental for a global economy to provide an online reliable space that encapsulate and simplify a radical notion: work as we know it (1-to-1, meaning one worker works for one specific requester, even outside the internet) towards a model where anyone could work for anyone (N-to-N). In the history of humanity, whenever this kind of shift happened (it went from one guy in a cave trying to survive to groups of people forming guilds, to those guilds finding and interacting with other guilds and so on), the concept of a worker being one node in various workflows started to appear. Indigenous populations throughout the world separate their tasks in such manner that everybody provides various kinds of work that are needed by the community (even thought they usually separate by gender, with a few notable exceptions such as communities where women are the driving force behind their survival).What we're delivering by setting up a guild based crowdsourcing platform is the possibility for humanity to go from being one guy behind a computer screen to complex groups of people interacting globally to work and receive work using the online space. However interesting this idea might seem, we must not forget one key element of this equation: IoT and M2M communications. We don't need anymore to rely solely on work done by humans, and we also do not need to rely on work solely done by machines. While sci-fi movies scare crowds all over the planet making them think robots will steal their jobs, what is actually most likely to happen is the integration of humans and computers in the same workflows. A factory could use human crowdsourced work to import commodities from another country, computer crowdsourced work to assemble those commodities into finished products, human crowdsourced work to process legal issues and robots to ship items wherever they want to and so on. Or a police corporation could easily have a workflow that communicates with cars on the roads to report a suspect to a human-crowdsourced worker, so that he can check the likelihood of that person to actually be guilt in realtime before deciding to spend precious resources (cops, time and financial recources for example) on any given call and therefore reducing by huge leaps the costs of current police forces all over the world. In conclusion, while it might seem like a small step to build a guild-based crowdsourcing space, we might very well be shaping how humans and computers/robots will work together in the future, making use of artificial and human intelligence, as well as machines communicating with humans and another machines to form complex, cost-efficient global workflows.

Contributors

@teomoura @pierref @trygve @lucasbamidele @yashovardhan @gbayomi @horefice @scorpione @markwhiting @dilrukshi @acossette @m.kambal @yoni.dayan @rcompton @arichmondfuller