Milestone5draft

From crowdresearch
Jump to: navigation, search

Brief introduction of the system DaemoGuilds is an online crowdsourcing platform that aggregates workers and requesters to allow them to perform and request tasks. It has a built in Guild scheme that allows its users to group and be grouped into guilds and sub-guilds that appear over time and with respect to each different kind of request. Further explanation will be given throughout this document.

How is the system solving critical problems

The first issue an introduction to our system should address is: Why using guilds in a crowdsourcing platform?

It is known that current state-of-art crowdsourcing platforms allow its workers to address micro-tasks with a 1-to-1 relation, meaning that they only connect with one requesters at any given moment even if they are performing various tasks at a time (since they're always performing one task a specific requester introduced into the platform). What we envision by providing a guild structure is to change that paradigm, by allowing the requesters to address not only specific workers but also letting them target guilds to perform more complex tasks, introducing the concept of collective intelligence into our system. By providing an environment where guilds are formed automatically over time accordingly to which kind of tasks are being requested, we're able to build dynamic communities and sub-communities(guilds that are part of other guilds) inside our platform, making 1-to-N and ideally N-to-N connections possible in our crowdsourcing space. A important point to notice before we address the modules inside our application is that the definition of what a guild is might be a topic of discussion when it comes to different areas of specialty in Computer Science. In online gaming, guilds could be defined as [INSERT REFERENCE HERE]. However, even if we slightly change the style of the game in consideration the way guilds work in each of them may vary wildly. In some cases, the only thing in common between them is that they are groups of users who gather in an online community to achieve goals that they couldn't prior to that, but internal politics, structure and objectives may be very different from one another. From this point and beyond, we will provide our own definition of what a guild in a crowdsourcing is, so that we're able to provide a common ground for understanding. A guild in a crowdsourcing platform is defined by a group of workers who are put together inside a sub-community of our main community with the main parameters for that relocation being the history of that specific worker W to perform a determined amount of "N" tasks inside a scope of requests S within a reasonable amount of time T for each task with a standard of quality Q that reach a minimal threshold stablished by the admins of the platform. So, if we specify that a guild is formed each time a worker reaches 98% accepted tasks within the "#ImageProcessing" scope after doing 100 tasks, for example, every other worker that also get to that level will be automatically put in the same sub-community inside our platform.

Introducing modules of the system[edit]

Module 1: Guilds

Problem/Limitations In the current crowdsourcing platforms such as MTurk, tasks are displayed to every worker and there is no reliable way for a requester to ensure that one specific worker has worked on that kind of task before or that he will actually be able to perform the task in hand. Therefore, several problems arise: How do we increase the likelihood that a worker will be able to perform one given task with an acceptable rate of quality? Is grouping workers by skill-based guilds a good solution to this problem? If so, should we have one general guild on the platform or several guilds?

Module preview Two possibilites come to mind when one thinks about implementing guilds into crowdsourcing platforms. The first one is to have one general guild in which every worker is placed whenever he/she enters the platform. On the other hand, we could have several guilds divided by a certain parameter. We believe that having multiple guilds spread across the platform is the best way to ensure that our approach solves the reputation problem faced by crowdsourcing services nowadays because it allows the platform to provide a very well specified structure of growth for the community over time. It doesn't matter if a user is a worker who only do tasks in his/her spare time to earn a few dollars or if he/she works full time. If we implement guilds we're always able to provide an appropriate environment for them to find and perform tasks of any type of complexity . We do so by asking each requester to tag his request with one general tag (such as "#ImageProcessing" or "#WebDevelopment" for example) that while looks like a simple hashtag for the requester, it actually means a guild to the system itself.

System details As soon as the SysAdmin sets up the crowdsourcing software, we ask him to provide the number of tasks requested under a certain tag so that that tag becomes a guild (for example, if the sysadmin sets up "100" as this parameter, each time a tag such as #specificTag reach 100 tasks requested, it becomes a guild), and also the number of tasks a worker needs to properly perform under a specific tag to become part of that guild (if set up to "200", for example, than after 100 tasks had been successfully completed under #specificArea so that it becomes a guild and the worker performed 200 tasks under #specificArea, than that worker automatically becomes part of that guild, and a badge shows up on his profile page). A similar computation should also be performed in order to level up a group from a regular guild to a major guild, which represents a major area on the crowdsourcing platform such as the aforementioned #imageProcessing and #WebDevelopment. We suggest this calculation to be done in respect to the percentage of tasks under a certain guild in relation to the other ones. So, if #WebDevelopment is present on 20% of all tasks on the systems, for example, it should probably be suggested to requesters as a general tag. This way, we guarantee that there will always be major groups within our system, allowing us to implement the module described next, sub-guilds.


Module 2: sub-guilds

Problem/Limitations[edit] Having major guilds allows our requesters to target N workers (1-to-N relation), which is already an enhancement from the current 1-to-1. However, how do we provide the requester with the best-suited workers for a specific task (without relying on tests, which workers could exchange information with one another in order to game the system)? Also, how do we select the workers whose work history is most similar to the experience required by a determined task (aka his reputation towards a specific skill set)?

Module preview We establish the conditions in which sub-communities are formed within a main community (aka how sub-guilds form within guilds) and how we use that structure to determine which workers are the best suited to perform a certain task taking in consideration their reputation and their actual ability to manipulate certain pieces of software or other specific tools.

System details After we ask for a general tag from the requester, we also ask him to follow up with more specific tags that reflects the nature of that task (for example, for a "#ImageProcessing" task further tags could be "#OpenCV" and "#C++" which would mean that anybody in the #ImageProcessing guild would be suitable to handle the task, but it would be optimal to have a worker who is also specialised using the OpenCV library written in C++ to work on that task). What we envision with this proposal is to allow sub-communities to develop within major ones, with the purpose of allowing us to keep the track-record of a specific worker, considering his reputation to be a consequence of the major and sub-groups he chooses to join within the community. Since guilds and sub-guilds always have a certain threshold of entry in terms of number of tasks concluded and the quality of those tasks, we believe that the likelihood of a worker being able to complete a certain task increases with the number of levels he matches with the task in hand. It might sound as a confusing idea, so to clear it up consider the following scenario: A requester posts a request with the tags #PhotoEditing (major one), #PhotoShop and #ContrastAndSharpeness. We consider that the likelihood of someone being the most appropriate person to complete that task increases with the number of guilds he matches with the task. So, a worker who belongs to the #PhotoEditing guild is probably a good one. However, if he also belongs to the #PhotoShop sub-guild, it is even more likely that he is suited to that task, and the same goes for someone who also happens to be in the #ContrastAndSharpeness sub-guild. Therefore, we show that task first for someone within those communities. By setting up such an environment, we're able not only to determine that a specific worker has a solid history of working with photography (aka his reputation is high) but also to increase the likelihood of him actually being able to perform that specific task, since he is proven to be a specialist using the tools asked by the requester.

Module 3: Collective Intelligence Tasks

Problem/Limitations

We have suggested solutions to the problems of a requester who posts micro-tasks only. However, how do we prepare our system to handle complex tasks? Ones that not only involve performing a simple procedure but also some level of interdependence with other tasks to accomplish a certain goal?

Module preview We allow the requester to link tasks and select the guilds to whom those tasks should be addressed to, therefore automating the process of completing a complex task. As an example, if a requester needs to have a photo converted to black and white, printed and delivered somewhere, all he has to do is to create 3 tasks and select the guilds that would handle them.

System details The requester creates a Collective Intelligence Task, which is composed by several other micro-tasks. He specifies what those requests are and address them to guilds either of his choosing or suggested by the system. The system than uses the structure of sub-guilds to quickly find who are the best workers to show those tasks to. If the Collective Intelligence Task has components that rely on each other (such as the example given above), the following tasks are only shown to the workers after the previous step has been successfully complete. After all steps are done, the Collective Intelligence Task is considered complete. A problem that may arise is how to determine if the requester should approve each step or only the final result. As of now, it is suggested that each step should be validated by the requester, but it is possible that in the future automated mechanisms using machine learning techniques will be suited to handle that. This module allows us to build complex tasks, but only within a certain context (do x y z to a photo, or w q s to a song).

Module 4: Complex Collective Intelligence Tasks

[this one is optional to the system, but would be very, very cool to see happening]

Problem/Limitations Having the ability to complete tasks that are considered complex to nowadays standards, how do we use guilds to unleash the full power of crowdsourcing communities? And how do we coordinate the process of dealing with several different Collective Intelligence Tasks at the same time?

Module preview We allow requesters to create multiple Complex Collective Tasks and group them together to create intricate crowdsourced workflows. An example would be a software project with millions of lines of code going directly to workers in a #Design(major tag), #iOS, #flat, #sketch3 guild so that they enhance the look and feel of the application and then delivering that to a #Deployment(major tag), #RubyOnRails, #Capistrano, #AWS worker to deploy the application. If well specified, a complex project could take from some days to a few hours to complete, which is far better than the current standards.

System details The requester creates a Workflow composed by two or more Complex Collective Intelligence Tasks or regular tasks connected to each other by relations of cause/consequence. Therefore, one Complex Collective Task will only begin after the previous ones have finished, in the case that they are dependent on one another. For each step of the workflow, the requester approve or not the results. This module allows requesters to perform very complex workflows by using guilds and sub-guilds, therefore making the crowdsourcing platform a major solution to people who only need simple micro-tasks done but also for people who need to build huge projects within a budget.