Difference between revisions of "WinterMilestone 3 Despicables DarkHorseIdea: A Holacratic World"

From crowdresearch
Jump to: navigation, search
(Our basic idea)
Line 111: Line 111:
* This New RD is taken into account when calculating further reputation.
* This New RD is taken into account when calculating further reputation.
[https://en.wikipedia.org/wiki/Glicko_rating_system Glicko Rating on Wikipedia]
===== Reputation of a team =====
===== Reputation of a team =====
Stemming from the reputation of the individual members, the team will also have an overall reputation.
Stemming from the reputation of the individual members, the team will also have an overall reputation.

Revision as of 18:33, 31 January 2016

This is our dark horse idea milestone for the week of 26th January to 1st February, 2016.

Team Despicables

Team members

  • @varshine
  • @kr.prastut
  • @afreen
  • @rachitchopra

Our philosophy

"The future of crowd work market" mentions this statement:

Can we foresee a future crowd workplace in which we would want our children to participate?

We believe that a crucial point of building a self-sustainable crowd-sourcing platform is that it must have a strong philosophy. A system. A framework. And we flared out to seek examples in the world where there was a system that is in place, where the human facet is of utmost importance. Since our vision is see that our children could participate, we moved onto schools and the system they use. They have groups and they have a sort of hierarchy in terms of teacher student relation. All these classes fit into one school. Since this wasn't an ideal base case for our idea so we moved onto the concept of governance and a special case of governance: Holacracy

We had several notions while we were brainstorming our idea: Why the need of system?

  • We want the reader to think in terms of people having "jobs" in the crowdsourcing industry.
  • The biggest and the major problem in the already existing crowdsourcing industry is communication.
  • The second major problem is that of trust and human motivation
  • The importance of human facet. That requestors and workers are actually humans.

The challenges to a system

  • People want autonomy - they like to pick and choose there kind of work, their timetable
  • No one wants to be managed, yet everyone wants better representation, better say - better managed.

We keep in mind that currently requestors post work if they want to post work, and workers work if the feel like working. That's the beauty of it, rather than a limitation.


Wikipedia defines it as follows,

Holacracy is a social technology or system of organizational governance in which authority and decision-making are distributed throughout a holarchy of self-organizing teams rather than being vested in a management hierarchy.

In layman terms, holacracy is a form of distributed governance as opposed to a central government like in democracy. The definition we are using is: Holacracy is a special case of governance where decision-making power is distributed across small, autonomous teams rather than one central management group.

A holacratic framework

Concept of Cohorts

We propose that we make small cohorts/teams where power can be distributed themselves. Such a system is envisioned keeping the following notions in mind:

  • Motivation: Cohorts will bring about healthy peer competition. If a person join's a group with higher performing workers, he/she would automatically want to work more ("You are the average of 5 people you around with" rule)
  • Highly skilled and educated population: This can be a major asset to any requestor. Having specialized teams whose group members have almost the same reputation will ensure quality results. The process of submission will also speed up.
  • Interactions: The cohort will lead to further interactions. If we could learn so much from the Daemo cohort, imagine what could happen with millions of people of different regions, cultures, experiences.

Cohortness can be based on:

  • Extending the idea of boomerang to create cohorts:
    • If a worker and requester rate each other well, they feature high up their work feed in the future.
    • So a certain requester may have a bunch of workers he/she is comfortable working with, and a certain worker has a list of requesters he/she is comfortable with.
    • Use this data to create small self-governing groups.
  • Similar skillsets & reputation with secondary factors like age/demographic/some common denominator.
  • A upper threshold can be set around 20 people (not too big nor too small). A mix of requesters and workers that will use the holocracy concept to regulate the tasks and work they do.

Leaving cohort / Joining new cohort

  • You can leave the cohort any moment you like.
  • Since working in cohorts is highly advisable, you would have the option of joining another team.
    • The choice of cohort is not yours. You would automatically be added to a cohort team based on your reputation & relevant technical skills.
  • The algorithm of joining a cohort is based on pipe-lining structure.
    • If you leave the cohort, the next in line person get's added to the cohort.
    • Similarly you will also join a cohort automatically

Extra Benefits:

  • The requestor get's access to flash crowds.
    • Teams having same experience & same reputation will help the requestor finish tasks faster compared to individuals.

Feedback Mechanism : Through issue tracking

We felt that currently the feedback mechanism like issue resolving and guiding is done mostly by volunteering. That is why we wanted to make the already existing informal and unorganized set of people who help people out, be official people of contact. No implicit people would be defined. Anybody can volunteer and take responsibility.

  • Similar to Github's issue tracking mechanism, the team would be presented with a system where they can post issues pertaining to the team.
  • Anybody can volunteer to solve any issue.
  • We also propose a new factor in calculating reputation. An incentive will be given to the worker who closes the issue.

Reputation Calculation

As proposed above

New Reputation = Task based Reputation + reputation credits you get from solving issues relevant to the team.

We also introduce a modified version of Glicko Rating System for reputation calculation

Modified Glicko Rating for Reputation Calculation

The following is an example of what we propose where a new worker joins Daemo. We also introduce a concept of Reputation Deviation (derived from Rating Deviation)

  • The RD measures the accuracy of a worker's reputation.
  • For example, a worker with a reputation of 1500 and an RD of 50 has a real reputation between 1400 and 1600 with 95% confidence.
  • Twice the RD is added and subtracted from their reputation to calculate this range.
  • After they complete a task, the amount the reputation changes depends on the RD: the change is smaller when the worker's RD is low (since their reputation is already considered accurate), and also when their requestor's RD is high (since we are only taking in account requestor's RD). The RD itself decreases after a task completion based on the result, but it will increase slowly over time of inactivity.

 Step 1: Determining Reputation Deviation: New RD = min(sqroot(RD^2 + c^2*t), 350)
  • t is the amount of time (rating periods) since the last time a worker worked, and 350 is assumed for an new worker.
  • The constant c is based on the uncertainty of a worker's skill over a certain amount of time. It can be derived from a thorough data analysis, or estimated by considering the length of time that would to pass before a worker's reputation deviation would grow to that of an new worker. (He/She has become inactive)
 Step 2: Calculation of New Reputation

Factors of concerning the calculation of new reputation:

  • Old reputation
  • Result represents the outcome of the worker vs requester. Could quantify into:
    • Success = 1
    • Rejection = 0
    • Any other base case if wanted
  • Involve the reputation from contributing to the team
 Step 3: Determining New Reputation Deviation: New RD = formula involving old RD and the factors concerning the calculation of new reputation
  • The function of the prior RD calculation was to increase the RD appropriately to account for the increasing uncertainty in a workers's skill level during a period of non-observation by the model.
  • Now, the RD is updated (decreased) after the series of work.
  • This New RD is taken into account when calculating further reputation.

Glicko Rating on Wikipedia

Reputation of a team

Stemming from the reputation of the individual members, the team will also have an overall reputation.

New Reputation = Average of team individuals rating + percentage of closed issues/total issues.

Questions + Clarifications

The reader should be aware that the following system has been proposed only for the tasks which are posted on AMT and not oDesk (Basic tasks like image labelling etc.)

Q.1 How would new workers/requesters join a group?
Ans. Initially the new worker will be required to complete 'x' tasks to create data points. Then the modified glicko-repuation system would automatically boomerang the new worker to a cohort which is relevant to the worker's skill & reputation.

Q.2 Unfair to new workers/requestors or a lot of stress on the new workers
Ans. Imagine yourself in school. A new student joins. His/her value proposition is to actually perform. Nobody is going to spoonfeed the person to do stuff. It's also an added motivation to work hard initially to join a smart team. The whole experience of being a part of team, covering for one, getting upto speed, helping out others, adding to your rep, doing same issues where you can be heard seems viable enough.

Q.3 Do I have to do only the tasks assigned to the team?
Ans. No but you are advised not to. You could do the task even if the team advises against too. It's your choice. But since we don't want a court system to be in place to hear every issue out, we proposed a common issue tracker. On the downside, you are advised not to post issues which are not relevant to the team.
Let's take some cases to:

  • You are ill-advised by 10 people to not take the task:
    • Why would you still take it?
  • You are advised not to take a high paying task:
    • Everybody is in for money so there might be a valid reason.
  • You are advised against a task in which you feel you can do justice to:
    • Your cohort is loosely related to your skillset.
    • The tasks we are dealing with are basic in nature. Mostly everyone can transcribe or label images.

Observation: In the worker-requestor panel, most issues were common. The issues were common to such an extent that we could boil it down to 2 major issues: Representation & Reputation.

Q.4 What would happen if I leave suddenly or don't want responsibility? What if nobody else in the team wants to step up too?
Ans. You are free to leave and join another team. Pipe-lining structure will handle all such cases. About the responsibility - we want to hope that there are people in every cohort who actively volunteer. Since cohort reputation = average cohort members reputation + the amount of issues being solved, most teams members would want to solve issues which indirectly increases there reputation too.

Q.5 I'm a worker whose work got rejected - I think I did a good job for that description and the task could have been better designed. Now who'd solve the issue? and how? If we randomly make people into groups, then they may lack the knowledge behind the task to able to design it
Ans. Since design is perceptive, maybe some other member of your cohort could explain why your task got rejected. But if it's the requestor's fault, most cohort members would be having problems hence the requestor get's assigned to the issue. That's the beauty of our system. Rather than criticizing each other for our faults, the issues would help people learn about their shortcomings.

Q.6 Making cohorts is an investment of lot of time and effort. Currently people do it now out of their free time. If you make it official then it becomes work? Ans. Make effort now to make envision a better future. Creating a choice of choosing between jobs vs crowdsourcing for our children is a huge achievement.

Addressing global concerns

A global panel will consist of eminent worker's and requestors. Apart from moral duties, this panel also acts as mentors to new teams. Sometimes, a lot of people may be running through a common issue i.e., problems with the platform, a requester-organization being troublesome, issues with the overall system. For such issues, we propose a new way of collaborative issue-tracking:

Here is a video describing it.

This system allows for anonymous raising of issues by any entity in the crowdsourcing platform. If the issue is a common one, people can support it by voting on it. The holacratic panel of every group will address an issue only if it hits a certain threshold of support. The higher the upvoting, the more critical the issue is, the faster it will be addressed and hence, resolved.

A call for a better system

We see that with a foundation of holacracy which helps us address each entity personally and with a way to address the global concerns, all entities in the system do not feel like just computers, but actual human beings.