Guilds Milestone 9

From crowdresearch
Revision as of 19:13, 27 March 2016 by Markwhiting (Talk | contribs) (Finding Answers to Fundamental Questions About Guilds)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search


Micheal's Introduction

Finding Answers to Fundamental Questions About Guilds

We posed 3 questions, collected submissions and votes for more than 24 hours, then removed duplicates and grouped the responses according to themes. We then rewrote key themes with a focus on all the responses being of a similar tone and level of precision and reran the 3 questions with the newly written responses pre-populated as possible answers for another 24 hours. To get good answers and votes, we used All Our Ideas, a web service that allows for pairwise ranking of responses to a question.

Below we list the three questions and show the numbers of responses, and numbers of votes for each round. Results are also linked to, externally.

Questions (results linked) Round 1 Answers Round 1 Votes Round 2 Options Round 2 Votes
Q1 What is the social process for leveling in guilds? 48 181 6 50
Q2 What is the effect of getting promoted in guilds? 25 92 6 69
Q3 What is the minimum viable product of a social space for a guild? 23 110 5 52

Note: At the results links Combined Strength is an unweighted average of the second round votes and first round votes for ideas of similar group.

After collecting the data we hung out with members of the guild group and @michaelbernstein and discussed detailed answers for each question based on our findings. The recorded discussion is available online.

Q1. What is the social process for leveling in guilds?

Guild Leveling Diagram

Based on the Combined Strength of round 1 and 2 votes, leading ideas were:

  • Workers fulfill criteria set by the Guild & senior members review the request (might be paid for from worker)
  • Continuous leveling with occasional exams or critical tasks that are directly reviewed by senior members

In the first one, the guild would set a criteria (we proposed the default would be completing a certain number of tasks but this could change), once a worker completed that requirement they could ask to be evaluated for leveling, and then members of higher levels would discuss to make a decision about if that person could level up or not. This has the nice advantage that human reviewers may be very good at telling if a worker is a spammer by reviewing their work, while computational methods might fall short on that aspect. To support this system, the worker being reviewed would be asked to pay some nominal amount for the reviewers time.

In the second option as we discussed it, the guild would make random requests to higher leveled workers to review a task. The person doing the review would determine what level they thought the work was at. For example, for an image description task, a short loosely related description might get a level 1 rating, while a very detailed and accurate description might get a level 4 rating. This process would continue until a worker receives a critical ratio of reviews that state their work is at a higher level. In most cases, they would increment by one level but in some rare cases, when I worker is very talented, they might increment by more than one level at one time. This review process would be payed for by a minimal guild tax (although other mechanisms like review quotas for higher level workers, or one time payments at the time of leveling were also considered)

After discussing these options at length, we are going to move forward with the second option: Continuous Leveling with Randomized Reviews.

Q2. What is the effect of getting promoted in guilds?

Guild Payments Diagram (the requester approval process is not modeled here at this time)

Based on the Combined Strength of round 1 and 2 votes, leading ideas were:

  • More money for the work you do
  • Better work Opportunities (e.g. interesting or complex tasks)

The critical discussion here was around how we could provide more money or better tasks in a way that is both reasonable, reliable and fair. This is tricky for either idea because it is a competitive market place, and because anything we do to change that, like making a priority task feed for higher level workers, could insight problems in the community.

After discussion, we chose to use an approach in which there are not strict price requirements, or payment requirements for given tasks, but where a desired hourly wage is reported by the guild on a per level basis. So, when a requester is setting up their HITs they would see several levels, e.g. level 1, level 2, level 3…, each with a different hourly wage goal and worker percentile. We are assuming that requesters have good data on the amount of time a specific task takes, so they could quickly see what the different guild levels would mean for their costs. Because the hourly wage levels are just goals, they might underprice for a guild level, which at some extreme would lead guild workers at that level to not do the work. In initial rounds these price points will be set manually by the guild, but later they will have data on actual mean hourly wages for each guild level so they could adjust their recommended prices accordingly. This approach means that pricing is somewhat dynamic but based on real market behaviors such as the wants and needs of workers and requesters, as opposed to a computationally simplified model.

In short, higher level guild workers will expect to get higher hourly wages and requesters will get a sense of how much money to offer tasks at, in order to entice workers at a given level in the guild.

Q3. What is the minimum viable product of a social space for a guild?

Discourse Forum for Daemo

Based on the Combined Strength of round 1 and 2 votes, leading ideas were:

In discussion we decided that a forum with some kind of reputation features would be a good level to start with. Daemo already implements Discourse which can serve this purpose well. Additionally we talked about how most discussion should be public, but some high level administrative boards could be private.

In short, we will use Discourse as is, and adjust its set up if need be moving forward.

Mocking Up Guilds

We are making a few assertions about Guilds on Daemo:

  • That Daemo is all one guild, so there is no need for a login, or guild on boarding.
  • That everyone starts at level 1.
  • That requesters have data about task design from prototype tasks and that that system works fine (we don't need to design that right now)

What to Mock Up?

  1. Explaining how leveling works for the worker, and how peers in that process see what they have to do in order to rate someone's work.
  2. Conveying to requesters how they should price their work and what they get for it if they choose different level options.

Mock Up Links

Ideas on Visual Design and Interactions

Each aspect that we are mocking up has a lot of little details. For example, making visual explanations of how we teach workers about levels and tell them their current progress.

Here are a few aspects that mock ups could include to work well:

For all workers

  • Visualization of Level and Progress. Being able, much like in games such as RPGs, to see in a glimpse where workers start, stand, how much they have progressed after having completed X Tasks, and how much effort they need to put before reaching the next level etc. An example of this is also found in Visualization of leveling by @yoni.dayan, an alternative approach is to do this as something like an event log or news feed.
  • Conveying the value of leveling implicitly in a way that shows the benefits to work-life balance ("i know what i need to do if i want to have better tasks and a bigger role in the guild") through a less static crowd working conditions (prospects of evolution).

For senior workers

  • How do we make sure that their review process is fast and easy?
  • Are they forced to do those "peer reviews"? And if so, at which pace? Depending on the answers, we can envision implementations like notifications showing to senior members that they have X peer task reviews to do. If we adopt a "star map"/talent-tree like visualization for each guild member, perhaps their progress would be blocked until they review their peers.

For requesters

  • Teaching requesters about levels
  • Conveying to requesters how they should price their work and what they get for it if they choose different level options, much like the pricing plans of services.