Daemo Guilds Feedback

From crowdresearch
Jump to: navigation, search

Daemo is developing guild features to address problems with trust, wages, quality and community in crowd work.

Historically, guilds have used social action by worker communities to achieve great results like fair and standardized pay for factory workers and meaningful quality ranking for lawyers. Daemo will use peer review within the worker community to distinguish worker quality levels. These levels will offer wage standards and help requesters understand how to pay enough to get the quality results that they need. Daemo will also offer a rich discussion forum to strengthen the worker community.

This is just the beginning, so we would like to introduce you to our solution and get your feedback on how we can improve it. Below are 2 sections: 1. Guild Scenarios, showing how we hope the guild system would feel for different stakeholders and, 2. Daemo Guild Details, discussing in detail how the guild system works.

Please read through our descriptions and then provide feedback to help us improve. Here are a few questions we are particularly concerned by:

  1. Does this mechanism seem fair to you?
  2. Does this mechanism seem fair to requesters?
  3. Are you likely to choose to do reviews if they cost approximately a fare hourly wage for your experience level? Would you feel this was more fair if we asked you to review for the good of the system, or if there was a standardized rate to review tasks?
  4. Are there things about this you think will not work?

1. Guild Scenarios

Daemo for Experienced Workers

I'm working from home after my regular work hours. I log in to Daemo, and as soon as I enter the system I see a menu showing settings, my guild, tasks I'm working on and so on. In the center of the screen I see a list of tasks and since I'm a level 4 worker, all tasks I see gradually decrease from 4 until the very basic ones, level 1. I could reorder these by date and other attributes but this is quite convenient usually. I start working. A little later, a new task pops into my feed and gets my attention. It's highlighted with a purple tint is a request for a work review, posted by Daemo itself. These are nice because Daemo always pays fairly well and responds quickly. I select the task, and it refers me to review a C++ task that was submitted by a level 2 worker. I check it and despite some minor stylistic things that could be improved I believe it is a suitable submission for a level 2 worker, so I finish my review by selecting an option saying that it looks good for their level and leave a few comments for the worker. I submit it and I’m back to working on my other tasks.

Daemo for a New Worker

I'm a new worker on Daemo. I just joined this morning, in my home, and want to get into working, and I see a notification from Daemo saying “Welcome,” advising me to take a look around the site first and on the forum before jumping into the work. I decide to see what sort of tasks are being sent out by requesters, and what their expectations are like. I have a few questions about how to find the best tasks, so I explore the Daemo Discourse forum and find that there are already some great pointers that I can learn from. I’ve tried MTurk before and this is incredibly much more supportive.

I get the hang of the Daemo platform and start working on tasks in my feed. I’m writing witty titles for blog posts for $0.10 per title. I am finishing them relatively quickly, and at this rate I think I will make about $6 per hour. I continue doing small tasks like this on my laptop. I mess a few up, but not too badly. The next day I see that many of my titles were accepted by the requester, and also that about 3 of them were reviewed by senior members and they gave me feedback on what I did well, and how to do better. ⅔ of my reviews were positive but one was on one of the tasks I messed up and it was not positive. I talked to other workers on the Daemo Discourse forum that are pretty active and full of useful advice.

I continued to work like this, trying to earn at least $5 per day, which usually took me between 45 minutes and one hour. Every time I logged in I would have one or two more reviews from senior members on my work from the day before, and a bunch of tasks accepted by requesters. I continued to improve and get faster. After 2 weeks and about 30 reviews, I logged in to find a message saying “Congratulations, 8 of your last 10 reviews were positive, you’ve moved up to Level 2”. When I looked at the task list, I noticed the usual ones I had seen previously but also a new kind with a green border and a small “Lv. 2” icon. These were mostly higher priced, and some were a bit more complicated than what I had been working on. I now earn $7/hr on average and occasionally I get the option to review new workers tasks at a similar rate.

Daemo for a Requester

I'm a semi-regular requestor and I need someone to edit some photos in Adobe Photoshop (PS), and I need a senior worker, because these photos need to be absolutely perfect as they'll be used in a school newspaper. I write out what I need and submit the tasks as prototype tasks. A few minutes later I get feedback recommending that I change one word in my request and giving me an estimate of how long the work takes to complete. I decide that I need someone at least level 3, because last time I had images and got level 2 workers the result was not quite good enough for me to use without other minor edits. Typically, I would pay around $6/hour, but because I need someone of a higher skill level, I'll pay the recommended $8/hour to ensure that they know how important these photos are to the newspaper, as they may be put on the front page. Based off of seeing other PS workers complete similar tasks, it typically takes 3-4 hours per photo, so I decide that a few hours for a few photos, at $8/hour is fair. I wait a few days, after the HIT gets picked up, and receive my photos. At this point, I accept most tasks directly as I love what the worker has done and will gladly accept their work and send them their payment. In some cases, I'm satisfied, but I do see a few things that could be enhanced, and because this is a senior worker, rather than rejecting the work, I issue a “Return for Revision” request, asking them if they can go back and fix a few details, and I will pay them as soon as they are finished.

2. Daemo Guild Details

For Workers (those who are level 2 or higher)

Experienced workers will be given access to work commensurate with their level as well as lower level tasks too. Tasks at their level will generally have a higher average hourly wage and higher quality expectations. Some of their tasks are randomly reviewed by more experienced workers and they are leveled up based on having at least 80% of the last 10 reviews be positive (these numbers will be tuned as we get more data on what works). They will also occasionally receive review requests, where they will be paid, according to their experience, to report how well a task was completed by a junior worker.

Pros:

  • Exclusive access to level tasks
  • Ability to improve the operations of the community by reviewing and being active on the forums

Cons:

  • They are asked to do reviews that might get annoying
  • Potential fear of competition from other levels
  • Different skills being valued differently
  • Expectations of pay may not always align with actual pay

For New Workers

New workers are automatically included in the guild and start at level 1. Some of their tasks are randomly reviewed by more experienced workers and they are leveled up based on having at least 80% of the last 10 reviews be positive (these numbers will be tuned as we get more data on what works). They will also have access to the Daemo Discourse forum which includes areas for onboarding as well as learning from peers about finding good work and working effectively on the platform.

Pros:

  • Workers can feel that their work quality and pay level are aligned.
  • There is economic and experiential upward mobility built into the system

Cons:

  • Must finish low level tasks before becoming a high level worker
  • Not sure how much support the guild will give if work is rejected

For Requesters

Requesters will generate tasks and get feedback on how to make them clear and effective as well as learning their expected duration before submitting them to the platform. In this process, they will select which level of workers they would like for their task and a price at which the tasks will be posted. They will be shown data about the average hourly wage and worker percentile of each level and will make their decisions based on that.

Pros:

  • Higher quality work is more accessible and directly requitable
  • Higher speed, targeted work because workers are more likely to find optimal tasks for them quickly

Cons:

  • More expense in some cases
  • Somewhat more complexity in releasing task

Feedback Questions

  1. Does this mechanism seem fair to you?
  2. Does this mechanism seem fair to requesters?
  3. Are you likely to choose to do reviews if they cost approximately a fare hourly wage for your experience level? Would you feel this was more fair if we asked you to review for the good of the system, or if there was a standardized rate to review tasks?
  4. Are there things about this you think will not work?