WinterMilestone 2 MarkWhiting

From crowdresearch
Jump to: navigation, search

Attend a Panel to Hear from Workers and Requesters

I was able to watch the panel live. I had the following thoughts during the panel.

Though the selection of workers were arguably non typical, I observed that their seriousness about their interaction with Mechanical Turk was palpable. For example, it was mentioned that they had done things like woken up in the middle of the night for a certain type of task. Also, it was clear that the net value of using Mechanical Turk as a worker, despite any issues and challenges, was distinctly positive. People are too busy to take jobs, or have other reasons that normal jobs are not easily compatible with their lifestyles and MTurk provides for that context really well.

From the requester perspective, I observed that all of them had spent a lot of time trying to deal with the complexities of getting MTurk to actually work for them. They expressed a diversity of ways of doing this, but some common patterns emerged; trust, wage management, worker relations etc.

Reading Others' Insights

Worker perspective: Being a Turker

MTurk offers workers the ability to remain physically entirely anonymous. Nobody cares what they look like, just that they do the work. This seems to be a key benefit for many workers, and could be so for requesters too.

The management of workers is a complicated situation. There are tools in place but they often fail to work well and can easily be cheated by both the workers and requesters. An example of how this happens is the results we can observe of allowing requesters to be responsible for choosing how much to pay workers. Several things happen, but the most obvious one is that pay is lower and quality work is harder to find.

Worker perspective: Turkopticon

Workers and requesters both seem to benefit from a system like this, however, it also seems really limited in that it is external to the underlying work system. Considering MTurk in the same way we consider a server farm or something else is somewhat dehumanizing, but Amazon also seems to have not taken this idea to its logical end, by creating a highly refined API and a properly encapsulated system.

I found it interesting to think that there is a labor surplus on MTurk, which was mentioned in the paper. I think in practice, this is only true because of a lack of efficiency.

This paper also made me consider how natural systems evolve solutions to unplanned problems but often never move beyond the limitation in some way.

Requester perspective: Crowdsourcing User Studies with Mechanical Turk

We can see that there are some inherent complexities of treating humans like a line of code without strong tools to ensure that commands are interpreted correctly, time is accommodated appropriately, and checks are in place to look for errors.

This paper also left me with questions about how good workers are at detecting other worker cheating, and possibly designing systems to avoid it, being experts with similar incentives.

Requester perspective: The Need for Standardization in Crowdsourcing

An automated market approach seems like a really great way to go about managing crowdsourcing. This is what we generally do to manage computational systems, and I think it could be made to work really well in the crowd work setting too. One observation I took away from this post was that most plans are currently focused on attempting to entirely computationally manage crowdsourcing, but there might be interesting ways to do it with integrated third parties.

Both perspectives: A Plea to Amazon: Fix Mechanical Turk

I think one of the most important points gleaned from this post is that we know a lot about what is wrong with MTurk and we know the problems are pretty standard across the board. The problem is quite well defined, and right now it is not at all competitive.

From a requesters perspective, I found it interesting that the cost of using MTurk is sometimes greater than the cost of doing the work elsewhere, assuming that you might need to hire a programmer to use the MTurk tools well and so on.

The discussion of value of a worker and value of a requester was also very interesting. The addition of notions like this stands to make the system more complicated, but perhaps theres a computational curation that can be done to make the complexity of using MTurk much less burdensome.

I really liked the closing remark about how tiny MTurk is and how much larger of a potential business there is to be had in this space.

Soylent: A Word Processor with a Crowd Inside

This project is very neat. I tried it a few years ago and was fascinated by how well it worked.

Reading the paper again, I was really interested by the notion of 'patterns' of crowd work and I think this is an idea that could be efficiently leveraged in the future.

Do Needfinding by Browsing MTurk-related forums, blogs, Reddit, etc

I expected there to be many great submissions from other sources, so I have included some observations I found by running a survey on mechanical turk last week, asking workers to tell why the use the platform, what they do and don't like about it and what they would tell requesters if they could. The full data I'm working with is available here: File:Mark Whiting Milestone 1 Results.csv

Workers understand the system and how to use it.

"The keys to a successful project is this: 1) Proper use of qualifications, 2) Clear and concise instructions, 3) Fair and transparent approval/rejection policy, 4) Efficient HIT design, 5) Communication with the workforce, and 6) Fair compensation."

Workers want to be treated fairly

"Just contact me if you have a question about why I entered something the way I did instead of rejecting. Most of the time I can either explain it or accept the rejection and apologize for not doing it as they wanted. I admit when I'm wrong or if I messed up but most of the time, it's justifiable based on their instructions for the HIT. Really appreciate the work and confidence they have in me!"

"Rejections are more than just not paying a worker. It goes against our rating and can really really affect us. Rejections are completely deserved for people who don't follow rejections, of course, but sometimes requesters use it and abuse it."

Some workers prefer smaller tasks, and think that is more efficient.

"Instead of creating long tasks, they can create hits in parts. Also, I would like requester to pay a fair price. Just like this particular hit should be of at least $.25"

(note, I did pay all the workers more with bonuses after the fact)

Workers don't like CAPTCHAs

"Having to fill out a captcha every 35 HITs. Sometimes that's one captcha every minute depending on what I'm working on. And if you fill out a captcha too FAST, it tells you it's wrong, so sometimes I have to attempt the captcha multiple times before it lets me continue working. And if you fail a captcha 3 times in a row it locks you out of the site for 5 minutes! Also, if you fill out a captcha correctly, but someone else has taken the HIT you're trying to grab, it won't accept the captcha and you have to go to another HIT and try again. Captchas are very demoralizing and frustrating."

Workers are pragmatic, and strategic

"The feeling of competition with my fellow turkers. I am working on Mturk full time right now, which means I need HITs, which means I can't always help others as much as I want to. If I'm working on something amazing I want to get the most out of it before anyone else discovers it. That sort of makes the site feel like Lord of the Flies sometimes. I am helpful by nature and I hate being greedy but I have to pay my bills"

Workers want to be able to talk to requesters more easily

"If we want to tell the requester anything we are not in a position to tell them directly. This is a main problem facing a lot of people. Some requesters one day they will stop giving us work. The reason they wont tell. If we take their hit it is showing you are not able to do their hits. That is a very very bad thing. The day we do the work they will tell well done and all the next day no work. That is a very bad nature."

Synthesized Needs

Worker Needs

  • Workers need to remain anonymous in as many practical ways as possible. Evidence: Multiple papers discussed this as well as members of the panel mentioning how they value this. Interpretation: Worker IDs is not a great solution and there may be better ways to encapsulate this data from the requesters while also allowing for better decision making.
  • Workers know a lot about how MTurk works and have good insight on how to use it well. Evidence: I've quoted some above, and things like Turkopticon demonstrate this to an extent. Interpretation: Workers could be used to formally improve the use of MTurk for requesters.
  • Workers like the isolated experience but don't like the implications on use of the system. Evidence: Workers don't like CAPTCHAs as I quoted above, and also tend to be generally aware and not amused by other similar mechanisms. Interpretation: Perhaps there are ways to give the benefits of working flexibly from home without needing so much gold standard management, e.g. using a decentralized checks network.
  • Current pricing methods are really bad for workers. Evidence: This is an issue that many of my study participants mentioned, and many papers mentioned. Requesters are not experts and pricing, and workers often pay the 'price' as a result. Interpretation: Workers would greatly benefit from having automated pricing that followed laws and optimized for good value work.
  • Workers like to collaborate but it's not simple to do so. Evidence: My study reflected this in several cases where workers talked about the competitive nature of the landscape, Turkopticon is also positioned to deal with this but seems, based on the related paper, to be an imperfect solution. Interpretation: Integrating these features into the system would be a start, but its also important to think about about how these features actually work
  • Workers really want better ways to talk with Requester. Evidence: My study and the panel discussion both reflected this. Interpretation: There is no reason that a formal way to do this should not be built into the system. That said, its complicated to do a really good job.

Requester Needs

  • Requesters often need to single handedly do many of the things that employers do to manage their employees. Evidence: The panel discussion as several of the papers talked about how much work it takes to get good use of MTurk as a resource. Interpretation: Parts of this that we can, we should, automate, or further outsource.
  • Some requesters outsource taks management to workers. Evidence: This notion was loosely mentioned in the panel and also in many workflow related studies, including Soylent and last week's readings. Interpretation: As humans, we have a lot of knowledge about how to do this well in management literature etc. There are differences here, but we should capitalize on the similarities first.
  • Requesters don't know how much to pay, and this hurts them. Evidence: This is in some ways more from the worker side, and is mentioned above. Interpretation: Automated pricing tools for requesters would make life a lot easier, and I think for the most part, they would still end up getting good work done at a very efficient price.
  • The cost of doing business is higher than it looks for requesters. Evidence: The discussion of how requesters might pay a developer to make things run smoothly. Interpretation: We have to make it easier to do common things, and easier to adopt the valuable solutions other requesters have found for this.
  • Requesters may benefit from communicating more. Evidence: As the panel demonstrated, and some papers mentioned, if requesters shared what they knew more, it might be much easer for them to get good value out of the system. Interpretation: The same communication solutions discussed above might be really helpful for requesters too.
  • Requesters need something like 'patterns' for a lot of the work they have. Evidence: Requesters seem to have a few specific types of work and most seem to focus on only one or two of these most of the time. Interpretation: Finding best practices for each pattern and making those available as a kind of package deal for requesters would be a great way to automatically evolve the system to work better.

Other thoughts

  • Could we enforce a minimum wage standard for workers that can be dynamically augmented depending on demand and needed speed of work?
  • Could we think of MTurk not as a server farm but as a system of server farm, virtual machines, and APIs, in which both the farm and the machines are in some way represented by workers at different levels?
  • Could workers be used to formally manage workers?
  • Is it reasonable to think that almost all work could be done with a limited number of patterns?
  • What kinds of work can't be done with patterns and why?