WinterMilestone 1 Lone Wolf

From crowdresearch
Jump to: navigation, search

Hello, world! Shame on me, I thought our deadlines were all on 1/19 for some reason, so here I am - near the feedback deadline - just posting my initial Milestone 1 deliverable. I think I'd still like to join a team, but for now, consider this the first post from Team Lone Wolf - consisting of just me, a humble BSc Multidisciplinary Studies / 2016-2017 MSc Social Science of the Internet student looking to contribute to the Initiative/Collective.

Experience the life of a Worker on Mechanical Turk

I've always loved the idea of Mechanical Turk, and the reality was about what I'd expected: by turns monotonous (you want me to do what? for how much?) and exhilerating (what is this study about? you mean this submission will be read by a real human?).

What did I like?

  • When there were lots of HITs available, the ability to sort by price and "eligible for" to find higher-paying things that appealed to me.
    • Example: I found something involving writing an informal question to a real person's submitted answer. It ended up being sort of a playful, interesting diversion to try and develop a genuine enough interest in what someone wrote in order to be able to write a genuine-sounding response.
  • Once I got the hang of it, I could do a handful (up to the dictated limit) from a given requester rapidly, in a row, while watching TV.
  • I didn't seem to have any trouble getting paid. All of my submitted HITs were approved.
  • UI presented a relatively clear separation of overview, submitted/pending, and available work.
  • The feeling that if something really got out of hand, the trust I have in Amazon's brand would somehow translate into an agreeable outcome for me. This was never tested, but I think this gave me a sense of security.
  • The idea of "winning" a rarely available HIT was sort of exciting.

What did I not like?

  • When requesters want workers to leave MTurk to do a task, then return to MTurk with a copy-pasted code. This kind of integration felt tenuous and silly, leaving me a bit scared and off-put that my ability to get paid was at the mercy of someone else's ability to tie together reports or whatever from the two systems. Why wouldn't Amazon want to help more with that?
    • Example: Many of these kinds of tasks were survey tasks, so why not create a built-in survey tool? Amazon doesn't seem to be staying on top of how requesters are innovating use of the platform - sort of in the sense of "design patterns" for job types.
  • Seemingly no way to rate or view ratings for requesters--or perhaps there is one, but it's not being actively displayed in the HIT browsing view, where it needs to be. We have Glassdoor for companies now, eBay and Amazon have ratings for their marketplace sellers, professors are rated, doctors and tradespeople of all kinds... Why not someone who's asking me to do complicated work for fractions of a penny per hour?
  • Not knowing where I stand with some requesters with respect to future earning ability should I choose to work with them repeatedly.
    • Example: There was one particular requester, who I won't name, that had some kind of internally tracked "tier" system of e.g. Level 3, Level 2, and Level 1 workers. Sure, requesters probably want a way to weed out the better from worse workers who they maybe want to repeatedly work with (or avoid), but this particular requester's stated policy on this was "you will not know, unless we reach out to you... you can't know what exactly affects it, but if you do lots and lots then maybe you will go up," etc. It created a really high-bar "carrot" that, it seemed, was more designed to keep people working hard for them without having to pay as many people as much.
  • Overly difficult to find certain information that I would consider to be "getting started" or "basics".
    • Example: When a requester asked for my MTurk ID, I had no idea where to find it and it took me forever to do so. I searched MTurk's help, Googled it, and searched through page code, until finally I saw it was actually listed on the Overview page. Why wouldn't MTurk or the requester share any info about that? Maybe the broader MTurk forums/community could have helped, but... Now I'm spending my own time, for free, to come be a worker in Amazon's market? This way of offloading the burden of being competetive is not uncommon to a marketplace per se, but that leads me to my next point...
  • Working for pennies given the cognitive load and/or time burden of some tasks--and having difficult judging what an average or expected $/hour payout should be. That is, for me, at least, there seemed to be a dimension of picking tasks that I maybe enjoyed on some level, or felt a desire to contribute to in the name of Science, but there are many competitors for for exactly that kind of task outside of MTurk, so payment is going to be a large factor in my decision to choose one task vs. another. Why can't I therefore sort based on this tradeoff of $/hour?
    • Example: Two tasks both pay $0.08. One took me about 5-8 minutes per task and could be done while watching TV and surfing Twitter a bit. Another task, I had to concentrate on intently to be paid $0.02 for 3-5 minutes of work, doing nothing else. I can investigate individual HITs to perhaps make a guess about what one will take vs. the other, but why not create an ability to sort? Could you use machine learning to build up a profile of what kind of task a given user is good at, combined with how long others took to finsh a task (combined with their skillsets), etc. etc., to get something like that?
  • A big one for me, if perhaps a bit odd: no ability to use privacy-enhancing tools in Firefox such as µBlock, Privacy Badger (from EFF), Incognito Mode, cookie controls, etc. I'm not sure how you'd really get around this in the tool, but if more work could be done from within the platform, that might help.
  • Some tasks felt scammy. One in particular involved registering for a marketing webcast/mailing list. I did this... and then the HIT, which I'd already accepted, stated that they no longer needed help. After I turned off my anti-tracking software and gave them my email address. Pretty sketchy.
  • Last, bottom-line feeling I walked away with, from the last two especially, was one of not knowing what exactly my requester's goal was. Were they puffing up some marketing numbers by paying me pennies to do so? When I enjoyed creating a human-type exchange by typing a response, what was the experience on the other side (what was my work being "sold" as)? I'm not sure what the answer is to this problem.

Experience the life of a Requester on Mechanical Turk

This is still WIP, so nothing to report yet, but my intention is to create a playful and perhaps provocative request in which workers are asked to "sell" a dream to me. I will post more once I've created the request.

Explore alternative crowd-labor markets

One experience to report here: I installed an Android app called Gigwalk. I uninstalled it within a few days because:

  • The website security seemed flimsy overall. I had to register in the app itself, no profile page to speak of on their website. This created a feeling for me of a tenuousness and lack of follow-through in terms of having a way for me, as a worker, to maintain control over my experience, account information, payment preferences and status, etc.
  • I live near downtown Minneapolis, but there was basically nothing near me. There was one job worth $18 to scout out some bars nearby, but it required workers to be 30-40, and I'm not quite there yet. :)
  • The app experience was slow overall and buggy. The app seemed to be a simple browser wrapped around a mobile site, was slow to load (often taking upwards of 5 seconds to load a page), and frequently crashed.
  • Overall, being a worker was like being a second-class citizen. The site was 98% targeted at people looking to use Gigwalk to have work done for them. As a worker, my entire experience was confined to the app. I wanted more from the site, and wanted to see that Gigwalk cared more about me as a class of user.



  • What do you like about the system / what are its strengths?
    • Focus on improving on the "inaccessible task interfaces" of Amazon, that even as a skilled desktop computer user in the United States with English as my first language, I sometimes struggled with.
    • Consciously seeks to improve opportunities for marginalized workers first, rather than create the cheapest/smoothest experience for work requesters.
  • What do you think can be improved about the system?
    • Doing one kind of task over and over again, on probably a tiny screen with a tiny keyboard, seems like a good opportunity for repetetive stress injuries to occur. Who is liable in the event that workers develop eye or hand/wrist injuries? MobileWorks, task requesters, cell phone manufacturers, or workers? My guess is the latter.
    • No mention given here to how user community is enabled or managed as such, nor how user voice is necessarily incorporated into product design. This was a research project, so it's not necessarily strange to find these missing, but I'd look for these right away if this ever became a "real" product.


  • What do you like about the system / what are its strengths?
    • I like that it specifically works on the "fundamental attribution error" through the prototype task concept. This concept initially sounded a bit complicated and burdensome to me, but it now strikes me as a good strength of the platform, not burdensome.
    • Requesters scores are available publicly, as is user feedback about tasks and requesters.
    • If "project category" means categorization into e.g. "write some text," "human OCR," etc., then I like that a lot!
  • What do you think can be improved about the system?
    • This might be out of scope, but how will collusion be combated? On eBay for example, you can't sell to someone living at your same place of residence and provide a feedback rating/score to each other, or something like that. A friend had his account erroneously shut down this way because of a bad detection on eBay's part, but seeing this happen made the idea stick with me ever since: eBay is looking for this kind of activity. Is Daemo?

Flash Teams

  • What do you like about the system / what are its strengths?
    • Whether it can deliver on this promise or not is another question, but the goal ability of coordinating expert, team-based work seems unique and exciting.
    • WIP! The Milestone 1/2 livecast is happening now, so I'm going to focus on that a bit. :)
  • What do you think can be improved about the system?

Milestone Contributors