Milestone 1 crowdshrimp

From crowdresearch
Jump to: navigation, search

Working as a Mechnical Turk Worker

Kristine Hoang

What I like is that you’re given clear instructions when you click on a task. In this sense, the work is easy and attainable. But I’ve found that there’s more I don’t like than what I like.

I didn’t particularly like how unsustainable the whole system seemed to be. For me, I was most drawn to the tasks that were higher paying regardless of what the tasks was, and for people who also are drawn to the quality of the work rather than pay, it’s difficult to feel motivated when the end-goal (financial, intellectual, emotional) doesn’t seem grand or guaranteed.

Thus, I think the question that ultimately consumes me is this: is some sort of financial reward enough to drive a crowdsourcing platform’s economic sustainability? (By sustainability, I’m referring to worker fulfillment [and, subsequently, commitment] and financial comfort). After reading “MobileWorks: A Mobile Crowdsourcing Platform for Workers at the Bottom of the Pyramid,” I was also curious to know how much percentage of Indians making money off of MobileWorks are making above or below their basic living standards.

Further elaborating on the relationship between my financial goals and finding quality work, there was an instance where I thought I had found a $0.50 task (which was to tag home design photos), but when I found out it would take two hours to finish the task, I was no longer motivated to do it. Though that would be 25 cents an hour, which is more than what most rewards for individual tasks on MTurk are, it still wasn’t enough.

Also, when it was also frustrating when I get message:

“You are not qualified to accept this HIT. You either do not have the required Qualification or, if you possess the Qualification, your value does not match the value required for you to accept the HIT.”

I got this message from work that asked me to find out if a company was hiring and then again for work asking me to watch videos. As a new MTurk worker, it’s hard to understand why I would not be qualified for something that seems short and easy when it paid so little. Thus, I think a good approach to future crowdsourcing platforms might be to recognize the frustrations of workers, and find solutions to them. At least in my experience, frustration was a defining characteristic of finding and doing work on MTurk.

Note: I tried to finish HITs to reach $1, but for some reason my account got “temporarily suspended” – I think from trying to submit something twice. There was one task where I was trying to submit and it wouldn’t go through, so I kept pressing the submit button and refreshing.


Adam Lui

I've had a Worker account with Amazon Mechanical Turk since January 2013, using it on and off throughout the years to make a side income. Recently, I asked myself: could I make a full-time living at this? I put this idea to the test last summer, and began turking for up to 8 hours a day.

My earnings were drastically increasing with each passing week, with the support of forum members and new friends I'd met through the turking community. By the fall, I was setting weekly goals of $2000 a week. Then just these last few days, I challenged myself to something unprecedented among turkers: could I actually make $1000 in 2 days?


Well, I guess the answer was yes! This is what amazes me about mturk. You can make as much as the amount of time you spend and knowledge you apply honing your skills, similar to other systems for making money. Take poker for example. A beginner, recreational player just starting out can make a lot in his or her session, but will typically win only a small amount, or lose money. Over time, if the player chooses to learn from other players, or even develop his own unique strategy, his earnings increase. If the player chooses to spend eight hours or more per session, he is no longer recreational and the profits reach yet another level (as variance in earnings from factors such as luck diminishes).

What's bad for the player who has reached the final levels described is also bad for the career turker. Unlike a wage job (or even a salary one), piece-pay work incentivizes the worker to create higher earnings that correlate to the volume of tasks completed. In poker, there's no set hourly wage, either; if the player considers himself better than the majority of his opponents at the table (and thus possess a higher edge than most) he or she will want to play longer sessions to optimize his earnings. This same desire to optimize earnings affects career turkers who also feel they have an edge over casual or amateur turkers. It consumes their days, their nights, even their dreams!

Does anyone ever dream about turking? I had a dream last night that all the HITs I have pending got rejected and my approved percentage plummeted. Which is why the first thing I did when I woke up was check my account. Maybe I need to take a break. -TurkMileyTurk (!!!&p=540108&viewfull=1#post540108)

Experience the Life of a Requestor

Kristine Hoang

I enjoyed the requestor part of MTurk much more than being a worker. Creating tasks is more fun and requires a bit of creativity (especially for surveys or writing tasks; I created one asking, "Write a summary of Breaking Bad." I felt like both being a worker and requestor are frustrating at first (this is my first time experiencing MTurk), but I'd imagine that after a while you get used to the interface and, even, the frustration of not getting something right away (however, I still think making easier-to-understand interfaces is always a noble goal.) When I first landed on the Create Project portion of being a requestor, I got confused at first as to what I was supposed to do on the right side of the page (which is where the example of the types of projects are laid out). It took me a couple of minutes to realize they were only examples for what was on the left -- the types of tasks. I actually spent a couple of minutes creating a project I didn't really want to create because I hadn't noticed you could switch to a creating a different kind of task.

Though I created some batches, I wasn't able to get any batches ready for review. It could be perhaps that I didn't create the project correctly, no one has been interested in the tasks I created, or my reward for the tasks is too low (5 cents per task).

Christine Spiegel

I find the requester interface to be very frustrating. It helps that I am a full time worker, since I know the possibilities of it once you have learned it. Still, it is not very intuitive, and it quickly becomes clear why there are so many new requester mistakes. For example, Mturk defaults all tasks as "Masters" which considerably cuts down on your worker pool; you have to go into advanced options to remove that qualification. Once you know that, it is much easier to curate a personal work force. I removed the Master's qualification and instead set a minimum number of approved tasks and a minimum approval rate. From there, I posted test hits, and granted qualifications to people I know from worker forums and anyone who submitted quality work. Another observation I've made is that if you don't have the technical knowledge, making more involved batches can be difficult. For example, I've posted work before as a requester just to learn how to do image moderation- I was going to have my workers categorize vacation photographs from the last few years so I could more easily make an album for my mother. I was unable to do so, because I had to manually create the .csv to upload and it ended up being easier to just do the work myself.

File:Batch 1822126 batch results.csv

That being said, if I were better able to upload files, there are millions of possibilities. From sorting my pictures to uploading essays to be proof read, there are so many ways this could be used. Sorting tasks into more manageable chunks to be completed by many workers will increase accuracy, and once you have your own trusted workers things run incredibly smoothly. I now have my own turkforce that I can tap into at a moments notice. This is nice because I have a ready source of quality data I can use at any time, rather than posting work to "strangers" and potentially getting data I will have to toss for inaccuracy.

Exploring Other Crowdsourcing Markets

Kristine Hoang

I chose to explore oDesk, which I've heard of before and would later learn I don't like. It seems extremely difficult to get work even if you are intelligent and qualified, and the amount you get paid seems extremely low. This definitely works for new freelance an independent workers, I believe, but not seasoned ones who know how to charge private clients an amount that's fair to themselves. The need to break into oDesk -- that is, the seemingly initial struggle of getting any work on there at all -- seems like something that would push people away.


Kristine Hoang

MobileWorks: A Mobile Crowdsourcing Platform for Workers at the Bottom of the Pyramid

The reading on MobileWorks was an insightful look into what crowdsourcing can mean to people in developing countries. What I'm curious about, though, is how culture affects perception of quality of work. Does socioeconomic environment make someone from a developing country view work differently from how I view it, or do intellectual fulfillment and financial longevity still matter to them the way it does to someone from a first world country (who might be less motivated to work for less money). I also found it intriguing how Turkers in India are often more educated and would like to know their views on why they enjoy/dislike MobileWorks.

As for as technology goes, I think simplicity works for something like this. A useful thing would be to know how qualified you are for a certain task instantly without any clicks, and if there was a built-in system and display that could calculate your hourly rate in real-time.

Break It Down: A Comparison of Macro- and Microtasks

What I found interesting here is that a task takes longer for microtasks than it does macrotasks, yet the quality of work is still better. What I wish to learn further from this study, though, is how much more or less money a person made doing microtasks for one day than if they did macrotasks.

mClerk: enabling crowdsourcing in developing regions

For mClerk, I liked that it re-purposes cheap mobile phones and makes them of more use to people, and that it uses photos -- that is, visual language -- for its primary task. As for the problem with acquiring local language font support, I think there might be a way to create a system that contains a database/bank of special characters with the help of linguists and local experts. Perhaps we could integrate this database/bank into mClerk -- or any similar app, for the matter -- as an extension. But I think the 'broken language' approach mClerk works as well.

Flash Teams

I still need to marinate my thoughts on using flash teams for crowdsourcing work. I have yet to figure out exactly how this works economically, and how it affects the workers psychologically (my main question: for a system like this, what exactly will the workers be motivated by?) However, I do like the idea of reusable blocks because it allows the workers to utilize their areas of expertise.