Milestone 1 AltaMira

From crowdresearch
Jump to: navigation, search

Team Alta Mira Milestone 1 submission.

Experience the life of a Worker on Mechanical Turk

This is our experience of being a worker on Mechanical Turk (MTurk), Microworker, CrowdFlower and Ipoll.

MTurk requires approval to start working and earning money, we were not able to get approval as we joined the project on March 3rd and it takes a minimum of 48hrs. We were able to complete tasks on Microworker, CrowdFlower and Ipoll.


MTurk

The start up process on MTurk was much more difficult than with other platforms. This platform requires workers to submit their SSN and they take the longest time to approve/respond. They have the notion of qualifications which a person must obtain to earn money for a task. Obtaining qualifications also requires having your account approved. While the obstacles to get started are greater than with any other platform, they also have the highest number of job listings. I felt the price workers get paid on MTurk was very low compared to polling sites. Some of the tasks require a Master qualification, which requires you to take a quiz and pass it but only pay a few cents. The work also has to be approved by Requester to earn money. The type of jobs we found on MTurk were of many different categories from finding information, guaging sentiment to many other varieties. They also post the work to be done directly on the site so rarely do you have to go and register for some service that asks you for personal details.

Microworker

This was a relatively frictionless start and I was able to start working right away. The work is different from MTurk, this involved mostly going to third party sites, taking surveys and completeing forms. There was also work where I had to find information about something and enter it e.g. look up what a business does. I thought the payout rate here was a bit higher than MTurk. There were fewer HITs but I found payout rates around $0.10 vs $0.03 on average. I found the link to Ipoll here as well, it was one of the task listings and it led me to iPoll where I had to create an account and take surveys. I was able to earn money by entering data that proved that I completed (generally some token that appears at the end).

CrowdFlower

CrowdFlower's system was better than MTurk but still required more time. They had different portals for workers and requesters and it was pretty confusing as the design elements were different on the 2 sites. On the worker's side, it was difficult to even tell if I was still with CrowdFlower or some third party site. Their system for working on HITs was incremental. You have to earn your way to the top by completing extremely inexpensive tasks and earn your way to the more expensive ones which sometimes require successful completion of 100 tasks. I thought this system was the most successful at driving me away as I had to complete a lot of site information suveys for $.01 before I could proceed to anything more worthwhile. This system made it notriously hard to earn substantial money and I had to abandon it. Doing the surveys themselves required doing a Google search, typing in some text string, checking where it was on the page and recording this data, and there was a right and wrong answer which made it difficult to earn any money. Needless to say I abandoned this platform very quickly.

IPoll

This was the most successful platform I was able to use. I was able to register quickly and I started taking surveys right away and was able to earn a $1.50 within a few minutes. They also gave me $5 just for getting started. The tasks were interesting, I had to see ads and talk about how they related to me and what I took away from it. The system was glitchfree and worked perfectly. I was impressed with this system compared to all others.


Takeaways

MTurk was the hardest system to get started with but they also had the greatest variety of tasks. CrowdFlower's incremental system was the worst and their right/wrong answer system led me to abandon it very quickly. Microworker and Ipoll were ther most successful and I was able to earn money with them. The tasks were interesting and well defined and the survey's on Ipoll actually used my data to serve me appropriate data. Regardless of the platform, I felt it was quite difficult for a worker to make substantial amount of money from the platform. Working 8 hrs a day and earning $1-2 per hour (optimisitic by measurements above) is still not enough to sustain oneself. As a beginner, I was not able to find profitable and engaging tasks and finding legitimate work is difficult. A lot of time is wasted finding the "right answer" on CloudFlower and you feel like you are at the mercy of the Requester in most cases.

Experience the life of a Requester on Mechanical Turk

We tried a few different platforms as requesters.

MTurk

I created 2 sample tasks, a sentiment analysis task and a survey to guage people's preferences. Amazon automatically adds "Master's" qualification for tasks and charges slightly more money. I was very surprised to receive this email from a worker (presumably):


Hi there, I’m not sure if you are aware, but your task is set to require Masters qualification (Amazon puts it on by default).

By using Masters, you: 1) Severely limit the number of workers who can access your task; 2) Pay a 30% premium to Amazon for using it, and; 3) Miss out on a lot of great workers (such as myself, with 250,000+ approved HITs and a 99.9% approval rate – but no Master’s qual!)

You could save money and get the same results by setting your qualifications to “Total Approved HITs is not less than 5000” and “HIT Approval Rate is not less than 98%”. With these quals, you will attract serious, dedicated workers, avoid scammers, and not have to pay the 30% fee to Amazon.

Regards,

William


This shows the MTurk community is pretty large and people are paying attention to the task postings. Both the tasks were completed in a few hours, with fairly good responses. I felt it was much easier to be a requester on MTurk, the process was seamless. I had to set up a Payments account but as soon as I transfered the amount, it was very easy to get up and running. There was no delay in getting started. Posting a HIT was also quite easy and getting responses was quick. The posting interface was organized very well with sample repsonses. The only issue I found was that at one point I had to type in some HTML, otherwise it was pleasant. I could also transfer micro sums of money (any amount I chose and there was no minimum, which was nice). Amazon also provides analysis on the response data if it fits into a provided format so there is no need to analyze CSVs. An interesting thing I found was that people were bad guages of sentiment eg: "This was undoubtedly one of the best wastes of money ever invented", a majority of the responses were positive while the sentiment here is clearly negative.

Microworker

Creating a campaign on Microworker was a bit harder. I had to set up my payment account, make a minimum payment of $10 and wait for approval. The format of HITs is different, it is a requirement and expected reponse format where you can define what is expected from the worker. I was able to set up a campaign to measure page load speeds on a website and received responses fairly quickly from 30 individuals. My account was activated within a few hours of entering payment information and making a payment. This was also a hassle free process but I felt the quality of responses on MTurk was higher.

These are our attached results from our HITs and Campaigns (Microworker) File:Batch 1845203 batch results.csv File:Batch 1844652 batch results.csv File:Microworkers ca6834a54ba4.csv File:Batch 1844715 batch results.csv

Explore alternative crowd-labor markets

Explored Freelancer, Scriptlance and Taskrabbit. Freelancer and Scriptlance are used primarily for programming/design work while Taskrabbit is used for all kinds of tasks.

Freelancer is a huge marketplace for programming/design and data entry tasks. There were quite a few tasks that were around building websites, redoing apps, generating sample designs etc. You have to purchase (yes, purchase but you are given some free) bids and bid on projects for which you might or might not get selected and your bid gets used up. As an experienced programmer, I felt there were ridiculously low prices for the work that was being asked. I felt it would be difficult to bid such low amounts for projects that require weeks to complete and it would be difficult for a new entrant to get started and become competitive. Scriptlance is similar but the interface is not as smooth and fast as Freelancer. I did have to pay to get started with being a Requester. Hiring someone on Freelancer also required posting a job requirement and waiting for people to bid. The bids trickled in pretty quickly and eventually there was someone willing to do design work for little money but the quality difference in their portfolio was obvious, many did not even have portfolios. Their profile and history system helped a lot with picking the right people.

Taskrabbit was probably the most modern of the three with a fast validation, intuitive UI and quick responses. I didn't even have to sign up to see who was availble for help with moving.

- The most important difference between this and MTurk was the type of jobs available. - Freelancer and Scriptlance were mostly programming and design related. TaskRabbit was for household work, I found moving, fixing things, carrying luggage, running errands. - The difference was also in the interface, MTurk was cluttered and looked complicated to get started where as these sites were much clearer and easier to get started. - There was no startup cost with MTurk but with Freelancer, I had to purchase bids. - The price of a task was significantly different, a person could make a living from working on Task Rabbit living in the US, I felt it would be difficult with MTurk.

Readings

MobileWorks

  • What do you like about the system / what are its strengths?

- Opens the micro worker ecosystem to those who don't have desktop access. - Mobile is the entry point into internet for many low income people in India, this is a great way to encourage them to make money from it. - The UI is simple enough that anyone can get started with it.

  • What do you think can be improved about the system?

- There is no notion of making more money by getting better, the only way to make more money is by doing more tasks, which doesn't give workers motivation to get better. - For a slum worker, there needs to be some guarantee of employment, this doesn't provide that. Once the tasks run out, so does the income, this is too risky for most people who, if they were labourers, could easily find other jobs. - The study assumes the average wage to be much lower than what people actually need to survive in India and doesn't take into account the different levels on income inside slums like Dharavi. There are different hierarchies of income inside slums and this wouldn't appeal to anyone but people at the lowest levels who may or may not have access to a cellphone.

mClerk

  • What do you like about the system / what are its strengths?

- Add images to txtEagle (now Jana mobile focusing on mobile apps) and empower people without accesss to smartphone/computers - A unique problem - digitization of local language text - Uses SMS protocol to send images, which is very interesting, responses transliterated in English

  • What do you think can be improved about the system?

- A SMS picture message uses 3 SMS messages, are users ok with something like this - This helps recharge their mobile balance, will it ever make them enough to be sustainable - There are several mentions of workers waiting at the bus stop and sending texts or doing it in class which alludes to them being students, it is unclear if a person can do this full time and survive with the amount of money made from it.

Since the article is 3 years old, it would also be interesting to know why this system has not grown as significantly as they predicted in India where they initially had viral growth.

Flash Teams

  • What do you like about the system / what are its strengths?

- Lego block style modular combining on teams - Foundry - flash teams, split tasks into micro tasks, crowdsource teams/experts, machine understandable sets of tasks - Auto generation of teams / path search - UI is well thought out - it remembers previous tasks, interfaces with oDesk, provides an easy way to drag/drop organize tasks and intelligently adapts to changes.

  • What do you think can be improved about the system?

- For someone who in unaware of how it works, I feel the concept/product has a learning curve. A project manager who decides to use it would need to understand it and relate it back to thier development practices (agile, waterfall), if there was an easy way to get started with it, it might make the transition easier. The product also requires trusting it to some degree with automated generation, this might seem unclear to people who are using it without fully understanding it.