Milestone 1 Team Innovation

From crowdresearch
Jump to: navigation, search

Kristy: I live in Canada and this quote is not a lie or exaggeration. In my experience, life as a worker is a must. I don’t Turk because I want to, I Turk because it feeds my family. It keeps a roof over my head and it pays my tuition. Unlike most on this project, this week I Turked because I have nothing if I don’t. When I Turked full-time, I made a great living. Since going back to school in September 2013, I don't have as much time to work, but when I do, I still do well.

141227.gif

This was the last month I Turked full-time. Since then I have mostly been working for Requesters off-mTurk, helping them post HITs, run surveys, analyse data, etc. I try to do some HITs every day to keep in touch with the platform, though. This week I made my dollar, but that's it as it was midterms! :)

150305.gif

Will: I work on Mechanical Turk as a supplemental income source. In today’s harsh economic climate, it can be tough to make ends meet. Mechanical Turk helps me to provide for my family, whether it be household cleaning supplies, Christmas presents or home appliances. Every little bit helps.

Andrea: I have been an mturker for 10 months now. I feel that this platform has its advantages and its disadvantages.

As a worker

What I like:

  • the flexibility of the work. I do not want quotas on time or work submitted. I want to only work when I feel like working.
  • I can do as much work as I want and make as much as I want. There is no restriction on how much I can do or make, or how many hours I spend on mTurk
  • I can pick and choose between what work is available and do what I feel comfortable with as far as pay and content
  • Amazon runs it, so the pay always comes through
  • Requesters are debited up front so I can rely on payment
  • there are forums, scripts, etc. made by workers to help other workers
  • there are lots of Requesters and lots of work in comparison to other platforms, work never runs out
  • I can tell people I work for Amazon ;)
  • I can trust Amazon, they make me feel safe and secure
  • It’s the next wave in work, so I feel like I’m on the cutting edge
  • I have the flexibility to choose what work I want to do. What kind of work matches my interests or skill set.
  • Building relationships with requesters and influencing the workflow, HIT design or compensation
  • Building relationships with other worker through online message forums

What I don’t like:

  • I can’t move up in the company
  • it’s hard to get started on the platform
  • it’s hard to make money at first
  • it’s intimidating to work here at first due to the uphill battle of growing stats, etc.
  • I can’t put these skills on my resume
  • there isn’t always work I’m willing to do
  • work is often immoral or harmful
  • work often underpays for tasks that require some skill
  • there isn’t much skilled work which pays more
  • it is hard to communicate with Amazon
  • Amazon doesn’t mediate disputes (blocks, rejections, etc.)
  • there is no official forum for talking to Amazon and/or Requesters
  • communicating with Requesters is hard
  • Requesters don’t get all email we send them
  • there is no way for workers to talk to each other
  • there is no information on how many workers there are, where they’re from, who they are, etc.
  • demographic isn’t retained to make it easier to qualify for surveys
  • tax paperwork isn’t sent unless I make a ton of money
  • it is hard to determine how much I actually made in the year before
  • the dashboard offers very little - no info on bonuses (you can’t see who bonused you, or for what you received bonuses, or for how much), no statistics, no goal setting, can’t see the HITs you’ve submitted
  • you can’t change a HIT you’ve submitted to fix it (should at least be able to send a note to the Requester)
  • Requesters who don’t feel like paying can take the work and reject the HITs (so they pay nothing) without recourse; there is no Requester accountability
  • there is no Requester rating system
  • the search functionality is lacking, doesn’t allow you to whitelist/blacklist Requesters or keywords, etc.
  • the GUI is pathetic, there is no interaction, no customization, no efficiency
  • when your account is suspended, it’s impossible to deal with CS to get it back
  • Masters is handed out randomly
  • international users must shop at the company store / there aren’t enough payment methods available
  • you have to use Amazon Payments, needs alternative payment methods
  • we had to come up with Dynamo to fight the exploitation we face, and yet we have no way to tell workers about it
  • we have to fight to be a part of the discourse around mTurk
  • we have to spend so much unpaid time searching for work, there should be a system that suggests work to us
  • we have to come up with our own scripts, we should be able to suggest the functionality we need and have it added to the platform
  • there is no customization on mturk, so you can’t have a better experience
  • we have to have Turk Alert and other scripts to alert us to new work by our favourite Requesters, keywords, etc., there should be alerts from the platform
  • there is no app for that
  • we have to have Turkopticon to rate Requesters
  • the forums are all marginalized from one another (not sure if anything could be done about that)
  • Turkers aren’t a part of the system; could be focus groups, getting a job at the platform, community liasons, customer service, forum moderators, client services for Requesters who offer help with design or advice or pricing, etc. but they should inform policy and changes
  • there is no reward for being a long-time worker
  • there is no reward for being a high-quality worker
  • it lacks a system for policing bad HITs/Requesters
  • rejections should drop off your statistics after a while, we all make mistakes
  • there is no training system for basic HIT categories
  • there is nothing to work towards, like special qualifications (like badges)
  • we don’t know what’s going on at mTurk
  • we are left out of mTurk conferences, articles, etc.
  • verifying my account can be difficult
  • Requester accounts aren’t verified
  • Requesters can easily put up TOS-breaking HITs; should have to deposit $100 to start and forfeit it if they continually break the rules
  • it seems like no one is advertising/promoting mTurk
  • nothing is done to give Turkers a better reputation; seen as “third world”, unskilled, etc.
  • there is no affiliate program
  • there is no data made public about workers (could be used to show we’re educated, etc.)
  • the system/documentation isn’t translated for users in other places
  • Amazon does not invest in system upgrades
  • Platform is lacking functionality
  • If HIT expires, there is no record to see what we abandoned
  • Cannot search our completed HITs. This is critical when a requester requires that we not repeat a HIT
  • Cannot search for our qualifications assigned to us
  • Long time and accurate workers do not get the masters qualification
  • Amazon provides no criteria for workers to earn the masters qualification
  • No protection against workers who scam the system using scripts or bots
  • No way to settle disputes with requesters - they can just ignore your emails
  • No dispute arbitration
  • Customer services is severely lacking. All enquires are met with boilerplate responses, regardless of the content of the query. If the response is not satisfactory/appropriate, there is no mechanism to reply. No opportunity for communication. This is extremely frustrating.
  • Support team does not investigate requesters in a timely manner

As a Requester

I like:

  • I can throw work up on the platform and ignore it and still get good results
  • I don’t have to communicate with workers if I don’t feel like it
  • it is easy to pay participants
  • the platform is always available and there are always workers around
  • there are thousands of workers available
  • the workers are predominantly college educated
  • the workers are at least as representative as WEIRD undergrads
  • Amazon is reliable, so the platform won’t disappear overnight
  • my IRB doesn’t think Turkers are human, so I don’t need approval to use mTurk
  • I don’t have to pay fair or reasonable wages if I don’t want to
  • I can have a large volume of tasks completed simultaneously by hundreds of workers
  • I can use a qualification system to curate my own trained and qualified workforce
  • I have many coding options using the API
  • I can have a wide variety of different tasks completed
  • I can build custom projects on my own website and use ExternalHit to direct workers to it

I don’t like:

  • workers don’t have their demographic information stored so I can easily use qualifications for my work/studies
  • there aren’t many international workers
  • international Requesters like me can’t sign up
  • the API is a mess, hard to use, documentation is lacking
  • there aren’t many tools to make using the system easier
  • there aren’t many tools for statistics, visualizing data, etc.
  • there is no testing for workers, so we don’t know about language proficiency, etc. which would make qualifications easier
  • the GUI templates force you to use Masters, or qualification levels that suck (10k HITs isn’t that much)
  • there aren’t enough payment methods available
  • data must be in CSV format for upload and download
  • there isn’t easy integration with survey tools
  • the GUI has few tools, no visualizations, few templates, makes posting HITs hard
  • there is no rating system for workers beyond HIT statistics
  • workers aren’t held accountable for scamming, spamming, cheating, etc.
  • Masters isn’t based on real tests of skill, language, etc.
  • there are so few basic qualifications
  • people look down on my data because I used mTurk
  • there is no app
  • there is no clear TOS that tells me exactly what sort of HITs will be pulled
  • Amazon runs TOS-breaking HITs all the time, but mine which do the same thing get pulled; there should be a system where you can get permission for asking for email addresses, downloads, etc.
  • workers aren’t truly anonymous
  • workers should have training available
  • qualification tests are hard to make
  • autogranting qualifications is hard to do
  • it is hard to make it so people who have done my work before can’t do it again
  • the calculator stating how much people are making per hour is broken
  • there is no way to pay per hour, I must pay per HIT; there should be a way to pay only for the amount of time taken
  • the documentation is difficult to use, not comprehensive
  • there is no way for me to talk to workers about best practices, design, pay, etc.
  • I have to pay for help from Amazon, even with basic things
  • verifying my account is difficult
  • worker accounts aren’t verified immediately (by phone, send in ID, etc.) so there is no proof they are who they say they are
  • there is no way to port it to intranets/replicate it for niche work/etc.
  • there is no affiliate program
  • there is no data available on workers (for research)
  • not accessible to those with disabilities (CAPTCHA not audio, etc.)
  • Comparators are not consistent across programming languages
  • GUI is lacking in functionality (Can’t send messages to workers - really?)
  • Masters is a default setting. You’ve over-charged me and limited my pool of potential workers.
  • Categorization and Sentiment templates don’t allow me to set location qualifications
  • Categorization and Sentiment templates don’t allow me to reject shoddy work
  • Built-in qualifications should include basic demographic information - ie Male or female
  • No way to match/identify workers of a specific skill set to my task
  • Can’t bonus worker without an Assignment ID. I would like to send standing bonuses to my best workers.
  • Functionality is not consistent across GUI/CLT/API
  • I have no idea how many workers are available overall, at any given time, or within the demographics I need

Alternative platform: LeadGenius

I like:

  • hourly pay
  • I can move up in the company (training, orientation, etc.)
  • the interface is modern, easy to use
  • lots of training and documentation
  • there is lots of support

I don’t like:

  • I have to meet quotas, be on a certain number of hours
  • hourly pay as I am making as much as slower people
  • it seems like a giant glorified research project where the guys at the top get all the praise
  • it’s only open to a single niche product
  • the work itself isn’t for me
  • the training is excessive, takes forever
  • the signup process is excessive, takes forever
  • Payment by paypal

Alternative Platforms: Taskrabbit/Crowdtask

I don’t like:

  • They only use Paypal to pay you, so you lose some in fees, plus Paypal can be hacked so I don’t like linking it to my bank account; need multiple payment methods

Interesting comment from peer review: Regarding the TaskRabbit platform, I would add two main differences compared to AMT: the requestor can select the worker he wants to perform the task and the worker gets paid regardless the satisfaction of the requestor. The requestor can 'complain' by rating the worker on the system, however the worker is more protected compared to AMT.

Alternative platform: Clickworker

I like:

  • that tasks are posted in large volume
  • Open to Canadians
  • Payment by PayPal

I don’t like:

  • there is rarely work available
  • Account is de-activated if you don’t work for a month - which is tough when there is rarely work available
  • Task are underpaid
  • payment by Paypal (as above)

Alternative platform: Microworkers

I like:

  • Open to multiple countries

I don’t like:

  • Tasks are ALL scammy - buying votes, buying likes, clicking affiliate links, SEO manipulation

MobileWorks Paper

I like:

  • that it’s trying to include all sorts of workers, even those with little technology
  • they try to match offline wages
  • they care about workers’ input on the project
  • the flexibility of working from anywhere

I don’t like:

  • It’s exclusive to a specific niche of workers, so I can’t access other workers, or I can’t work there (it's only for people in x country, or with mobile phones, etc.)
  • I don’t buy the accuracy rate

Interesting comment from a peer reviewer: I would add two main weaknesses: the system doesn’t use the quality rating of the workers for quality assurance (only for the payment) and the fact that only limited types of tasks that can be performed due to the small size of a mobile.

mClerk Paper

I like:

  • that it’s trying to include all sorts of workers, even those with little technology
  • the flexibility of working from anywhere

I don’t like:

  • It’s exclusive to a specific niche of workers, so I can’t access other workers, or I can’t work there
  • must know some English, be a fast typist, have tech, etc.
  • payment in mobile airtime
  • majority rules is easy to game (two cheaters team up to get paid)
  • “competitively priced”, not intended as a replacement for full-time job = exploiting workers; we work all day long, why should we then spend our leisure time working for tiny wages? The “free” market has not done us any favours offline, it will not do us any online either
  • TURKERS DO NOT EARN $2/HR, if you offer crack cocaine in return for answering a survey you can’t be shocked when only drug addicts take that survey; Ross underpaid for his survey, so he only got workers willing to accept underpaid work

Foundry/Flash Teams Paper

I like:

  • dynamically assembles teams
  • expert teams, can complete complex jobs including those needed different kinds of experts
  • somewhat managed by the computer, not me the Requester
  • total flexibility for the Requester
  • hiring is elastic, output can be pipelined
  • workers get to communicate with each other, the Requester

I don’t like:

  • there may not be work for me to do at any given time
  • I have to be an expert to participate