WinterMilestone 1 westcoastsfcr

From crowdresearch
Jump to: navigation, search

Experience the life of a Worker on Mechanical Turk

Reflect on your experience as a worker on Mechanical Turk. What did you like? What did you dislike? If you're from outside the USA and unable to access MTurk, try try the worker sandbox, or other sites like CrowdFlower or Microworkers or Clickworker.

What did you like?

  • There was such a broad amount of hits to choose from
  • We can have specialized search for the type of HITs we would like - for example - time needed etc
  • You could search for the amount you wanted for a hit
  • You can search for HITs you're qualified for
  • HITs need to be approved before they are accepted - this helps to improve quality of results for the requester.
  • You're given a reason why your HIT was rejected
  • The time limit placed for a HIT - this helps me to work diligently for the HIT and not do it at leisure after I accept a HIT.
  • Some HITs had opportunities to earn bonuses.

What did you dislike?

  • The amount of higher paying hits was way less than lower paying hits
  • Hits pend and you have to wait for payment
  • Requesters can decide whether or not your work suffices
  • You should receive payment regardless. It's a risk the requester is taking by using this service.
  • The only rating system was based on what a requester thought of your work
  • Interface/website design looks depressing/grimy. Almost as if it was made for lower class workers (similar to craigslist).
  • The submit button for a HIT is not standardized across all hits. (There may be one within the HIT and another one outside the HIT part of the regular Mechanical Turk interface); It would be nice if there was a standard interface for all the HITs
  • Pending status for HIT: it would be nice if there was a deadline of some sort or something that keeps the worker when their HITs are likely to approved
  • When you log in, the landing page defaults to all hits rather than ones you are able to do
  • When doing the same HIT multiple times, you had accept the hit each time; default accept might be better
  • The HIT is displayed when you click on the "View a HIT in this group"; It's misleading because you have to accept the HIT first or the HIT may change, but you can already start working on it in the window - maybe Accept Hit should be more prominent on the page (or eliminated, and the HIT will be counted as "Accepted" whenever work is submitted)
  • The HIT listings are very long and horizontal-based. It would be easier to read if they were more compact.
  • Instead of having pagination, it would be nice if it was an infinite scroll, so the worker doesn't have to keep changing pages
  • The amount paid by requester is very less and its very hard to earn money. Fair payment needed. Some standard needed to assess the worth of a HIT.
  • Certain HITs involve large amount of work e.g., writing the text of audio clips - but are paid less. Need to have a breakdown or a max size of HIT.
  • Some HITs lacked sufficient directions describing what was required for the task.

Experience the life of a Requester on Mechanical Turk

Reflect on your experience as a requester on Mechanical Turk.

What did you like?

  • They provide a filled out template when you create the listing.
  • Easy for creating listings that have the same format
  • They provide examples and a tutorial for the CSV creation

What did you dislike?

  • The layout for the page where you select the type of project to work on makes it hard to notice that they are several types of projects you can pick from on the left-side. I thought they were sequential steps of the creation of a project.
  • Creating the layout of the listing: Not sure how to do certain things from a GUI perspective, i.e. add text boxes for input
  • Creating the layout of the listing: To effectively edit the template, knowledge of HTML is required
  • When editing the template, you have to constant switch back and forth from the source code to the preview
  • Have to go through upload process of CSV file multiple times every time you want to change something before you publish
  • Would like a statement of confirmation saying my hits have been published
  • Don't particularly like creating a separate file (CSV) on a different program for the HIT
  • The interface doesn't make it clear that there have been people who have submitted for your HIT recently
  • There are some workers who don't care about the quality and just want to do the minimum amount of work possible; they didn't even read the instructions
  • It was not clear how to expand the interface with the results of your HITs
  • I felt bad rejecting all the work someone did, so I ended approve one mediocre HIT
  • When downloading the results, I wasn't given the option to only download approved results
  • It's a pain approve previously rejected results

Attach the CSV file generated when you download the HIT results.

Explore alternative crowd-labor markets

Compare and contrast the crowd-labor market you just explored (TaskRabbit/oDesk/GalaxyZoo) to Mechanical Turk.



What do you like about the system / what are its strengths?

  • It's extremely accessible to the majority of the population
  • Has an influence on the economy
  • It's minimalistic
  • It paid well
  • Is accessible to those of lower socioeconomic classes from making money
  • Worker compensation is based on the quality of the work that they’ve produced, which provides an incentive directly related to why they are doing this in the first place.
  • The multiple entry systems seem to be very good at ensuring quality.
  • The rating or score scheme for the worker is very useful for both the worker and the requester. It might help the requester make his approval decisions faster or easier based on the worker's rating.
  • The multiple entry system also provides incentive for answering to the best of your ability, since depending on how your answer compares with the second party's, your quality score will increase or decrease.
  • The workers can see how much they earn in real-time.

What do you think can be improved about the system?

  • Being a bit more strict on what users can participate based on their accuracy
  • Could increase the size of micro tasks
  • Could expand to be on both mobile and desktop in order to allow for larger tasks to be completed
  • Network connectivity through mobile devices can be slow and unreliable, especially through the web browser. Perhaps a native mobile app could be offered as well (although I do see issues with updates and incompatibility, and support)
  • The rate at which the application is refreshed for new jobs might be high, so interface support that makes it easy to do that, if not already supported, might be helpful.
  • extend to OCR tasks in different languages since India has a vast diversity of languages.
  • add language specific OCR tasks based on the profile of the worker. The worker might mention his preferences in his profile.


What do you like about the system / what are its strengths?

  • Boomerang
  • Prototype tasks and task feedback
  • The interface looks modern
  • improvement system in which recent scores are given heavier weighting, and newer workers are given some leeway

What do you think can be improved about the system?

  • Perhaps implement team tasks
  • Guarantee pay for worker (work can't be rejected)
  • If possible, implement some sort of critique system so workers can know how to improve their work
  • Communication system between the workers and requesters in order to better clarify tasks
  • People might have already gotten use to inflated results from other systems (i.e. Uber), which leads to them believing any rating lower than a 4/5 is a low quality worker
  • You could prevent the requester from proceeding before providing the worker a rating

Flash Teams

What do you like about the system / what are its strengths?

  • Solution for a more complex crowdsourcing problem: more expert, skilled work
  • Having so many people working on it provides a diverse set of opinions and viewpoints, possibly making the product even better
  • Communication between the workers
  • Foundry was very useful

What do you think can be improved about the system?

  • How cohesive would the end product be?
  • All of the work is remote, so people don’t get to bond with a lot of their workers; as they mentioned, they’re missing camaraderie, maybe some kind of Skype hang out to get to know each other
  • If the work isn't finished on time and the requester refuses to pay for that reason the requester shouldn't be given access to the work that has been created (It didn't specify if this was a feature or not, but thought it should be mentioned)

My MTurk (half) Workday

What do you like about the system / what are its strengths?

  • Easy access multiple jobs at once
  • When it was the reuester's fault he still paid him

What do you think can be improved about the system?

  • The way wages were extremely low
  • Poor instruction
  • Directions weren't clear

Milestone Contributors

Slack usernames of all who helped create this wiki page submission: @dmajeti, @alexstolzoff, @clao, @tgotfrid