Pumas usability evaluation findings

From crowdresearch
Revision as of 21:29, 14 July 2015 by Saiphsavage (Talk | contribs) (Methodology)

Jump to: navigation, search

Pumas Team: Usability Evaluation Findings


- Claudia Flores Saviaga

- William Dai

- Saiph Savage


We recruited 6 participants both offline and online to avoid having only highly active requestors. To recruit online, we sent direct messages to requestors on social media who had recently made posts on Twitter related to Amazon Mechanical Turk, UpWork, or Freelancer. To recruit offline, we approached people in out proximity who we knew were requestors at least once (these were mainly lab mates and friends) We briefly interviewed participants to understand their practices as requestors, particularly how long they had been requestors and type of tasks they conducted.

We created an account for participants, logged them in, and asked them to use the platform to carryout their normal requestors tasks. While they conducted the tasks they identified a series of problems with the platform. The users were asked to evaluate based on the ten steps of Nielsen's Heuristic Evaluation[1]

We present below some problems encountered for each interface, and ways of overcoming the problem. We also categorized  problems based on the usability heuristic the problem relates to and offer ways of solving the problems.


Specific Problems in Interfaces Task Design Workflow

1.-Create and Account
3.-Project Category
4.-Project Details
6.-Design Task
9.-Payment (user returned to that section)
9.-Summary (user returned to that section)

General Notes: The user had trouble in the part of task design because he didn't know what the properties panel was for or how to use it. He also had problems in the payment section because the user could put anything in there, which caused errors reflected in the summary section. Therefore, the user had to press the back button to correct that, and even when he erased the service fee part, he got errors at the end. The user also expressed confusion in the Milestones section. The user didn't understand what was for the option where the use of a webpage was allowed.

User2 Crop 10 Images to Passport Photo Size/General Evaluation

create project choose category page
feedback between project category and project details
project details
search bar function test
some textbox feedback 
design page...and feedback on button
Broke it? $Error!
any text is accepted?
error just won't go away
submit page
final test! finding the project

General Notes: This user had never done anything related to crowdsourcing and thought that additional text instructions would be very useful. He found the same error regarding payment as User1, and was very confused by the design page. Additional problems he found are in the images.

Aesthetic and Minimalistic Design

User 1: "I like the design and the fact that the amount of data to fill was acceptable"

Consistency and Standards

User 1: "I do not have problem with this"

Error Prevention

User 1: "The same as user control and freedom, in the summary i got an error on the payment that i wrote, and i didn't know why."

Flexibility and Efficiency of Use

User 1: "Some parts where confusing like task Design section"

Help and Documentation

User 1: "There were parts that where confusing and no help, or examples were available"

Problem: Error message for password, has weird message (what does password1 mean?): "Field password1 must be at least 8 characters long." Solution: Add to form information of the type of password that is expected. Correct error message to remove "password1"

In Skills: User had no way of adding new skills after one particular skill had been added.

Helps Users Recognize, Diagnose and Recover from Errors

User 1: "There was no solution suggested for the errors that i got and I didn't know how to fix them."

Match between System and World

User 1: "Some concepts are not clear to me as i stated in the feedback on the screenshots"

  • The payment field is too big for what is being asked. reduce size to show it is of a number.

Same for how much you will pay for each task. Also it was unclear what "task" meant. Was it the milestones? Was it the entire work? What is Service Fee?

  • In the whole create project the sections hardly are highlighted, and it is unclear in which section we

are in. There are no icons on the task description. In milestones, this sentence is super confusing: "Most people need five different inputs to gather feedback on their prototype tasks

  • Amount of time field again very long. It should be a small fuield as it is just a number. Confusing to have such long


  • In Task Design: Why do I Have to put the project if I am puting my own website?
  • Why does my projects show projects that are not even mine.
  • What is Task Feed Tab? Very confusing

Recognition Rather than Recall

User 1: -NA-

User Control and Freedom

User 1: "The part of the payment i didn't understand. I was being able to change the service fee without warning"

  • Design task in create project forces users to just use drag and drop.

Solution: enable html text editing.

  • Visibility of system status

User considered this screen to be very confusing. He was unable to drag anything, He did not understand what he had to do. Although his task was simple (translation), the user felt the interface was very confusing.

Visibility of System Status

User 1: "I know which part of the task design workflow i am at each step thanks to the progress bar at the top."