Pumas usability evaluation findings

From crowdresearch
Revision as of 21:23, 14 July 2015 by Saiphsavage (Talk | contribs) (Help and Documentation)

Jump to: navigation, search

Pumas Team: Usability Evaluation Findings


- Claudia Flores Saviaga

- William Dai

- Saiph Savage

- Team Member (add name)

- Team Member (add name)

Here we present the results of our usability evaluations. We asked X users to use the task design section of the Daemo platform. The users were asked to evaluate based on the ten steps of Nielsen's Heuristic Evaluation[1]:


We recruited 6 participants both offline and online to avoid having only highly active requestors. To recruit online, we sent direct messages to requestors on social media who had recently made posts on Twitter related to Amazon Mechanical Turk, UpWork, or Freelancer. To recruit offline, we approached people in out proximity who we knew were requestors at least once (these were mainly lab mates and friends) We briefly interviewed participants to understand their practices as requestors, particularly how long they had been requestors and type of tasks they conducted.

We created an account for participants, logged them in, and asked them to use the platform to carryout their normal requestors tasks. While they conducted the tasks they identified a series of problems with the platform. We present below some problems encountered for each interface, and ways of overcoming the problem. We also categorized problems based on the usability heuristic the problem relates to and offer ways of solving the problems.


Specific Problems in Interfaces Task Design Workflow

1.-Create and Account
3.-Project Category
4.-Project Details
6.-Design Task
9.-Payment (user returned to that section)
9.-Summary (user returned to that section)

General Notes: The user had trouble in the part of task design because he didn't know what the properties panel was for or how to use it. He also had problems in the payment section because the user could put anything in there, which caused errors reflected in the summary section. Therefore, the user had to press the back button to correct that, and even when he erased the service fee part, he got errors at the end. The user also expressed confusion in the Milestones section. The user didn't understand what was for the option where the use of a webpage was allowed.

User2 Crop 10 Images to Passport Photo Size/General Evaluation

create project choose category page
feedback between project category and project details
project details
search bar function test
some textbox feedback 
design page...and feedback on button
Broke it? $Error!
any text is accepted?
error just won't go away
submit page
final test! finding the project

General Notes: This user had never done anything related to crowdsourcing and thought that additional text instructions would be very useful. He found the same error regarding payment as User1, and was very confused by the design page. Additional problems he found are in the images.

Aesthetic and Minimalistic Design

User 1: "I like the design and the fact that the amount of data to fill was acceptable"

Consistency and Standards

User 1: "I do not have problem with this"

Error Prevention

User 1: "The same as user control and freedom, in the summary i got an error on the payment that i wrote, and i didn't know why."

Flexibility and Efficiency of Use

User 1: "Some parts where confusing like task Design section"

Help and Documentation

User 1: "There were parts that where confusing and no help, or examples were available"

Problem: Error message for password, has weird message (what does password1 mean?): "Field password1 must be at least 8 characters long." Solution: Add to form information of the type of password that is expected. Correct error message to remove "password1"

In Skills: User had no way of adding new skills after one particular skill had been added.

Helps Users Recognize, Diagnose and Recover from Errors

User 1: "There was no solution suggested for the errors that i got and I didn't know how to fix them."

Match between System and World

User 1: "Some concepts are not clear to me as i stated in the feedback on the screenshots"

Recognition Rather than Recall

User 1: -NA-

User Control and Freedom

User 1: "The part of the payment i didn't understand. I was being able to change the service fee without warning"

Visibility of System Status

User 1: "I know which part of the task design workflow i am at each step thanks to the progress bar at the top."