Pumas usability evaluation findings

From crowdresearch
Jump to: navigation, search

Pumas Team: Usability Evaluation Findings

Contributors:

- Claudia Flores Saviaga

- William Dai

- Saiph Savage

- Nisha K. K.

Methodology

We recruited 11 participants both offline and online to avoid having only highly active requestors. To recruit online, we sent direct messages to requestors on social media who had recently made posts on Twitter related to Amazon Mechanical Turk, UpWork, or Freelancer. To recruit offline, we approached people in out proximity who we knew were requestors at least once (these were mainly lab mates and friends) We briefly interviewed participants to understand their practices as requestors, particularly how long they had been requestors and type of tasks they conducted.

We created an account for participants, logged them in, and asked them to use the platform to carryout their normal requestors tasks. While they conducted the tasks they identified a series of problems with the platform. The users were asked to evaluate based on the ten steps of Nielsen's Heuristic Evaluation[1] We present below some problems encountered for each interface, and ways of overcoming the problem. We also categorized problems based on the usability heuristic the problem relates to and offer ways of solving the problems.

Results

Specific Problems in Interfaces

User1 Create a Task for Webpage Design

Task Design Workflow

1.-Create and Account
2.-Login
3.-Project Category
4.-Project Details
5.-Milestones
6.-Design Task
7.-Payment
8.-Summary
9.-Payment (user returned to that section)
9.-Summary (user returned to that section)

General Notes: The user had trouble in the part of task design because he didn't know what the properties panel was for or how to use it. He also had problems in the payment section because the user could put anything in there, which caused errors reflected in the summary section. Therefore, the user had to press the back button to correct that, and even when he erased the service fee part, he got errors at the end. The user also expressed confusion in the Milestones section. The user didn't understand what was for the option where the use of a webpage was allowed.

User2 Crop 10 Images to Passport Photo Size/General Evaluation

create project choose category page
feedback between project category and project details
project details
search bar function test
some textbox feedback 
milestones
design page...and feedback on button
Broke it? $Error!
any text is accepted?
error just won't go away
submit page
final test! finding the project

General Notes: This user had never done anything related to crowdsourcing and thought that additional text instructions would be very useful. He found the same error regarding payment as User1, and was very confused by the design page. Additional problems he found are in the images.

User 3 Please see if there are coding flaws

Sign Up Error Message
Sign Up Password Field Error Message
Error Message for Duplicate Signup
Too Much Javascript?
Looks Like This On Mobile 
Good Server Verification

General Notes: This user is not a crowdsourcing expert, but specializes in mobile application development, software engineering, and cyber security.


Code Improvement Suggestions

User 3: "Quick note looking at the request sent when I have a backslash '\' in my password that gets escaped to \\ that could cause problems... also if you're doing password logic properly, you shouldn't have to escape passwords like that... it should be hashed which is irrelevant of characters entered"

Solution: Hash passwords?

User 3: [see [Good Server Verification] image above] "Props for verifying your info on the server ☻ Instead of relying on Angular for it haha. So just a quick note since you're passing the password params over JSON I guess escaping the backslashes is fine. since the hash and the original password are going to have \\s in them. So you can disregard that Aside from that, everything seems pretty solid as far as I can tell o.o"

Solution: YAY, ignore the earlier code improvement message ☺

User 3: [not so much a coding error, but lack thereof] [mobile platform support is needed] [see [Looks Like This On Mobile] image above] "Phones are going to have a really tough time logging in. also, Google said that they are going to drop the rankings of sites that do not have mobile support so I'd really make that a priority."

Solution: Have the Research-Engineering Team create sub-teams of mobile application developers, for both iOS and Android...don't want Google to drop our site from top hit searches...and fix the HTML screen configuration to fit the screen of whatever device is being used. Since some people are going to use mobile web browsers instead of downloading applications.

Aesthetic and Minimalistic Design

User 1: "I like the design and the fact that the amount of data to fill was acceptable"

User 3: "[see [Sign Up Error Message] image above] I'd fix that. Small display bug. Also differing behavior. The rest just highlight the field red. But if you don't enter a password, it shows that error message. So take a look at that.

[see [Sign Up Password Field Error Message] image above] Again with the verbiage. I'd rather word it as "Passwords must match" or something to that effect rather than specifying the field names."

Looks like the site is still very work-in-progress. One other thing I would say for good UX is on the login screen. No need to put "This field required" for username and password especially in red. It's already implied and it looks like right off the bat I've done something wrong lol.

Also, for the title on that I'd leave it as just "Sign In" unless you have multiple services and are specifying that this is Crowd Research otherwise it's just redundant. Plus the inconsistency between CrowdResearch as the page title and on the top left and then Crowd Research (two words) in the sign in. But that's rather nit-picky. I can tell this still is very work in progress."

User 3: [minimalistic...] [see [Too Much Javascript] image above] "Holy Javascript batman No way you need that much for a simple login page that is going to slow down your site a LOT if you're loading that much content on EVERY page [I mention that we’re using angular, which should speed things up] at LEAST half of that is not Angular and also on top of that Angular is a HUGE framework there is no need to include ALL of it on EVERY page"

Solution: :( Looks like some serious code simplification/streamlining is in order...

User 3: [mobile platform support is needed][it's really crowded and ugly...aesthetics?] [see [Looks Like This On Mobile] image above] "Phones are going to have a really tough time logging in. also, Google said that they are going to drop the rankings of sites that do not have mobile support so I'd really make that a priority."

Solution: Have the Research-Engineering Team create sub-teams of mobile application developers, for both iOS and Android...don't want Google to drop our site from top hit searches...and fix the HTML screen configuration to fit the screen of whatever device is being used. Since some people are going to use mobile web browsers instead of downloading applications.

Consistency and Standards

User 1: "I do not have problem with this"

User 3: "[see [Sign Up Error Message] image above] I'd fix that. Small display bug. Also differing behavior. The rest just highlight the field red. But if you don't enter a password, it shows that error message. So take a look at that.

[see [Sign Up Password Field Error Message] image above] Again with the verbiage. I'd rather word it as "Passwords must match" or something to that effect rather than specifying the field names."

Solution: Make the error messages consistent. Also make them appear only when the user has done something wrong. Trivial but annoying.

User 3: "Plus the inconsistency between CrowdResearch as the page title and on the top left and then Crowd Research (two words) in the sign in. But that's rather nit-picky. I can tell this still is very work in progress."

Solution: Well that logo isn't the official one yet...

Error Prevention

User 1: "The same as user control and freedom, in the summary i got an error on the payment that i wrote, and i didn't know why."

Solution: Textbox should have a "$" sign placed in front of the textbox to make it clear that the value entered is in $ currency. Also it will help to show that the user need not enter the currency sign. Just the numerical value is enough.

Some programs allow the programmer to specify which type of input will be accepted.

Flexibility and Efficiency of Use

User 1: "Some parts where confusing like task Design section"

Solution: Many users found this section confusing and did not understand the purpose. A little description is required to for this section to help users understand. Even a small example can be given to explain this.

User 3: [mobile unfriendly. mobile platform support is needed] [see [Looks Like This On Mobile] image above] "Phones are going to have a really tough time logging in. also, Google said that they are going to drop the rankings of sites that do not have mobile support so I'd really make that a priority."

Solution: Have the Research-Engineering Team create sub-teams of mobile application developers, for both iOS and Android...don't want Google to drop our site from top hit searches...and fix the HTML screen configuration to fit the screen of whatever device is being used. Since some people are going to use mobile web browsers instead of downloading applications.

Help and Documentation

User 1: "There were parts that where confusing and no help, or examples were available"

Problem: Error message for password, has weird message (what does password1 mean?): "Field password1 must be at least 8 characters long." Solution: Add to form information of the type of password that is expected. Correct error message to remove "password1"

In Skills: User had no way of adding new skills after one particular skill had been added.

Solution: While creating account, the interface need to mention the type of characters required in the password while creating the password.

The user must be able to add as many skills as he wants in the Skills section. This feature can be enabled by putting a "+" sign which on clicking lets the user add a new skill to his profile.

As mentioned earlier, there should also be a little description for the task design section.

User 3: "[see Error Message for Duplicate Signup image above] Not a very helpful error message. Need to instead say "An account with this email has already been registered" or something to that effect. Because technically, it is unique but they are already registered.

Solution: Change the error text.

User 3: "[see [Sign Up Error Message] image above] I'd fix that. Small display bug. Also differing behavior. The rest just highlight the field red. But if you don't enter a password, it shows that error message. So take a look at that.

[see [Sign Up Password Field Error Message] image above] Again with the verbiage. I'd rather word it as "Passwords must match" or something to that effect rather than specifying the field names."

Solution: Make the error messages appear only when an error has occurred.

Helps Users Recognize, Diagnose and Recover from Errors

User 1: "There was no solution suggested for the errors that i got and I didn't know how to fix them."

Solution: Most of the users faced errors while creating password and during payment section. As mentioned earlier, these sections need to be made more clear.

Match between System and World

User 1: "Some concepts are not clear to me as i stated in the feedback on the screenshots"

  • The payment field is too big for what is being asked. reduce size to show it is of a number.

Same for how much you will pay for each task. Also it was unclear what "task" meant. Was it the milestones? Was it the entire work? What is Service Fee?

  • In the whole create project the sections hardly are highlighted, and it is unclear in which section we

are in. There are no icons on the task description. In milestones, this sentence is super confusing: "Most people need five different inputs to gather feedback on their prototype tasks

  • Amount of time field again very long. It should be a small fuield as it is just a number. Confusing to have such long

text.

  • In Task Design: Why do I Have to put the project if I am puting my own website?
  • Why does my projects show projects that are not even mine.
  • What is Task Feed Tab? Very confusing

Solution: The length of the payment field must be reduced. Also the service fee is not clear to a few users. There must be some help provided by giving the description as to what it really means. One way is to put "?" sign next to Service Fee field. Clicking on that "?" will open a pop up explaining Service Fee.

While creating project, the stage at which the user is currently, for example the user has reached the payment stage, then that stage must be visibly highlighted.

The milestone creation description needs to be made more clear and precise.

Recognition Rather than Recall

User 1: -NA-

User Control and Freedom

User 1: "The part of the payment i didn't understand. I was being able to change the service fee without warning"

  • Design task in create project forces users to just use drag and drop.

Solution: enable html text editing.

  • Visibility of system status

User considered this screen to be very confusing. He was unable to drag anything, He did not understand what he had to do. Although his task was simple (translation), the user felt the interface was very confusing.

Visibility of System Status

User 1: "I know which part of the task design workflow i am at each step thanks to the progress bar at the top."

Summarising the Design Solutions

1. Payment Section: Many users got error in this page. The kind of value to be entered in the field is not clear. Need to mention the currency. Can also provide a placeholder in the textfield as an example. The service fee concept is not clear with the users. Need to provide a little description for that as well. This can be done by placing a "?" sign next to the Service Fee label. Clicking on the "?" will open a pop up showing the description of the service fee.

2. Task Design Page: This is another very confusing page for the users. Many users don't understand the need for this and just skip the section. A little description is required to make it clear.

3. Clear use of terms: Few places are confusing due to the terms used. For eg., time required to finish the task - which task: milestone or the complete project?, or amount to be paid for which task?

4. Formatting required: Few general formattings are required as requested by the users. The length of the payment textbox and the amount of time required textbox needs to be reduced. Highlight the stage at which the user is currently.

5. Few terms need description and help, like task feed. For the first time users or if the task feed is empty, that time instead of showing it empty rather show what it is supposed to have, like an example or a brief description.

Original Task Design Proposal

This is how the task interface as originally coded by Team Pumas per @arichmondfuller 's design powerpoint looked. Our users favor this over the existing working prototype interface.

Categories
Create a Task
Prototype Task - One Task
Prototype Task - Multiple Tasks 
Review
Payment