How Usability Testing Drastically Improved My Client’s App

Advertisement

Most designers spend too much time with their designs to be objective about them. The best thing any designer can do is to collect feedback from real users. Testing uncovers pain points and flaws in a design that are not otherwise obvious.

Recently, I had an opportunity to experience this firsthand when iterating on HelloSign, the iOS app that enables users to scan, sign and send documents from their phone using the built-in camera. Thanks to testing, the app went from four stars to a solid five stars after a redesign. We’ll look at how the app started, how we ran the tests and how the product ended up with five stars.

The Original Design

This app has four primary sections: authentication, welcome, document creation and document editing. The biggest changes we made to the app were on the authentication and welcome screens. We’ll first briefly review the original designs of these screens, as well as the document creation and editing screens, to understand how the app works.

Authentication and Welcome

The authentication and welcome screens are important moments in the user’s initial experience of the product, because we want to move the user from signing up or — if they already have an account on the HelloSign website1 — signing in to creating documents. The app was designed to complement the website, with the understanding that users would already be somewhat familiar with the product and would likely have a user name. Had this been designed as a standalone app, authentication would have been a secondary option, rather than a requirement.

2
The original versions of the sign-in, sign-up and welcome screens.

Document Creation

The document creation process consists, in essence, of a camera, with guides to position the document in the frame. In designing this process, we looked to the camera screens of the iPhone’s native Camera app and of Instagram, as well as to the framing markers found in products such as Schwab’s app for depositing checks and Card.io’s app for scanning credit cards.

Document Creation3
The final document creation screens.

Document Editing

After creating a document, the user is presented with the editing screen. Here, they can modify the document by adding signatures, text, checkboxes and date stamps.

Document Editor4
The final document editing screens.

User Testing

A few months after the initial version landed in the App Store, I was given an opportunity to iterate on the app. Before beginning, I decided to run a user test to better understand what in the app was effective and what could be improved.

While several user-testing services are available, such as Verify5 and uTest6, I decided on UserTesting.com7 because of the quality and value that you get for the relatively inexpensive price. Different testing services offer different benefits, features and options; review them8 for yourself, and select the one that best fits your goals and needs. With UserTesting.com, you purchase a package of tests, and users will record videos as they go through whatever task you’ve assigned to them. You may also ask several follow-up questions of each user, such as, “What was the most frustrating part of your experience?” Conveniently, UserTesting.com provides four very strong default questions that will suit most tests.

As explained by Jakob Nielsen9, testing interfaces on many users is not necessary in order to identify issues; even three to five users might suffice. Because this app is for on-the-fly document signing, we ran a test that took users through a typical use case: creating, editing and sending a document. Follow-up questions were based on UserTesting.com’s suggestions and queried users on ease of use, areas of difficulty and areas for improvement. You could ask many different questions10. Once the tests were completed, after just a couple of hours, I could immediately identify several issues.

I reviewed each video twice. The first pass was merely to identify any glaring issues and to familiarize myself with the tester’s recording style. On the second pass, I paid closer attention, noting specific problems. During testing, most users won’t articulate problems they’re having, but the problems will be fairly obvious from their behavior. A problem is obvious when the user does any of the following:

  • pauses for a few seconds when trying to complete a task,
  • stumbles and has to backtrack in their steps or has to undo an action,
  • expresses audible frustration (a sigh or grumble),
  • takes a longer route to achieve a goal than expected,
  • fails entirely at a task.

While there are certainly many others, these are some of the most common indicators of a problem worth delving into. The more tests you run (with small groups), the more problems you will identify. Running a test after each new design iteration is highly valuable, if you can afford to do so. Some problems might affect only a small percentage of your user base, while others might affect the vast majority. Spend time prioritizing the problems that you identify according to your existing list of features, goals and user requests. Not every problem is worth resolving.

The Redesign

The goal of HelloSign’s redesign was to identify and fix any major problems for users. As mentioned, one can take forever to resolve every last issue; budget limitations kept our scope small.

What Did We Learn?

The tests revealed a major problem with the authentication screens. It seemed to be a stumbling block because most users weren’t clear about whether they were in the “sign up” or “sign in” state. Users often tapped back and forth between authentication states before understanding which screen was for signing up and which was for signing in.

We also discovered the following:

  • Creating pages was too repetitive a process.
    After taking each photo, the user had to tap “Add page,” take another photo and then repeat. This was tedious, and some testers remarked that the process felt unnecessarily repetitive, while others expressed audible frustration.
  • Users could not edit a date after adding one.
    One tester wanted to add a past date to a document. While a “date” object could be added to the document, it showed only the current date (i.e. of creation) and could not be edited. This was confusing and unnecessarily restrictive.
  • Markers for aligning the document during scanning needed refinement.
    Users had trouble lining up the document with the boundaries on the camera screen, with some expressing frustration and with many giving up.

Fixing Authentication

Our most substantial changes were to the authentication process, which was changed almost entirely. Dumping the user right into the sign-up screen, with only a “Sign up” button as a state indicator, proved to be confusing. During testing, most users seemed to expect this screen to let them sign in, not register. The revised screen added a step, forcing the user to explicitly select “Sign up” or “Sign in,” making it a conscious decision. An alternative solution could have been to add distinct labeling above the user name and password fields, because users typically read from top to bottom, although I was afraid that wouldn’t entirely resolve the issue.

Original vs. New Start Screen11
The original (left) and revised (right) welcome screens.

After selecting one of the two options, users were brought to one of two nearly identical screens, the difference being only in the labelling of the button, “Sign up” or “Sign in”. As you can see in the revision, the layout was simplified and Google authentication was added. Despite both screens being the same, forcing the user to choose a path cleared up any confusion.

Original vs. New Sign In Screen12
The original (left) and revised (right) sign-in screens.

Lastly, the home screen was heavily revised. While certainly clear before, it de-emphasized “Help” far too much and generally felt clunky and heavy-handed. The revision brought “Help” to the forefront and highlighted the “Scan” action, scanning being the primary purpose of the app.

Original versus final home screen13
The original (left) and revised (right) home screens.

The Takeaway

Design is a highly iterative process, and all of the intuition in the world won’t help you to identify gaps in your product. As designers, we’re just too familiar with our own work to be able to easily spot where it fails. The only way to truly improve a design is to test it on real users14 and watch how they interact with it. Testing with a live app uncovered problems that led us to turn a four-star effort into a five-star product, with only a little work.

There are many ways to test a app, at an array of price points. It could be as simple as sitting down with a few friends and having each of them use your app for a few minutes, or as complex as hiring a moderator to bring in a variety of users to your office. There is also A/B testing15, which can — and probably should — be done in conjunction with user testing. While user testing is great for big updates and for identifying major problems, A/B testing, which is less costly, is great for continually testing new ideas and underlying assumptions.

Remote services are inexpensive, and they structure the tests for you and free you from having to hunt down users. No matter how tight your budget or how simple the app, testing your design on real users is always worthwhile and will help you better understand where the product can be improved.

(al, ea)

Footnotes

  1. 1 http://hellosign.com
  2. 2 http://www.smashingmagazine.com/wp-content/uploads/2013/11/10-final-opt.png
  3. 3 http://www.smashingmagazine.com/wp-content/uploads/2013/11/13-creation-opt.png
  4. 4 http://www.smashingmagazine.com/wp-content/uploads/2013/11/14-editor-opt.png
  5. 5 http://verifyapp.com/
  6. 6 http://www.utest.com/
  7. 7 http://usertesting.com
  8. 8 http://www.smashingmagazine.com/2011/10/20/comprehensive-review-usability-user-experience-testing-tools/
  9. 9 http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/
  10. 10 http://www.smashingmagazine.com/2009/06/29/45-incredibly-useful-web-design-checklists-and-questionnaires/
  11. 11 http://www.smashingmagazine.com/wp-content/uploads/2013/11/19-final-start-opt.png
  12. 12 http://www.smashingmagazine.com/wp-content/uploads/2013/11/20-final-signin-opt.png
  13. 13 http://www.smashingmagazine.com/wp-content/uploads/2013/11/21-final-home-opt.png
  14. 14 http://blog.allyis.com/the-value-of-user-testing/
  15. 15 http://www.wired.com/wiredenterprise/2013/11/leanplum/

↑ Back to topShare on Twitter

Josh is principal at Planetary, a design and development agency based out of Brooklyn, New York. He is currently running a project called OneHour.me, providing product strategy consulting for startups by the hour.

Advertising
  1. 1

    Good points. IMHO the brand name too has a “sign” word in it and that was probably adding more confusion to the current problem. How about placing the brand name at extreme top but then the body remains empty? Needs more work.

    -1
  2. 2

    - 33%+ used for branding certainly leaves room for improvement – especially in terms of usability.
    - the revised “home screen” – despite making the emphasis clearer – makes all buttons signigicantly smaller for no good reason.
    - what’s the story with the inconsisten colouring of certain actions? sometimes yellow, sometimes blue for what should be a variant of the “next step”

    1
  3. 3

    I get what the app does after having read it and looking at the revised home screen where you highlighted the “Scan” button my first thought was: “How does a phone scan something?” Scan is what I use a scanner for so rather than the eyeball above it why not put your tiny camera icon above the word scan. This lets me see that I will be using a camera to scan with, not my eye or some other thing I couldn’t imagine.

    2
  4. 4

    The originals look better, sorry. I’m tired of people throwing “usability testing” around like they have a clue what it means.

    Please explain why you added an extra press for logins, instead of merging them all onto the first screen with the choice to sign up as the fork.

    Oh, crappy usability testing is the blame?

    screen 1, 2 text boxes, 3 submits
    a. enter username + password
    a2. sign in button
    b. auth with google
    c. sign up

    screen 2:
    a. YOU ARE LOGGED IN NOW
    b. google account verify
    c. signup page

    screen 3:
    b. YOU ARE LOGGED IN NOW
    c. signup verify / authentication

    The “cancel” button is meh, help reinforce the standard of “swiping” to go “back” or exit. This bit of reinforcement cleans up all of the rest of the interface needing cancel / back buttons.

    3D Drop shadow tabs as buttons also meh. Definitely out of iOS7′s style, so disorienting at best.

    At some point “usability testing” and “focus grouping” is more about commercial success than teaching users how to be better, more intelligent users. Pandering.

    0
    • 5

      Heh.

      Good luck up there in your ivory tower.

      Anyone having the time to ‘teach users how to be better more intelligent users’ is either in possession of a ton of their own or someone else’s cash, or not working to the same deadlines as everyone else on this rock

      0
    • 6

      I kind of agree, this post is flawed in areas, and as for user testing its hard to get an exact replica of your intended user so the feedback is usually so..so

      0
    • 7

      Still think, I mean, believe you know what your talking about? You couldn’t be MORE wrong, UX redesign after usability testing was proven to be far better and shows in their current success and regards for the mobile app. But what do I know, I am just a Creative Director :)

      0
  5. 8

    “Not every problem is worth resolving.”
    I couldn’t agree more.

    0
  6. 9

    Really helpful article, thanks a lot for sharing!

    0
  7. 10

    This was a cool, interesting case study. Ignore the user testing haters. Generally they are agency “creatives” or one hit wonder entrepreneurs.

    0
  8. 11

    Great article – really wanted to share it on Twitter and finally found the small link at the bottom (my eye automatically looks for the bird – usually at the top of an article). Just some usability feedback to take into account for future Smashing article design…

    0
  9. 12

    Great Read ! User Testing – This is what it is all about.

    If designers are thinking of structured, in-house test cases, think again. Nothing so very nice and comfy anymore. It’s a lot more legwork and a lot less air conditioning to test mobile app user experience these days! Taking the app out for a ride to real life users is what counts !!

    0
  10. 13

    very helpful

    0
  11. 14

    Nice article. Would be better why color changes made explained.

    0
  12. 15

    Really great article yo! Thanks for sharing. I am gonna share it on Twitter.

    I am a junior web dev with designing background. Feel free to visit my portfolio at http://fat-cow.net, give me some comment please!

    0
  13. 16

    I think there is still a big problem. User won’t realize that he can use an Google account for log in if he didn’t click on “I have an account ” button.

    0
  14. 17

    Great read – thanks. Just one thing. As one of the readers above mentioned, the use of the word “Scan” actually isn’t that good. The app isn’t actually scanning anything, it’s taking a photo.

    It might sound like nitpicking, but when you think about it, it can also add to confusion.

    0
  15. 18

    You made a point of saying that Help was not emphasised enough. I still haven’t seen a user attempt to access the help during usability testing. I also worked on an application used for over 10 years by around 1000 institutions and hundreds of users in each institution…I took sample data from over 100 and the help was accessed in single digits and I think at least a couple of those were miss clicks. I am curious at what kind of help access rate you actually experience during testing?

    0

Leave a Comment

Yay! You've decided to leave a comment. That's fantastic! Please keep in mind that comments are moderated and rel="nofollow" is in use. So, please do not use a spammy keyword or a domain as your name, or else it will be deleted. Let's have a personal and meaningful conversation instead. Thanks for dropping by!

↑ Back to top