Menu Search
Jump to the content X X
SmashingConf London Avatar

We use ad-blockers as well, you know. We gotta keep those servers running though. Did you know that we publish useful books and run friendly conferences — crafted for pros like yourself? E.g. our upcoming SmashingConf London, dedicated to all things web performance.

Improving Your Website Usability Tests

In one of the first usability tests I ever did, I met a lovely old lady who could not use a mouse. She kept lifting it in the air and pointing at the screen, speaking words of encouragement to the cursor. At the end of the test I got absolutely nothing, but she did think I was a “lovely boy” who should meet her granddaughter. Very quickly I learned the value of setting very clear criteria for participant recruitment.

If you’ve ever run a usability test before, you’ll know that it’s not as easy as it looks. Although it’s not rocket science, there are some intricacies that can make a big difference. In this article I share some of the lessons I have learned which should help you avoid your user test turning into a frustrating experience for you or the test participant.

Further Reading on SmashingMag:

That first year of my career was the most valuable experience I could have had and while I believe that learning through your own mistakes is the best way to learn, we do not always have the luxury to do so. Here are some tips I have learned along the way which should help you quickly improve your usability testing skills and avoid some pitfalls.

Design Your Usability Test Script To Answer Specific Research Questions Link

When starting a new usability test, don’t assume that all you need to do is pick out the main areas of the website and ask users to complete those tasks. You may well find some useful insights with this approach but don’t be surprised if when you present back your findings you get bombarded with questions from the project stakeholders that you cannot answer.

Key Takeaway Link

Talk to the people you’ll report back to and ask them what key questions they need the research to answer for them. If you end up with lots of questions, prioritize them and then work out a way to answer them as best as you can. If the question seems too vague or you’re unsure why they are asking it, get clarification. The more you understand the reasons behind the questions, the better equipped you are at answering them by adapting your tasks and questions mid-test.

Give Participants The Confidence To Behave Naturally Link

When participants turn up to a test, they’re usually not sure what to expect. They’re probably a bit nervous with a camera in their face and someone looking over their shoulder. Don’t be surprised if they look to you for guidance at the beginning. If you’re too controlling at the start of the test, you’ll reinforce to them that they need to get permission from you before they do anything.

Give participants the confidence to behave naturally.
Give participants the confidence to behave naturally.

Key Takeaway Link

Encourage users to show you their natural behavior by starting off your test with a broad task to allow them to go off and explore in any direction they like. I use pre-test questions to uncover a real problem they face within the context of the test and then I let them off the leash to answer it as naturally as possible. For example, I wanted to test an online property law website, so in the first task I asked people to search for a house in the area in which they would like to buy, within a specific budget range. This allowed us to get a realistic view of how they use the web, while also setting the context for the next tasks in the test.

Leave Room For User Freedom To Complete The Task In Their Way Link

In the early days, I used to set out a task in the test script and as soon as users started to stray off the task I’d reign them in and ask them to try again. Not only was I extremely controlling, sometimes losing rapport with participants, I also denied myself the chance to learn something I hadn’t already expected to find.

Key Takeaway Link

Always leave room for users to freely roam around the website and go off track a little before bringing them back to the purpose of the task. You may feel you are losing control or that the participant has misunderstood the task, but try to resist temptation for a while longer because it can be fascinating to see where they go and why. Often you’ll uncover some real gems here, so try your best to let it happen. If you need to pull them back on track you can, but not until you’re sure they won’t find their own way back.

Relax, Shut Up And See What Unfolds Link

It’s easy to be rigid and controlling and only focus on what you need users to do for you. When they do something interesting or unexpected, it is extremely useful to ask them what they are thinking. But do this too early or too often and, like I mentioned above, you could miss out on observing natural behavior.

Let the users’ thought processes flow. (Image by Drew Clemens5.)

Key Takeaway Link

Try your best not to interrupt the flow of the participant’s thought process. The more you interrupt, the less likely they are to have the confidence to complete tasks unaided. If you’re asking them something every 30 seconds, they will keep losing their flow and you won’t see anything like natural behavior. You can always bring them back and ask them what happened afterwards. I see a lot of people new to usability testing make this mistake, so I remind them that it is impossible to ask questions and observe what users are doing at the same time.

Tailor The Tasks To The Participant In Front Of You Link

I’ve mentioned being too rigid a lot so far. I think its because when you do something new, you like to control the variables and lock down the unknown. But with experience, you learn to release control as you get more confident that you’ll handle anything that comes up.

In my early days I liked to write out the exact scenario I was going to give to users to set the scene for a task. But I soon learned that users simply do not engage as much when I set tasks that do not match with what they would normally do. I remember a time when I asked a 19-year-old guy to imagine he was a mother of three in order to complete a task. Needless to say, he looked at me strangely and didn’t really engage with the task at all before he gave up and said he couldn’t find it.

Key Takeaway Link

Set the overall task you want users to complete but try to be generic and then tailor the scenario to the participant. While this isn’t always possible, there is huge value in spending a little time at the beginning of the test to learn about who the participant is and their current use of similar products or services. If you can then use this to build a test scenario that fits with a real problem or scenario they would like to solve, you can learn so much more than when someone simply “pretends” to be in a scenario.

Always Include Tasks On Peer Or Competitor Websites Link

Spending a whole hour on a single website can be boring for you and your participant. But boredom isn’t the only problem. All your findings and observations are based on an isolated case. You have no real understanding of whether that person always goes to the search box, or whether they just did it on your website because they were confused by the navigation options. Just looking at one website doesn’t give you a realistic picture of how people use the web.

Key Takeaway Link

Make the time in your test plan to look at competitor or peer websites as part of your test. The best way to do this is to ask participants at the start of the test what websites they currently use and ask them to show you. Then you can introduce a competitor or peer website they haven’t used before. You’ll find that you learn much more about their patterns of behavior and why they choose one website over another, and more importantly you’ll learn what works well on other websites and why. This is a great source of inspiration when you need to solve a tricky usability problem on your website.

Don’t Let Them Know Which Website You Are Testing Straight Away Link

I have made the mistake in the past of making it obvious what website I am testing. Sometimes this is difficult to avoid, but if you can, I would advise it. The main reason is that it can be very hard for anyone to be completely honest about their experience with a website when you work for the company either as an employee or as an agent.

Key Takeaway Link

If I haven’t been involved in the design of the website prior to the test, I always emphasize my independence. Another trick is to get participants to look at competitor websites and give you honest feedback on them before visiting the actual website you are testing. If you can do this without them knowing which website you are really testing, you have a much higher chance of them offering their honest initial thoughts. Towards the end of the test it is likely to be obvious as you’ve spent most of your time setting tasks on one website, but by then you should have been able to get a good understanding of their honest first impressions.

In Summary Link

If you want to improve your usability testing technique, there is no substitute for doing more tests. However, as I’ve highlighted here, you can try to be aware of how the design of your tasks and how you interact with the participant can affect the outcome of your research. Designing your test to focus on key research questions and not being too rigid with your tasks is a good starting point. In addition, using competitors as part of the test and encouraging users to behave as naturally as possible can yield better results.

If you want to learn more about how to plan, design and moderate usability tests, I have listed some highly recommended books below:

Finally, if you have any questions please post them in the comments below. I’ll do my best to answer them there, or if warranted, I’ll expand on them further in a future article.


Footnotes Link

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8

↑ Back to top Tweet itShare on Facebook

Damian is a Director at Experience Solutions a usability testing agency supporting leading UK brands. Damian has 13 years experience as a UI design specialist for companies like BBC and National Air Traffic Services researching & designing websites, apps, voice recognition, and air traffic control interfaces. Damian regularly writes for the Experience Solutions Blog and tweets for @experiencesolns

  1. 1

    Very good article.
    I guess you can apply all this not only for website but for any interface you want to test (mobile, desktop, etc). All these concepts are more generic than “website” IMHO.

  2. 4

    Andreas Ibsen

    January 8, 2013 6:16 am

    There is a newer 2nd edition of Handbook of Usability Testing.

    How do you collect and analyze data? I would consider it harder to compare individual tests if you build a test subject specific scenario each time? I know it’s not quantitative testing, yet I think some parameters for comparison are needed.

    • 5

      Hey Andreas

      Yes you’re right, this method is very qualitative and we use it when we want to observe natural behaviour to uncover the real reasons why the website is not performing as well as expected. There are occasions where we will want to be a bit more quantitative to compare against a benchmark and we use a slightly more rigid process for this. It is possible to mix those tasks with the more explorative tasks in the same test design.


  3. 7

    Damian, Thank you for sharing your insight with us! I am just settling into my first job, and I’m working on my first project that’s going to require usability testing. I’ve known for years that usability testing is essential to the success of any website, especially one that is application-heavy (as my current project is). I am thrilled to have your knowledge, guidance, and experience at my fingertips as we move forward and begin to set up tests and tasks for our client. I will definitely be referencing this time and again moving forward. Thank you again!

    • 8

      Great to hear it is useful. If you have any more questions feel free to contact me directly

  4. 9

    Steve Krug’s “Don’t Make Me Think” is also a great reference on usability tests. My copy is dog eared and full of post it notes from all the people I have loaned it out to.

    • 10

      Norbert Sienkiewicz

      January 8, 2013 1:27 pm

      My first thought seeing the title of this article was a reminder of Dont make me think. A must have for everyone !

  5. 11

    SM has done it again! Perfect timing!

    This week I have been working with my web dev team to establish a Usability Testing script for a new website we are launching soon.

    We have the volunteers arriving next week! This article will be very useful to us. Thank you for taking the time to write and share it Damian!

  6. 13

    Great article Damian, thanks for sharing. I’d also add the need for clearly defined success metrics, because this can be such a subjective area. It could be timings, error rates or basic criteria for success (e.g. a target CTA or page) as a few examples. You could argue that someone who takes 5 minutes to successfully complete a task is not realistic and in a natural context of use they may well hit their frustration tolerance and bounce, and that because they are in a test situation that they will spend more effort to complete the task than they would do naturally.

    The other thing to add was the ‘Dr House’ school of thought when it comes to user test responses – that what they say and what they do are often at odds with one another. Validating what someone says in the opening interview with observed behaviours can bring tremendous insight and reveal misconceptions.

    I love the ‘shut up and see what unfolds’ insight. Even after many years of user testing I find myself having to bite my tongue and not ‘lead the witness’. Allowing the user to struggle is as important, nay more important, than watching the user succeed. Whilst it’s sometimes hard (and patience wearing), it’s such a valuable practice that brings massive benefits.

    • 14

      Hey Nick

      Yes we do see users struggle for an hour using a website only to tell us how easy it was to use in the post-test interview. But that’s a whole different article.

      Shutting up and seeing what unfolds is the hardest thing to teach as well. But definitely one of the most valuable things you can do if you want to see natural behaviour.


  7. 15

    Great advice Damian! My favorite (by far) is your advice on not letting them know which website you’re testing straight away. I’ve goofed this up a few times and always regretted it. One time I invited someone to our office and then was surprised how they know what company we were with. D’oh…common sense fail.

    • 16

      :) Yep, we’ve had a struggle with that when having to test in our client’s offices. Sometimes it is unavoidable but can be extremely useful in getting honest and natural responses from users.

  8. 17

    Thanks for your post Damian.

    As Nick Fine says it is very important to establish the success metrics and remembering the a usability test is specially important to get quantative data. Don’t trust what they say, trust what they do. You have better methods as focus groups or individual interviews for users to tell you what their problems are.

    It is important to mention that a good report must contain the impact of the findings plus proposed solutions. The biggest added value it is if you can meet the dev team to agree on possible solutions and give estimates to fix/improve the artifact. I have seen too many reports in the bin because the project managers did not know what to do with the findings afterwards. With estimates – time/money – PM’s can better understand what a user test is made for (and thus more likely to call you again).

    Important! select users based on the target groups. If you users will be mothers of three, then making a test with teenagers will give you completely wrong results. You actually did something wrong before starting the tests. Selecting appropriate users might be time consuming and even cost you money but it is the key.

    Finally, and related to my previous point I disagree with your suggestion to tailor the scenarios to a user. If you do proper target groups and the screening is correct, then the scenario should fit perfectly. Tailor a scenario has the risk to lead two users on the same target group to two different conceptions giving you a wrong insight. Tailor a scenario also gives the unfair responsibility to the tester to be creative 5 minutes before doing the scenario which might lead to a poor scenario. A user test has a lot of research behind, and although J. Nielsen helped us to bring back user tests to real working environment it is still a scientific method.

    And for all newcomers, reading the books that Damian mentioned should give you a good background to do proper user tests. Good luck to all!

    • 18

      Thanks for your comments David. I guess that’s what makes the area of User Experience so interesting. We all have different methods to get to the same end goal. All I can say is that over the years these strategies have formed and now serve me very well.

      Thanks for adding some more thoughts to the article


  9. 19

    Nick Gassmann

    January 8, 2013 5:04 pm

    Holy smokes. Did you and I both run our first usability test on the same woman? I got the same from one of the ladies in my first usability group (couldn’t use a mouse and asked me to meet her daughter, not grand. :D)

    • 20

      Maybe she was doing the rounds at the time Nick :) I’m quite glad I managed to laugh off her (quite serious) comments about meeting her grand daughter otherwise that could have been a very awkward blind date!

  10. 21

    wow! amazingly written.., “like” this

  11. 22

    Hester Kerpel

    January 10, 2013 4:30 am

    Thanks for this great article. Although most of these pointers aren’t new to me, I must admit I forgot all of them when I conducted my first usability test. It’s apparently very natural to be really annoyed by some one using any website other then you would do, and really curious about the why of this. Which, in my case, resulted in a series of too early asking what they were thinking, and questions which made sure they knew they were doing something “wrong” (questions like “Why would you do THAT?!”).

    I think usability testing is one of those things where practise makes perfect. But then, it’s never really perfect, because you’re dealing with real people who sometimes don’t know how to use a mouse.

  12. 23

    A very useful article. This whole website is amazing !!

  13. 24

    Great article Damian, but how would you tweak this if your product is a paid product hidden behind a login wall? You can only test with current users and they would know the website. I’m facing this problem now!

  14. 25

    Great article. I have conducted some interviews with some of the above suggestions. It’s a great way to build user tester confidence and partnership for the app you are trying to build. Thanks for sharing.

  15. 26

    Hi Damian,

    I just want echo others’ sentiments here and say that this is an excellent article. There are tips in here that I will definitely implementing in some upcoming usability test I am running.

    I was wondering, from your experience, how long you felt was an appropriate length of time for a usability test?


↑ Back to top