Menu Search
Jump to the content X X
SmashingConf London Avatar

We use ad-blockers as well, you know. We gotta keep those servers running though. Did you know that we publish useful books and run friendly conferences — crafted for pros like yourself? E.g. our upcoming SmashingConf London, dedicated to all things web performance.

How A/B Testing And Analytics Can Make You A Better Designer

Metrics can be a touchy subject in design. When I say things like, “Designers should embrace A/B testing” or “Metrics can improve design,” I often hear concerns.

Many designers tell me they feel that metrics displace creativity or create a paint-by-numbers scenario. They don’t want their training and intuition to be overruled by what a chart says a link color should be.

These are valid concerns, if your company thinks it can replace design with metrics. But if you use them correctly, metrics can vastly improve design and make you an even better designer. First, when I talk about metrics, I’m talking about making use of a couple of very specific tools: User analytics. A/B or multivariate testing.

Further Reading on SmashingMag: Link

User analytics are what you might get from Google Analytics5, KISSmetrics6 or Mixpanel7. They tell you things like which pages users have viewed the most, which call-to-action buttons they’ve clicked, and how many tasks they’ve performed while using your product. They can also show you where people drop out of critical flows, such as registration and purchasing.

KISSmetrics Shopping Cart Infographic8
Shopping Cart Abandonment Infographic by KISSmetrics9

A/B and multivariate testing involve looking at how changes affect key metrics such as revenue and retention. When you run an A/B test on a change, you’re comparing user behavior with the new design to user behavior with the old design in order to answer the rather important question of “Which design caused the user to do more of what I wanted and less of what I didn’t want?”

By incorporating analytics and A/B testing into their process, designers can not only improve the business outcomes of their redesign projects, but also become better at design.

Metrics Tell You Where Design Is Needed Most Link

Unless your resources are unlimited, prioritizing design changes can be tricky. For example, if you’re working on an e-commerce website, who’s to say whether optimizing the sign-up flow is more important than revamping the check-out flow or rebuilding a product page or simplifying the navigation? Any of these could be pretty good ideas.

Metrics are fabulous for making these decisions. For example, analytics can show exactly how much of a drop-off your shopping cart is getting. It can show you precisely how many visitors aren’t converting into signed-in users. It can pinpoint how many people bounce right off of product pages.

Metrics tell you where your biggest business problems really are, so that you can use design to fix them.

Of course, metrics can’t tell you how to actually fix the problems. Your job as a designer is to learn why users might be having problems in certain areas of your product and come up with brilliant solutions.

How Should You Use This? Link

When deciding on your highest design priority, examine your product’s analytics to identify where your biggest problems are.

Metrics Help You Track Real User Behavior Link

How many times have you been designing and thought something like, “I wonder how many products people have in their shopping carts on average when they drop out?” Or, “What is the average number of friends a user has in such-and-such a social network?”

Facebook friends10
(Image: Adriano Gasparri11)

Answers to questions like these can have an enormous impact on the way you design. After all, depending on whether users have no more than a few friends or they tend to have more than a hundred friends, the interaction for selecting from a list can be quite different.

Metrics and analytics free the designer from having to guess the answers to questions like these.

Again, knowing the answers doesn’t tell you how to design the product, but it does give you a lot more insight into the real problems you’re trying to solve.

How Should You Use This? Link

The next time you’re debating with someone about where users are dropping out of a flow or how many users are watching a video tutorial, remember that these questions are easily answered by metrics.

Or if you’re trying to decide what sort of widget to use to access some data, remember that the right solution often depends on how much data there will be, and that question is entirely answerable as well.

Metrics Tell You Which Changes Are Most Effective Link

Let’s be clear. Your job as a designer is to improve the customer experience in some measurable way, right? If you do a complete redesign, and every designer you know loves it, and it wins awards from experts, but every single one of your users hates it and leaves and stops giving you money, then that’s a failure for the company.

Design changes — like code changes, marketing changes and customer service changes — should eventually make a company more money, because that’s how the company will stay in business and pay the designer’s salary.

With metrics, you can learn exactly what sort of an effect your changes will have on the numbers you care about.

Let’s say you’ve been obsessing over the navigation of your website. You feel it’s confusing for users and is frustrating them and causing them to leave. So, you revamp the navigation and run an A/B test so that half of users see the old version and half see the new one.

After a few thousand people have seen both versions, you have a very good idea whether changing the navigation has resulted in good things like increased revenue, time on site, and the number of users who would recommend you to a friend.

Of course, not every change you make will have a direct and significant impact on revenue. But knowing that any major change you make is having a measurable influence on the things you care about is very nice.

As a bonus, once you’ve made a few demonstrable improvements to revenue, asking for a raise or for more design resources becomes a lot easier.

How Should You Use This? Link

Get into the habit of A/B testing your design changes.

Obviously, you can make a few changes that you know will negatively affect certain metrics because they’re part of a larger strategy. But at some point you need to put a stake in the ground and say, “If we make the following design changes, then important metrics such as revenue and retention will improve.”

If you test all of your design changes against the status quo, you should eventually see those key metrics go up as a result of your work.

Metrics Give You The Freedom To Take Design Risks Link

Big design changes can involve big risks. After all, if what you’re proposing has the potential to vastly increase revenue, then it also has the potential to vastly decrease revenue.

Knowing that all of your changes will be A/B tested gives you a tremendous amount of freedom to try new and potentially dangerous things. If they pay off, you’ll find out immediately. If they cause problems, you can identify and fix them quickly.

You no longer need to worry that some change you’ve made will mysteriously ruin everything. And if you weren’t worried about that before, you probably should have been, because even very small design and text changes can have an enormous impact on user behavior.

How Should You Use This? Link

Be bold! The next time you’re inclined not to make an important change because you’re concerned it might have a negative impact, go ahead and try it. Roll the change out to 10% of your users, and get real data on whether it helps or hurts the product.

What Metrics Won’t Do Link

A lot of the push-back I get on measuring design is based on a fundamental misunderstanding of A/B testing and analytics. Some designers hear about Google testing dozens of shades of blue, and they feel that design shouldn’t be reduced to that level. After all, design is more than about picking a shade of blue.

Which blue is better?12
Which blue is better? (Image: visualpanic13)

In fact, metrics are terrible for a lot of things that design does well. For example, metrics can’t tell you how exactly to improve a design. They can only tell you whether your design is better or worse than another. They can tell you exactly what users are doing, but they can’t tell you why they’re doing it or how to make them stop.

Metrics can be an incredibly powerful tool, but they don’t replace design or make it irrelevant. In the end, the designers are still the ones making the decisions. They’re just making them with better information.

Want To Get Started? Link


Footnotes Link

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16

↑ Back to top Tweet itShare on Facebook

  1. 1

    Agreed! It should be a no-brainer that metrics are an essential part of the design toolkit. They don’t solve problems, but they help tell you where the problems are.

  2. 2

    One of the great things about metrics is that they can help you take emotion and taste out of an argument. A stubborn client or designer may really want something, but if the users have spoken it can help them see the light.

    • 3

      That is my favorite aspect of testing. Client wants an animated Flash intro with no skip button? A/B testing against a site with no intro. Boom. End of intro!

    • 4

      The worse thing is when the analytic argument doesn’t work.

      Drives me crazy when metrics and sense tell a customer to do point form descriptions for products and they ‘decide’ in there great wisdom that the analytic’s are wrong and people really like to read long winded descriptions with a bunch of waffle in them.

      Taught me to accept that some clients really are idiots.

      • 5

        /signed. That’s why I recently gave up on a very promising long-term project I’ve been working for and with since January 2011.

        cu, w0lf.

  3. 6

    Hi Laura – nice article. I think it would’ve been helpful to see examples of “questions easily answered by metrics” to drive your point home a bit more. However, I’m very firmly on the “metrics” side of design and can’t see how designers wouldn’t want to use. Often, designers (smart, tech-oriented, etc.) aren’t the normal users and it can be too easy to overlook something that seems intuitive to them. And of course designers have great intuition in order for them to be successful designers, but it’s not always right for the common customer. That good intuition can’t transition into an “I know what’s best attitude.”

    I really like your “it takes the guessing out of the project” comment. If anything, just allow the metrics to give you insight then decide whether to take it or leave it. I don’t ever look at metrics as replacing design but rather something that enhances it, an accessory to the whole outfit.


  4. 7

    Bravo! Your explanation is right on the mark. I’m a design and a web analyst and it’s so wonderful to see this explained so thoroughly. Design, content, and analysis of visitor behavior must work in concert.

    Web analytics can be another tool in the designers toolbox that marries creative with effective conversions. Start talking to a potential employer the way this article talks and you’ll be hired. Start talking like this to your current employer and you might get a bonus! If the purpose of the site you design for is to sell something (and I bet it is), then embrace this. And start reaping the rewards.

  5. 8

    Great one!

  6. 9

    Simon W. Jackson

    September 21, 2011 4:36 am

    I think that it’s important to keep in mind how many users you will need to conduct an effective A/B test. Just like any other statistics measurement, you cant judge the result of an A/B test if you only have a handfull of people visiting your site on a daily basis.

  7. 10

    Great article! I definitely agree that when I was working closely with a web analytics person, I was able to target my interaction design to the problems at hand on the real site. I always felt that talking the analytics gave me a better read on the current user– which was key, since we didn’t always have the resources to do early user interviews.

    When I used analytics, though, I was at a digital marketing agency– the goal was always conversion. I switched jobs a few months back and now focus simply on digital experiences, including things like online portals. Can metrics tell a story here as well, when the goal is just to give the user a good experience?

  8. 11

    That’s right developers! Start listening to your marketing team. :)

  9. 12

    marvelous post Paul !!

    done well Laura, you have done a tremendous effort which shows your experience in web designing , keep it up . we are also developing and designing websites, just because we like to give our expertise in web designing and developing. We are in the habit of developing and implementing suggestions. Autson works with innovative ideas to bring in new variations. your useful info will be helpful to get an idea..

    We are here


↑ Back to top