Menu Search
Jump to the content X X
SmashingConf London Avatar

We use ad-blockers as well, you know. We gotta keep those servers running though. Did you know that we publish useful books and run friendly conferences — crafted for pros like yourself? E.g. our upcoming SmashingConf London, dedicated to all things web performance.

Multivariate Testing 101: A Scientific Method Of Optimizing Design

In a previous article on Smashing Magazine, I described A/B testing1 and various resources related to it. I have also covered the basics of multivariate testing2 in the past, yet in this post I’ll go deeper in the technical details of multivariate testing which is similar to A/B testing but with crucial differences.

multivariate testing

In a multivariate test, a Web page is treated as a combination of elements (including headlines, images, buttons and text) that affect the conversion rate. Essentially, you decompose a Web page into distinct units and create variations of those units. For example, if your page is composed of a headline, an image and accompanying text, then you would create variations for each of them. To illustrate the example, let’s assume you make the following variations:

  • Headline: headline 1 and headline 2
  • Text: text 1 and text 2
  • Image: image 1 and image 2

The scenario above has three variables (headline, text and image), each with two versions. In a multivariate test, your objective is to see which combination of these versions achieves the highest conversion rate. By combinations, I mean one of the eight (2 × 2 × 2) versions of the Web page that we’ll come up with when we combine variations of the sections:

  • Headline 1 + Text 1 + Image 1
  • Headline 1 + Text 1 + Image 2
  • Headline 1 + Text 2 + Image 1
  • Headline 1 + Text 2 + Image 2
  • Headline 2 + Text 1 + Image 1
  • Headline 2 + Text 1 + Image 2
  • Headline 2 + Text 2 + Image 1
  • Headline 2 + Text 2 + Image 2

In multivariate testing, you split traffic between these eight different versions of the page and see which combination produces the highest conversion rate — just like in A/B testing, where you split traffic between two versions of a page.

Getting Started With Multivariate Testing Link

To create your first multivariate test, first choose a tool or framework that supports multivariate testing. You can use one of the tools listed in the section “Tools” in the end of this article. Please note that not all A/B testing tools support multivariate testing, so make sure your tool of choice allows it.

Once you’ve decided which tool to use, choose which sections to include in the test. As you know, a Web page can contain tens or hundreds of different sections (footer, headline, sidebar, log-in form, navigation buttons, etc.). You cannot include all of these sections in the test; creating variations for all of them would be an enormous task (and, as you’ll read below, the traffic requirements for the test will grow exponentially with each new section). Narrow it down to the few sections of the page that you think are most important to the conversion goal.

The following parts of a page (listed in order of importance) are typically included in a multivariate test:

  • Headline and heading,
  • Call-to-action buttons (color, text, size, placement),
  • Text copy (content, length, size),
  • Image (type, placement, size),
  • Form length.

The Difference Between A/B Testing And Multivariate Testing Link

Conceptually, the two techniques are similar, but there are crucial differences. First and foremost, the traffic requirements are different. As I said, the number of combinations that need to be tested grows exponentially in a multivariate test. You can test three or four versions in an A/B test and tens or hundreds of versions in a multivariate test. Clearly, then, a lot of traffic — and time — is required to arrive at meaningful results.

For example, if you have three sections with three variations each, the number of combinations is 27. Add another section with three variations, and the total number of combinations jumps to 81. If you want meaningful results, you can’t keep adding sections to the test. Be selective. A good rule is to limit the total number of combinations to 25 or fewer.

Varation Testing3
Use A/B testing for large scale changes, not to refine or optimize existing designs. Image by Meet the Chumbeques4

Another difference is in how these techniques are used. A/B testing is usually reserved for large radical changes (such as completely changing a landing page or displaying two different offers). Multivariate testing is used to refine and optimize an existing design. For the mathematically inclined, A/B testing is used to optimize for a global optimum, while multivariate testing is used to optimize for a local optimum.

One advantage of multivariate testing over A/B split testing is that it can tell you which part of the page is most influential on conversion goals. Say you’re testing the headline, text and image on your landing page. How do you know which part has the most impact? Most multivariate testing tools will give you a metric, called the “impact factor,” in their reports that tells you which sections influence the conversion rate and which don’t. You don’t get this information from A/B testing because all sections are lumped into one variation.

Types Of Multivariate Tests Link

Based on how you distribute traffic to your combinations, there are several types of multivariate tests (MVT):

Full factorial testing
This is the kind people generally refer to when they talk about multivariate testing. By this method, one distributes website traffic equally among all combinations. If there are 16 combinations, each one will receive one-sixteenth of all the website traffic. Because each combination gets the same amount of traffic, this method provides all of the data needed to determine which particular combination and section performed best. You might discover that a certain image had no effect on the conversion rate, while the headline was most influential. Because the full factorial method makes no assumptions with regard to statistics or the mathematics of testing, I recommend it for multivariate testing.

Results for ItoWorld5
Record and compare the resulting traffic for each tested version. Image by ItoWorld6

Partial or fractional factorial testing
As the name suggests, in this method only a fraction of all combinations are exposed to website traffic. The conversion rate for unexposed combinations is inferred from the ones that were included in the test. For example, if there are 16 combinations, then traffic is split among only eight of those. For the remaining eight, we get no conversion data, and hence we need to resort to fancy mathematics (with a few assumptions) for insight. For obvious reasons, I don’t recommend this method: even though there are fewer traffic requirements for partial factorial testing, the method forces too many assumptions. No matter how advanced the mathematics are, hard data is always better than inference.

Taguchi testing
This is the most esoteric method of all. A quick Google search reveals a lot of tools claiming to cut your testing time and traffic requirements drastically with Taguchi testing. Some might disagree, but I believe the Taguchi method is bit of a sham; it’s a set of heuristics, not a theoretically sound method. It was originally used in the manufacturing industry, where specific assumptions were made in order to decrease the number of combinations needing to be tested for QA and other experiments. These assumptions are not applicable to online testing, so you shouldn’t need to do any Taguchi testing. Stick to the other methods.

Do’s And Don’ts Link

I have observed hundreds of multivariate tests, and I have seen many people make the same mistakes. Here is some practical advice, direct from my experience.

Don’ts Link

  • Don’t include a lot of sections in the test.
    Every section you add effectively doubles the number of combinations to test. For example, if you’re testing a headline and image, then there are a total of four combinations (2 × 2). If you add a button to the test, there are suddenly eight combinations to test (2 × 2 × 2). The more combinations, the more traffic you’ll need to get significant results.

Do’s Link

  • Do preview all combinations.
    In multivariate testing, variations of a section (image, headline, button, etc.) are combined to create page variations. One of the combinations might be odd-looking or, worse, illogical or incompatible. For example, one combination might put together a headline that says “$15 off” and a button that says “Free subscription.” Those two messages are incompatible. Detect and remove incompatibilities at the preview stage.
  • Do decide which sections are most worthy of inclusion in the test.
    In a multivariate test, not all sections will have an equal impact on the conversion rate. For example, if you include a headline, a call-to-action button and a footer, you might come to realize that footer variations have little impact, and that headline and call-to-action variations produce winning combinations. You get a powerful section-specific report. Below is a sample report from Visual Website Optimizer. Notice how the button has more impact (91%) than the headline (65%):
    MVT report7
  • Do estimate the traffic needed for significant results.
    Before testing, get a clear idea of how much traffic you’ll need in order to get statistically significant results. I’ve seen people add tens of sections to a page that gets just 100 visitors per day. Significant results from such a test would take months to accumulate. I suggest using a calculator, such as this A/B split and multivariate testing duration calculator8, to estimate how much traffic your test will require. If it’s more than what’s acceptable, reduce some sections.

Case Studies Link

A lot of A/B testing case studies are on the Web, but unfortunately, finding multivariate test case studies is still difficult. So, I scoured the Internet and compiled relevant ones.

Software Download Case Study: downloads increased by 60%9
This is one multivariate test I did to compare different versions of headlines and links. In the end, one of the variations resulted in a more than 60% increase in downloads.


Microsoft Multivariate Testing Case Study11
This presentation details the variations that were tested for this website and the ultimate winner.

SiteSpect Case Studies12
This page presents a dozen of multivariate testing case studies of large companies using multivariate testing and behavioral targeting to optimize their sites.

Maxymiser Case Studies13
Another set of multivariate testing case studies.

Look Inside a 1,024-Recipe Multivariate Experiment14
YouTube did a gigantic multivariate test in 2009. It can afford to do tests with a thousand-plus combinations because it has sufficient traffic.

Multivariate testing of an email newsletter
An agency tested color and text on the call-to-action button of its email newsletter. The best button had the highest CTR: 60%.

Multivariate Testing Tools And Resources Link

Tools Link

Google Website Optimizer15
A free basic multivariate testing tool by Google. It’s great if you want to test the waters before investing money in multivariate testing. The downside? You’ll need to tag different sections of the Web page with JavaScript, which can be cumbersome. It’s also prone to error and forces you to rely on others (like the technology department) for implementation.

Visual Website Optimizer16 (Disclaimer: I am the developer of this tool)
The main advantage of this paid tool is that you can create a multivariate test visually in a WYSIWYG editor by choosing different sections of the page. You can then run the test without having to tag sections individually (although a snippet of code is required in the header). The tool includes heat map and click map reports.

A website that publishes user reviews of all of the multivariate testing tools available on the market. If you are planning to adopt a multivariate testing tool for your organization, do your research on this website.

Enterprise testing tools
Omniture’s Test&Target18, Autonomy’s Optimost19, Vertster20, Webtrends’ Optimize21, and SiteSpect22.

Resources Link

Expert Guide to Multivariate Testing Success23, by Jonathan Mendez
A series of blog posts detailing different aspects of multivariate testing.

Fail Faster With Multivariate Testing24 (PDF)
An excellent free mini-guide to multivariate testing.

Online Testing Vendor Landscape25
A commercial report by Forrester that compares the various testing vendors out there.

Lessons Learned from 21 Case Studies in Conversion Rate Optimization26
This article discusses ideas for conversion rate optimization detailed through different case studies.

You may be interested in the following related articles:


Footnotes Link

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
  28. 28
  29. 29
  30. 30

↑ Back to top Tweet itShare on Facebook

Paras Chopra is founder of Visual Website Optimizer, the world's easiest A/B testing tool. Used by thousands of companies worldwide across 75+ countries, it allows marketers and designers to create A/B tests and make them live on websites in less than 10 minutes.

  1. 1

    That is cool! This is my first time I heard about Multivariate. Taguchi testing? Haha, sound like tamagotchi (handheld digital pet) :P

  2. 2

    Marc Van Rymenant

    April 4, 2011 2:40 am

    Hi Paras, many thanks for sharing this insight with us.

    Nevertheless, the real problem with Multivariate testing is that you have “how much” data (how much user clic on this solution and how much clic on the other solutions) but you never know “why” a solution works better than another ;)

    Behavioral Sciences help in that field …


    • 3

      Great point, Marc. Just having the data isn’t enough, you have to do something with it. Taking the time to analyze data and identify trends is a vital (yet oft overlooked) process. Investing the time gains invaluable insight into customer values/behavior.

  3. 4

    Mocambo khus hua… Just joking.

    Really a nice post. helpfull in future…

  4. 5

    Hmmm, not something I’d ever see myself using unless I were on a gigantic budget and needed something to pass time in all honesty :x

    Thx for sharing this though, was an interesting read!

    • 6


      I agree. This is something that I can see being worthwhile, but on medium/biggish projects where I have so much actual ‘making’ to do I don’t know if I have the time. I will try to find the time though!

  5. 7

    Paras, very useful article. Suggest pointing out that you are the lead developer of Visual Website Optimizer in the article itself rather than only in the bio at the end, because when it is syndicated on rss that information doesn’t get included… So, when I went to VWO from a link in my RSS feed, I was surprised to see the article author hadn’t mentioned his affiliation…

    • 8

      Paras Chopra

      April 4, 2011 9:48 pm

      I had put a disclaimer in the article, but Smashing Mag team removed it before publishing it. If you see my previous articles, you’d see I make sure I mention my affiliation to Visual Website Optimizer.

  6. 9

    I enjoy stats and this was probably the most fun literature I’ve read involving multivariate analysis. Does Google have any SPSS-style variance testing?

  7. 10

    David Sundrum

    April 4, 2011 1:04 pm

    Great article. I enjoyed learning about the ins and outs of these two testing methods. I tend to work on larger sites that get lots of traffic, and are looking to increase the traffic even more. We’ve done some work with A/B testing, so it was nice to get a grasp of the advantages and limitations of multivariate testing as well. Thanks.

  8. 11

    John Waterman

    April 4, 2011 3:00 pm

    The more powerful the microscope or telescope, the farther up your ar*e you get.

  9. 12

    The problem I always have with such testing is that any test is inevitably and constantly changing the sample size.

    IE at the start of the test you are sampling everyone: and that includes your most important customers who always get taken out early.

    What I mean is, at the start the 20% of your customers who account for 80% of your revenue will respond a certain way, and then be removed from the next day’s sample.

    Eventually you are left with the 80% of people who either account for just 20% of your business or never buy.

    You can tell when this happens because your graph lines will cross over.

    The question you then have to ask is: Do you want to optimize for you best customers who bought at the start of the test, or your worst laggards who may or may not buy later?

  10. 13

    I have implemented a number of multivariate tests for agile marketing campaigns utilitising the tools with Ektron CMS400.Net (I believe the module is called the ‘marketing optimisation suite’).

    It is very easy to administer with the drag and drop widget CMS user interface and the simple way to define experiment sections and targets. Easy to test ideas like copy, layout, imagery, and call-to-action – ideal so that marketers are able to deliver the best possible web experience that drives visitors to take action.

    I have seen double digit improvements of conversation rates on campaigns (and landing pages) from using this tool.

    I recommend including this in your list of tools for a commercial product.
    Ektron Marketing Optimisation Suite:

  11. 14

    Taguchi’s methods are still useful for web site testing. The premise for Taguchi is to reduce the size of a full factorial matrix. In the physical world you would have to build parts to fill all those boxes on the matrix “largest pin” “smallest hole” “heat treat” “no heat treat” and so on. Web testing will be the same… build a ‘blue’ theme and a ‘red’ theme and have menu bar across the top vs down the side and so on. If you build a full factorial matrix of test pages you’ll be at it a long time. Taguchi techniques allow you to reduce the number of test variations and still get the primary drivers.

    To see the advantage of Taguchi: If you are testing a 10-variable matrix, that’s 100 different variable matches you must make in a traditional full-factorial experiment. Taguchi methods allow you to test with only 50 or 25 variable matches. Then you can build and run the test to get feedback in a few days vs a few weeks.

    What you give up with this method is all the magical interactions: a blue theme vs a red theme might not matter, but a blue theme with a menu down the side has 1000x the impact of any other setup…and you might miss that ‘interaction’ combination using Taguchi.

    But you don’t want to spend six months running a full factorial experiment either. .. However, there are ways to get both a fast test cycle and more information out of the test.

  12. 15

    This is the first time I heard about this tool … thanks for sharing

  13. 16

    Thank you very much testing different variables and draw conclusions, but very interesting! this article I feel very good

  14. 17

    I wish people knew the difference between scientific and empirical.

  15. 18

    Mark @ Alchemy United

    April 12, 2011 4:37 am

    Good stuff. Thank you.

    I’d like to add that if you’re using PPC to drive in traffic that too becomes one of the variables. Depending on how many keywords and ads you have, permutations can increase quite a bit.

    That said, I think the key take away is that set it and forget it is not a best practice. At any given moment some resources should be devoted to R&D. While sample size, etc. might not always be ideal, don’t take the mad scientist had off :)


↑ Back to top