The Ultimate Guide To A/B Testing

Advertisement

A/B testing isn’t a buzz term. A lot of savvy marketers and designs are using it right now to gain insight into visitor behavior and to increase conversion rate. And yet A/B testing is still not as common as such Internet marketing subjects as SEO, Web analytics and usability. People just aren’t as aware of it. They don’t completely understand what it is or how it could benefit them or how they should use it. This article is meant to be the best guide you will ever need for A/B testing.

What Is A/B Testing?

At its core, A/B testing is exactly what it sounds like: you have two versions of an element (A and B) and a metric that defines success. To determine which version is better, you subject both versions to experimentation simultaneously. In the end, you measure which version was more successful and select that version for real-world use.

This is similar to the experiments you did in Science 101. Remember the experiment in which you tested various substances to see which supports plant growth and which suppresses it. At different intervals, you measured the growth of plants as they were subjected to different conditions, and in the end you tallied the increase in height of the different plants.

A/B testing example1
Large version2

A/B testing on the Web is similar. You have two designs of a website: A and B. Typically, A is the existing design (called the control), and B is the new design. You split your website traffic between these two versions and measure their performance using metrics that you care about (conversion rate, sales, bounce rate, etc.). In the end, you select the version that performs best.

What To Test?

Your choice of what to test will obviously depend on your goals. For example, if your goal is to increase the number of sign-ups, then you might test the following: length of the sign-up form, types of fields in the form, display of privacy policy, “social proof,” etc. The goal of A/B testing in this case is to figure out what prevents visitors from signing up. Is the form’s length intimidating? Are visitors concerned about privacy? Or does the website do a bad job of convincing visitors to sign up? All of these questions can be answered one by one by testing the appropriate website elements.

Even though every A/B test is unique, certain elements are usually tested:

  • The call to action’s (i.e. the button’s) wording, size, color and placement,
  • Headline or product description,
  • Form’s length and types of fields,
  • Layout and style of website,
  • Product pricing and promotional offers,
  • Images on landing and product pages,
  • Amount of text on the page (short vs. long).

Create Your First A/B Test

Once you’ve decided what to test, the next step, of course, is to select a tool for the job. If you want a free basic tool and don’t mind fiddling with HTML and JavaScript, go with Google Website Optimizer283. If you want an easier alternative with extra features, go with Visual Website Optimizer4 (disclaimer: my start-up). Other options are available, which I discuss at the end of this post. Setting up the core test is more or less similar for all tools, so we can discuss it while remaining tool-agnostic.

You can set up an A/B test in one of two ways:

  • Replace the element to be tested before the page loads
    If you are testing a single element on a Web page—say, the sign-up button—then you’ll need to create variations of that button (in HTML) in your testing tool. When the test is live, the A/B tool will randomly replace the original button on the page with one of the variations before displaying the page to the visitor.
  • Redirect to another page
    If you want to A/B test an entire page—say, a green theme vs. a red theme—then you’ll need to create and upload a new page on your website. For example, if your home page is http://www.example.com/index.html, then you’ll need to create a variation located at http://www.example.com/index1.html. When the test runs, your tool will redirect some visitors to one of your alternate URLs.

Once you have set up your variations using one of these two methods, the next step is to set up your conversion goal. Typically, you will get a piece of JavaScript code, which you would copy and paste onto a page that would represent a successful test were a visitor to arrive there. For example, if you have an e-commerce store and you are testing the color of the “Buy now” button, then your conversion goal would be the “Thank you” page that is displayed to visitors after they complete a purchase.

As soon as a conversion event occurs on your website, the A/B testing tool records the variation that was shown to the visitor. After a sufficient number of visitors and conversions, you can check the results to find out which variation drove the most conversions. That’s it! Setting up and running an A/B test is indeed quite simple.

Do’s And Don’ts

Even though A/B testing is super-simple in concept, keep some practical things in mind. These suggestions are a result of my real-world experience of doing many A/B tests (read: making numerous mistakes).

Don’ts

  • When doing A/B testing, never ever wait to test the variation until after you’ve tested the control. Always test both versions simultaneously. If you test one version one week and the second the next, you’re doing it wrong. It’s possible that version B was actually worse but you just happened to have better sales while testing it. Always split traffic between two versions.
  • Don’t conclude too early. There is a concept called “statistical confidence” that determines whether your test results are significant (that is, whether you should take the results seriously). It prevents you from reading too much into the results if you have only a few conversions or visitors for each variation. Most A/B testing tools report statistical confidence, but if you are testing manually, consider accounting for it with an online calculator5.
  • Don’t surprise regular visitors. If you are testing a core part of your website, include only new visitors in the test. You want to avoid shocking regular visitors, especially because the variations may not ultimately be implemented.
  • Don’t let your gut feeling overrule test results. The winners in A/B tests are often surprising or unintuitive. On a green-themed website, a stark red button could emerge as the winner. Even if the red button isn’t easy on the eye, don’t reject it outright. Your goal with the test is a better conversion rate, not aesthetics, so don’t reject the results because of your arbitrary judgment.

Do’s

  • Know how long to run a test before giving up. Giving up too early can cost you because you may have gotten meaningful results had you waited a little longer. Giving up too late isn’t good either, because poorly performing variations could cost you conversions and sales. Use a calculator (like this one6) to determine exactly how long to run a test before giving up.
  • Show repeat visitors the same variations. Your tool should have a mechanism for remembering which variation a visitor has seen. This prevents blunders, such as showing a user a different price or a different promotional offer.
  • Make your A/B test consistent across the whole website. If you are testing a sign-up button that appears in multiple locations, then a visitor should see the same variation everywhere. Showing one variation on page 1 and another variation on page 2 will skew the results.
  • Do many A/B tests. Let’s face it: chances are, your first A/B test will turn out a lemon. But don’t despair. An A/B test can have only three outcomes: no result, a negative result or a positive result. The key to optimizing conversion rates is to do a ton of A/B tests, so that all positive results add up to a huge boost to your sales and achieved goals.

Classic A/B Testing Case Studies

Here are some case studies to give you an idea of how people test in the wild.

Writing Decisions: Headline Tests on the Highrise Sign-Up Page7
37signals tested the headline on its pricing page. It found that “30-Day Free Trial on All Accounts” generated 30% more sign-ups than the original “Start a Highrise Account.”

37signals A/B test8

“You Should Follow Me on Twitter Here”9 (Dustin Curtis)
This much-hyped split-test involved testing multiple versions of a call to action for Twitter followers. Dustin found that “You should follow me on Twitter here” worked 173% better than his control text, “I’m on Twitter.”

You should follow me on Twitter here10

Human Photos Double Conversion Rates11
A surprising conclusion from two separate A/B tests: putting human photos on a website increases conversion rates by as much as double. Scientific research backs this up, saying that we are subconsciously attracted to images with people.

Human photos vs. generic icon12

Google Website Optimizer Case Study: Daily Burn, 20%+ Improvement13 (Tim Ferriss)
A simple variation that gave visitors fewer options too choose from resulted in a 20% increase in conversions. The winning version was also much easier on the eye than the control in its detail and text.

Home page screenshot14

Two Magical Words Increased Conversion Rate by 28%15
The words “It’s free” increased the clicks on this sign-up button by 28%, illustrating the importance of testing call-to-action buttons and how minor changes can have surprisingly major results.

It's free screenshot16

Changing the Sign-Up Button from Green to Red17
Along with its other A/B tests, CareLogger increased its conversion rate by 34% simply by changing the color of the sign-up button from green to red!

Green v/s Red18

Single page vs. multi-step checkout19
If you have an online store, it is quite common to see visitors abandoning the purchase process at the time of checkout. This A/B test found out that a single page checkout process works much better at completing sales than multiple-page checkout process.

checkout20

"Mad Libs" style form increases conversion 25-40%21
Defeating conventional wisdom, in this A/B test it was found out that a paragraph-styled form with inline input fields worked much better than traditional form layout. Though the result was probably specific to their offering as it wasn’t replicated in another, separate A/B test22.

madlibs23

Complete redesign of product page increased sales by 20%24
A software product company redesigned their product page to give it a modern look and added trust building elements (such as seals, guarentees, etc.). End result: they managed to increase total sales by 20%. This case study demonstrates the effect of design on sales.

aquasoft25

Marketing Experiments response capture case study – triple digit increase in conversions26
Through a series of A/B tests they optimized the mailing list opt-in rate by 258%. Focus was to remove all distractions and require the visitor to only provide email address. For completing his/her complete profile, the landing page motivated the visitors with an Amazon gift card (which was again split tested).

marketing-experiments27

Tools For A/B Testing

A number of tools are available for A/B testing, with different focuses, price points and feature sets. Here are some:

Resources For Deep-Diving Into A/B Testing

If you’ve read this far, then A/B testing has presumably piqued your interest. Here, then, are some cherry-picked resources on A/B testing from across the Web.

Get Ideas for Your Next A/B Test

Introductory Presentations and Articles

The Mathematics of A/B Testing

(al)

Footnotes

  1. 1 http://www.smashingmagazine.com/wp-content/uploads/2010/06/abtesting.gif
  2. 2 http://www.smashingmagazine.com/wp-content/uploads/2010/06/abtesting.gif
  3. 3 http://www.google.com/websiteoptimizer
  4. 4 http://visualwebsiteoptimizer.com/
  5. 5 http://visualwebsiteoptimizer.com/ab-split-significance-calculator/
  6. 6 http://visualwebsiteoptimizer.com/ab-split-test-duration/
  7. 7 http://37signals.com/svn/posts/1525-writing-decisions-headline-tests-on-the-highrise-signup-page
  8. 8 http://37signals.com/svn/posts/1525-writing-decisions-headline-tests-on-the-highrise-signup-page
  9. 9 http://dustincurtis.com/you_should_follow_me_on_twitter.html
  10. 10 http://dustincurtis.com/you_should_follow_me_on_twitter.html
  11. 11 http://carsonified.com/blog/design/human-photos-double-your-conversion-rate/
  12. 12 http://carsonified.com/blog/design/human-photos-double-your-conversion-rate/
  13. 13 http://www.fourhourworkweek.com/blog/2009/08/12/google-website-optimizer-case-study/
  14. 14 http://www.fourhourworkweek.com/blog/2009/08/12/google-website-optimizer-case-study/
  15. 15 http://visualwebsiteoptimizer.com/split-testing-blog/ab-test-case-study-how-two-magical-words-increased-conversion-rate-by-28/
  16. 16 http://visualwebsiteoptimizer.com/split-testing-blog/ab-test-case-study-how-two-magical-words-increased-conversion-rate-by-28/
  17. 17 http://dmix.ca/2010/05/how-we-increased-our-conversion-rate-by-72/
  18. 18 http://dmix.ca/2010/05/how-we-increased-our-conversion-rate-by-72/
  19. 19 http://www.getelastic.com/single-vs-two-page-checkout/
  20. 20 http://www.getelastic.com/single-vs-two-page-checkout/
  21. 21 http://www.lukew.com/ff/entry.asp?1007
  22. 22 http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-own-tests/
  23. 23 http://www.lukew.com/ff/entry.asp?1007
  24. 24 http://carsonified.com/blog/business/the-business-case-for-ab-testing/
  25. 25 http://carsonified.com/blog/business/the-business-case-for-ab-testing/
  26. 26 http://www.marketingexperiments.com/blog/research-topics/response-capture-case-study.html
  27. 27 http://www.marketingexperiments.com/blog/research-topics/response-capture-case-study.html
  28. 28 http://www.google.com/websiteoptimizer
  29. 29 http://www.bingocardcreator.com/abingo/
  30. 30 http://vanity.labnotes.org/
  31. 31 http://visualwebsiteoptimizer.com
  32. 32 http://unbounce.com/
  33. 33 http://performable.com/
  34. 34 http://vertster.com/
  35. 35 http://sitespect.com/
  36. 36 http://www.webtrends.com/products/optimize.aspx
  37. 37 http://www.omniture.com/en/products/conversion/testandtarget
  38. 38 http://whichtestwon.com/
  39. 39 http://www.conversion-rate-experts.com/articles/101-google-website-optimizer-tips/
  40. 40 http://abtests.com/
  41. 41 http://visualwebsiteoptimizer.com/ideafox.php
  42. 42 http://elem.com/~btilly/effective-ab-testing/
  43. 43 http://exp-platform.com/Documents/GuideControlledExperiments.pdf
  44. 44 http://20bits.com/articles/an-introduction-to-ab-testing/
  45. 45 http://20bits.com/articles/statistical-analysis-and-ab-testing/
  46. 46 http://www.evanmiller.org/how-not-to-run-an-ab-test.html
  47. 47 http://visualwebsiteoptimizer.com/split-testing-blog/what-you-really-need-to-know-about-mathematics-of-ab-split-testing/
  48. 48 http://blog.asmartbear.com/easy-statistics-for-adwords-ab-testing-and-hamsters.html
  49. 49 http://www.cennydd.co.uk/2009/statistical-significance-other-ab-test-pitfalls/

↑ Back to topShare on Twitter

Paras Chopra is founder of Visual Website Optimizer, the world's easiest A/B testing tool. Used by thousands of companies worldwide across 75+ countries, it allows marketers and designers to create A/B tests and make them live on websites in less than 10 minutes.

Advertising
  1. 1

    Interesting case studies, will look at implimenting some of that onto sites I am building.
    Good post, keep it up

    2
  2. 2

    “no words can describe the excellent post !”
    thanks alot

    2
  3. 3

    Great guide, Paras!

    0
  4. 5

    I really liked these case studies – make for great reading and inspiration.

    0
  5. 6

    How can it be an ultimate guide without even mentioning local minima/maxima in passing?

    And I disagree with the notion that you should never let artistic vision trump the results of a test. Artistic vision often includes knowledge of where things will be going in the future and you shouldn’t let a paltry 5% improvement get in the way of that.

    0
    • 7

      The issue of minima and maxima is not crucially important, in my opinion. That’s because A/B testing is a methodology which lets you do any time of test you want to perform. Though I agree you may want to do both: small changes and radical changes in order to see if you are truly optimizing global maxima.

      Well, artistic sense over test results is a subjective opinion! Though my own preference is to always let data speak for itself.

      3
      • 8

        Interesting, I think this comes down to the debate between the marketer and the designer. And it will always be a great debate. There is value on both sides. It’s finding common ground and a compromise between them both. Ultimately – the data should speak for itself…and ultimately, it’s the data that SHOULD guide the designer to understand what should be kept in mind for the future .. not just in the way of aesthetics but also when taking usability, conversions and overall success into account.

        0
      • 9

        This sounds a bit like someone with a hammer trying to turn every problem into a nail. A/B testing is great for certain very narrow, *but critical* channels of user experience — also for domains that are more or less traditional and fixed in their expected format. At the level of strategy, over-all site orientation and high-level goals, though, A/B tests are the equivalent of using a narrow beam flashlight to look for something that you don’t really know the size or contours of, somewhere in North America, in the dark (their search is ultra-focused, binary and random or grouped around the current location). It has its place and we need good tools, but please don’t make it out to be the answer for all things UX just because it’s your baby. (By the way, I think A/B testing is a mighty fine hammer, there just needs to be some balance in its application and the evangelism surrounding it.)

        0
  6. 10

    Interesting…very use-full article
    thanks for sharing paras

    0
  7. 11

    Great article!
    how about more UX case studies. I think it would be great.

    Thanks

    0
    • 12

      Paras – Thanks for mentioning our WhichTestWon.com site as one of your favorites for more testing info. Currently we have 65 a/b test and multivariate testing Case Studies, complete with creative samples and results data, up on our site for your surfing pleasure. We add new tests weekly too.

      The case studies include B2B tests, ecommerce tests, email tests, lead generation form tests… you name it! That’s our job as testing journalists – enjoy!

      0
  8. 13

    Good to see coverage of this.

    However, we should be careful not to overstate the results of a single test – the effects of a change can be very audience/context specific.

    The example used to demonstrate that “a single page checkout process works much better at completing sales than multiple-page checkout process” was, in my opinion flawed.

    As well as changing the checkout from a multi-page to a single page process, the test also significantly changed how ‘guest’ customers were treated.

    In the first example many first time customers may have assumed that they would have to create an account due the the page layout and the copy in the panel headers. It is not immediately clear that you can buy as a ‘guest’. In the second example this potential barrier has been removed. Instead, returning customers are asked for a login, whilst it is much clearer that new customers can continue without registering.

    3
    • 14

      John, great comments. I will go even as far as saying that you should never trust results of a case study. Yes, case studies give ideas and inspiration for A/B testing but applying them without doing any testing on one’s own website is a grave mistake. As you said, results depend on a ton of factors which include context, product offering, brand value, etc. So, all A/B case study results should be taken with a pinch of salt.

      0
  9. 15

    Great Article

    0
  10. 16

    Great Post Paras!

    0
  11. 17

    Jean-Francois Monfette

    June 24, 2010 5:39 am

    This is a very good post about A/B testing. I go to whichtestwon every week to try to see if I could pick the winning test. Sometimes results are surprising.

    Tools : There is a website called whichmvt that made a very nice table about the features and pricing of many a/b testing tools (including yours). It’s worth a look for anyone searching for a tool to do A/B tests.

    Good luck with your startup and keep on A/B testing !

    0
    • 18

      Jean-Francois, thanks for wishing me well! Yes whichmvt.com is an excellent site (though I didn’t include it because “multivariate testing” may have confused some readers).

      0
  12. 19

    If you’re using ExpressionEngine, I’ve started an add-on to enable easy, basic A/B Testing: http://www.markjreeves.com/ab-test-expressionengine/

    0
  13. 20

    Great article. Nice balance of theory and practice, without making the content too simple or too complex. Maybe next time discuss multivariate testing?

    1
    • 21

      Thanks alot – your answer solved all my problems after several days strggluing

      2
  14. 22

    Nice article. I particular like the case studies presented. GREAT JOB!

    0
  15. 23

    That’s a great article indeed, easy to follow and very informative. Thanks!

    0
  16. 24

    Great article. Very informative and you gave some great show cases.

    0
  17. 25

    Very good and informative article.

    Thanks Paras Chopra and SM

    0
  18. 26

    Have you seen http://mixpanel.com ? Seemed missing from the article. We use it at our company for a/b testing in our funnels and elsewhere.

    Cool article Paras

    0
  19. 27

    In the don’t section, the third bullet points says “Don’t surprise regular visitors.” How would someone be able to implement that?

    Well, I guess browser cookies might be helpful, as you can check whether the visitor to your site is a regular or visitor. What are some other ways to implement that though?

    0
    • 28

      Yes, most A/B testing tools store the variation displayed to visitor in a cookie so that on his repeat visits the same variation is shown.

      0
  20. 29

    I literally conducted a test with Google Website Optimizer yesterday and am implementing into our CMS now. How ironic that you post this article today. Thanks much!

    0
  21. 30

    Paras;
    Great article – thanks for sharing. I would like to add just two thoughts:
    1) Inbound Marketing and Marketing Automation tools also provide A/B Testing capabilities. Of course they aren’t free, but if you go down the road of Inbound Marketing Automation (IMA), you will get the capability as a “free” add on to the lead management, qualification and nurturing functions.
    2) As IMA tools also track people’s digital footprints around your site as they interact with your landing pages, in addition to the conversion metrics you get for the A/B test, you can also determine which pages held visitor’s interests and which ones resulted in them leaving the site. In other words, you also gain insight into your prospect’s behavior which add depth to the results of your A/B tests.

    0
    • 31

      Eric, interesting point. Can you name a few marketing automation tools that have A/B testing integrated? I’ll admit that I don’t have enough information on this front.

      0
  22. 32

    Very good, it is useful

    0
  23. 33

    thank you, very useful

    0
  24. 34

    Really good useful info. I’ll definitely check out your product next time I’m in need of an A/B test.

    0
  25. 35

    very interesting article. I want to read like this more in SM..

    0
  26. 36

    servicemagic.com is pro at this.

    0
  27. 37

    I’ve always been concerned with the performance implications of client-side A/B testing. Delivering separate content by replacing sections on the page with Javascript seems clunky to me. Does anyone know of any server-side A/B testing frameworks that do not run on Ruby? Ruby has A/Bingo and Vanity, but I’m looking for others.

    3
  28. 38

    Really a Conversant post.Thanks for sharing. Looking forward for more.

    0
  29. 39

    Good stuff Paras :)

    0
  30. 40

    thanks for the detailed info. I’m just starting to learn A/B testing and this is a good start. I think that you have there the advice for noob like me. Any more advice for A/B testing on blogs? I would love to know if there’s any. =)

    1
  31. 41

    Great and helpful article Paras and thanks also to the Smashing Magazine team. I’ve often heard “A/B Testing” terminologies from web developers but often wondered what really goes on into this process. After reading this it provides you a heads up and better understanding of the relation between marketing and design.

    0
  32. 42

    Marc von Brockdofff

    June 24, 2010 10:46 pm

    Great info… started doing A/B testing a few months ago and find it very interesting. I have found that whenever I did a test, the gap between the winner and the loser always diminishes as time passes, so it’s important to have a result which is statistically significant. I wrote about this in my blog a while back in case anyone’s interested:

    http://www.marcvonbrockdorff.com/2010/06/17/ab-testing-statistical-significance/

    1
  33. 43

    Paras, glad to see your article up on Smashing!

    I still think that Visual Website Optimizer is one of the best and easiest A/B testing tools to use.

    Patrick will also be happy for the A/Bingo shoutout.

    0
  34. 45

    Hi Paras Chopra,
    First of all its a nice article;
    I would like to know if there are any such tools which work with asp.net & c#.
    Let me know if there are any…

    ~Sree

    0
  35. 46

    a great article! thanks!

    1
  36. 47

    Hello Paras,

    thank you for this insightful article. I’ m just starting to learn about different types of usability tests and as your article indicates A/B tests sound very promising and, with the right tool at hand quite easy to perform.:-) Thanks for sharing your experience!

    0
  37. 48

    SINGUP!

    0
  38. 49

    Great article … it came across fairly strongly as a sales pitch for your start-up though.

    (disclaimer: my start-up)

    I actually would expect a little less bias in an article that is just supposed to present helpful info – not an advertorial cloaked as such.

    Corey

    0
  39. 50

    Fantastic article. I love the presentation of the case studies. Thank you for an informative and comprehensive read that is forever bookmark worthy!

    0
  40. 51

    ….. cool information, great post, very informative…. lol, but when will it ever end !! :)

    0
  41. 52

    Great article!
    One thing,
    ‘testing the color of the “Buy now” button, then your conversion goal would be the “Thank you” ‘
    Is this correct? I would think that the Buy Now color test would have the result page be the very first cart page. Only saying success on ‘thank you’ seems like a very broad test just for a color change.

    0
  42. 53

    WOW! great article – thanks!

    0
  43. 54

    It was interesting to see a reaction to the sign-up button switch.
    Very informative article. Picked up some good points – Thanks

    0
  44. 55

    Thanks for this extremely detailed and informative post.

    0
  45. 56

    Nice post on A/B testing, Paras. Really nice case studies you provided. Anyone interested can also check out a case studies from American Greetings, Citrix, and other companies here that use LiveBall for landing page testing and optimization: http://www.ioninteractive.com/why-i-love-liveball/. Of course what works for one person, may not work for someone else, but case studies and best practices do provide a base to start from. Thanks!

    0
    • 57

      Hi Kristina,

      Ah, sorry to have missed LiveBall. Should have mentioned it along with other A/B testing tools! Any way, your case studies are interesting and thanks for commenting on the post.

      -Paras

      0
  46. 58

    Superb post! Thanks for teaching us A/B testing.

    -1
  47. 59

    i think we are all becoming slaves to computers

    1
  48. 60

    > “Don’t surprise regular visitors”

    No, no, no — your sample *must* be completely random for it to be chosen, otherwise your inference will be invalid, because you no longer have representative samples. This is very, very important!

    Don’t go to all the effort (and expense) of preparing A/B tests, just to nullify any statistical significance by messing up the sample allocation. Your regular visitors will get version B anyway if you choose the alternate, so surely it can’t be bad to give it to some early?

    0
    • 61

      It doesn’t mean that you are not showing variation to your repeat visitors, it just means that across different visits of a visitor you should keep your variations consistent. In fact, not doing so will kill your statistical significance.

      In A/B test your base unit is a visitor, not visits! So, if you are showing different variations on different visit and the visitor completes a goal in 3rd visit, which variation are you going to attribute this success to?

      0
  49. 62
  50. 63

    Amazing article,Thanks for sharing.

    0
  51. 64

    Great article. Thx. We are currently thinking of looking into A/B testing for our product, your article and resources are a big help!

    0
  52. 65

    testing the control first, yet test it simultaneously. Seems like weird advice in the Do’s and Don’ts. Which one?

    0
  53. 66

    Fascinating article!

    Of course, now I find myself asking, “Is the article supposed to be on the left side of the page? Is the Post Comment button ALWAYS green?” I’ll never look at websites the same. :)

    0
  54. 67

    Excellent POST!

    0
  55. 68

    You might want to add FairlyCertain ( http://www.fairtutor.com/fairlycertain/ ) to your list. It’s a lightweight AB testing library for ASP.NET and ASP.NET MVC, modeled along the lines of A/Bingo.

    0
  56. 69

    Nice article Paras! I was exploring the topic and found this one very nice. Thanks!

    0
  57. 70

    I recently set up a test using google adwords with a very basic landing page to test company names and value propositions but Google shut down the campaign after a few hours. How can you run tests without having an elaborate website? All tips and resources are much appreciated!

    0
  58. 71

    i know how to use A/B in to MKV to MP4 Mac, i Need to change my ideas,i read a similar articles .With the page layout optimization also need to continue to learn, thank you, good article refueling.
    How to keep customers, learned.

    -1
  59. 72

    Great beast of an A/B test guide = cool! I have put together a less detailed top tips guide for A/B and multi variate testing you might find vaguely interesting – http://www.searchmuse.com/blog/2011/7/23/advice-for-a-b-and-multivariate-testing.html

    1
  60. 73

    Hi,

    I would like to know if is there a way to do the A/B Testing in facebook games. Do you any tool for that? Because I investigated the Google Website Optimizer and it’s just for websites, right?

    0
    • 74

      Hi Mil, the HoneyLizer which is a social optimization platform (for Facebook) that serves personalized, ‘best-fit’ page options to each user, automatically and in real-time, can be used also for A/B testing.
      Feel free to contact me if you want more info – yaron[at]BeesAndPollen.com

      1
  61. 75

    Nice blog i like it thanks

    -1
  62. 76

    Hi Paras, I have a question for you that I’ve been struggling with so it would be great if you could help. Can you use GWO to A/B test bounce rate?

    1
  63. 77

    Thnks ,usefull article

    0
  64. 78

    I’ve created a ruby A/B testing tool called Split.

    Split is a rack based A/B testing framework designed to work with Rails, Sinatra or any other rack based app.

    It’s heavily inspired by the Abingo and Vanity rails ab testing plugins and Resque in its use of Redis and is designed to be hacker friendly, allowing for maximum customisation and extensibility.

    https://github.com/andrew/split

    0
  65. 79

    Thank you for this article! It is a great piece of work – not only for someone who wants to get started with A/B testing.

    0
  66. 80

    Thanks for a brilliant article. I have had three websites so far and have learnt from them all. Every website developer has raved about landing pages. It never happened as I would not relinquish content control to them. The link to ‘unbounce.com’ is such a kindness. Thank you. I have a self-build from a host for small companies. It is so slow to upload that it is pointless. Basically I want the perfectly built website. I will have a look around your site for answers. I will go and start. Thanks.

    1
  67. 81

    Unfortunately, many of the conclusions drawn from these a/b tests are quite flimsy. As a psychology researcher the flaws in the methodology are quite striking.

    Fortunately there are people like myself who audit such testing and let companies know whether or not the results are salient and actionable, or just junk. Indeed much of a/b testing is no more relevant than science 101 tests.

    0
  68. 82

    For example, the D. Curtis “experiment” listed above is tremendously flawed.

    You can read my breakdown of just a few of the problems with his testing here:

    http://researchaudit.wordpress.com/2012/09/18/flaws-in-the-d-curtis-you-should-follow-me-on-twitter-experiment-ab-testing/

    0
  69. 83

    Great post! Thank you :)

    0
  70. 84

    Even 3 years after the post, highly relevant and useful. Especially when the Internet industry has begun its next boom in India.
    Well written !!

    0
  71. 85

    A/B Testing is more like a marketing method than a method used for Search Engine Optimiztaion. After all, you are checking the behavior of people. You are not looking at how to exploit the weaknesses of search engines in order to grow the rank of your website.

    0
  72. 86

    Multivariate testing and observation is highly necessary in any internet marketing process. As much as we have useful analytics and demographics, A/B testing is its’ ultimate completion.

    Den,
    searchzinger.com

    0
  73. 87

    It is one of the deepest article about a/b testing. one more i have click there called 101 tips it is also amazing topics. For beginners like me it will help to do websites optimization so well. A BIg THANKS

    0
  74. 88

    Even if I discover your article 3 years after its publication you’re still right, people lack information about AB Testing, and do not dare trying it. This is why articles like yours is very useful ;-) Thanks for the tips you give, I will use them a lot. Here is another article on testing that may interest your readers: http://www.atinsight.co.uk/actualite/test-procedure-7-steps-to-successful-testing/

    0
  75. 89

    Visual Website Optimizer is a great tool!! Congrats!!

    0
  76. 90

    i don’t want to comment, but just hit like or something..

    anyway – has smashingmagazine testet design/usability un A/B test?

    0
  77. 91

    Good article, well written.

    Thanks Paras!

    0
  78. 92

    Just wanted to thank you, Paras, for this comprehensive guide (and especially the added resources). I have a similarly-related glossary on SEO terms (in which I reference this guide, as well as the Visual Website Optimizer tool) if anyone is interested in taking a look and offering any input – http://tarungehani.com/seo-glossary

    0
  79. 93

    any good ideas for APP AB testing? E.g. Android or IOS

    0
  80. 94

    It’s easy to use and convinient option for finding better result.

    1

↑ Back to top