Reliable Cross-Browser Testing, Part 1: Internet Explorer

Advertisement

In a perfect world, cross-browser testing would be straightforward. We would download a legacy version of a browser, run it, and be able to instantly test our pages and scripts without a single care in the world. The reality of cross-browser testing, though, is very different. Issues such as runtime conflicts when running multiple versions of the same browser and inaccurate third-party testing tools mean we can spend hours just evaluating whether a testing set-up is anywhere near reliable.

I’m a user-interface developer at AOL (yes, we’re not dead yet!), and in this multi-part post I’ll take you through the exact set-up we use to accurately test content that will be potentially viewed by up to millions of users with a very diverse set of browsers. This set-up is similar to the one used by some of my colleagues at Opera, Mozilla and Google, so, fingers crossed, we’re doing this optimally.

A quick note before we begin. Setting up accurate testing for Internet Explorer as outlined in this post requires a bit of effort. So, please check your website analytics first to ensure that a sufficient number of IE users visit your website in the first place to warrant this effort.

Internet Explorer 6 To 10

Let’s begin with our old friend, Internet Explorer (IE). As most of us know, running multiple versions of the original Internet Explorer executables on the same system is very difficult due to issues ranging from runtime version conflicts to operating-system incompatibility. In truth, I don’t think Microsoft ever considered a scenario in which developers needed a way to achieve this back when it was first conceiving IE 6, 7 and 8.

This has left developers in a chasm of uncertainty, forced through trial and error to discover a way to accurately test what is (for better or worse) the world’s most widely used set of browsers.

In this section, I’ll take you through some IE testing options that you may be using or have heard of before. I’ll explain why they might not be reliable; and then I’ll present the solutions we currently use at AOL for production websites.

In case you’re interested, our team generally uses IE 7 as a baseline, although we do also test stable, beta and preview or dev-channel versions of Chrome, Firefox, Opera, Safari and, of course, IE 8 to 10.

Our reason for using IE 7 as a baseline comes down to our global website analytics: IE 6, 7 and 8 are the most common browsers used to access our websites. However, we stopped supporting IE 6 as of a few months ago. The reality is that IE 6 has major compatibility issues with modern technologies, and our team has found that we are able to deliver projects up to 20% more quickly when we don’t have to worry about catering a basic experience to it.

Most Third-Party IE Testing Tools Are Unreliable

A Google search for third-party IE testing tools will result in at least ten different options, nine of which are likely to be unreliable. Let’s go through a sample of them, in case you’re using one of them to test against staging or production websites at your work.

The following reviews may sound a little harsh, but the purpose is to stress the importance of using approaches that have been well engineered, tried and tested over time.

IETester

Unfortunately, IETester1 is probably the most popular third-party tool that I see designers and developers use to test multiple versions of IE. When I first tried it, I too was drawn by the promise of a single application that would solve all of my IE testing woes. However, all that glitters is not gold.

screenshot

The tool has a number of inconsistencies when testing IE 6, 7 and 8, with none providing a 1:1 experience of the original browsers. Our and many other teams have confirmed not only that the rendered output of IETester varies significantly from any real version of IE, but that pop-up interactions cause failures, Flash and CSS filters don’t work in user mode, conditional CSS comments often fail, and switching between versions makes it very prone to crashing.

In short, it’s unreliable, and this should be enough for any developer to consider alternatives.

Multiple IE

TredoSoft’s Multiple IE2 is another tool that often shows up when searching for an IE testing solution. Unfortunately, it too suffers from a number of issues, including inaccurate rendering. One common complaint is that people experience IE 5.5 rendering bugs even when they’re just testing IE 6; this certainly is not something we want to deal with at a time when many of us are trying to ditch IE 6 completely.

Multiple IE isn’t updated regularly either, and if the long thread of user issues experienced with it since its release doesn’t put you off, then consider that the tool’s IE textboxes actually break in a number of circumstances.

screenshot

To the best of my knowledge, both IETester and Multiple IE rely on an exploit known as DLL redirection in order to bypass issues with DLL naming conflicts, allowing the tools to attempt to run standalone copies of IE. I would recommend avoiding such tools, because implementing a completely sandboxed environment for IE that is as accurate as running the originals independently is very difficult.

Expression Web SuperPreview

Next up is Microsoft’s Expression Web SuperPreview3, which claims to simplify the process of testing and debugging layouts across multiple versions of Web browsers.

screenshot

Unfortunately, you won’t be able to test complete user interaction, JavaScript, DOM manipulation, animation or much else with this tool. We live in an age when the Web can be very dynamic and, in some cases, highly interactive. A tool like this might work for baseline testing, but not for accurate cross-browser testing.

BrowserLabs, Browsershots, BrowserCam

I personally use Adobe’s BrowserLabs4 — or sometimes Browsershots5, if a static layout test is absolutely necessary. But again, neither of these options allow you to test interactions with your pages. The same goes for BrowserCam6. These services simply weren’t designed for this purpose, but we still regularly see designers and developers using them as if they were.

I’m not in any way saying to flat out avoid these services, but rather that they’re inadequate for complete cross-browser testing. Designers and developers need to know exactly what a user sees when they interact with their website. No visitor will be using a static page renderer, so why rely on one yourself?

Windows Virtual PC

One other solution you’ll probably come across is Microsoft’s Virtual PC7, with time-bombed images8 for IE6 to 8.

For a brief time this was considered to be the answer to cries from developers for a better solution. Unfortunately, it is by far the most inadequate (and demanding) solution to testing that I’ve seen proposed in the past few years. At least 12 GB of disk space is required to install all of the images, and the images regularly expire during the year.

A cross-browser testing environment for legacy browsers should have reasonable system requirements and, for the most part, not require regular manual updating in order to continue using it. Because the Virtual PC option fails these criteria, I can’t recommend using it either.

Browser And Document Modes

We’ve looked at third-party tools, but what about Microsoft’s current solutions to these problems?

Both IE 9 and IE 10 PP2 support switchable document and browser modes via the F12 Developer Tools for cross-version compatibility testing. To be specific, browser-version testing here is made possible using a kind of emulation.

screenshot

“Document mode” determines what features a page has access to and what can be adjusted based on the page’s doc type, X-UA-compatible meta tag and headers. For example, the standards document mode allows the page to take advantage of IE’s implementation of ECMAScript 5 (ES5), while the IE 7 and 8 standards modes offer an alternative experience.

“Browser mode,” on the other hand, emulates different IE browser version behaviors and can be changed directly from the IE Developer Tools. Emulation is achieved in a few different ways, but it includes altering both the document mode and the user-agent string. In case you’re wondering, the UA string is adjusted here to ensure that code that relies on UA sniffing functions as though the correct version of the browser were being used.

screenshot

It is worth noting that IE 9 shipped with a newer JavaScript engine, called Chakra. While the browser itself supports a number of compatibility modes, because the JavaScript engine itself differs significantly from what shipped with IE 6, 7 and 8, there are acknowledged differences9 between the experience in IE 9 and testing in a standalone browser.

That aside, there are, unfortunately, a number of quirks in the way that both document mode and browser mode function. Our team has run into scenarios where IE returns the wrong user-agent string to the server; and, as with the third-party tools, there have been several instances of inaccuracies with the expected rendering when tested against the original browsers.

Finally (and rather strangely), these modes have a number of issues of their own that are not present in IE 6, 7 or 8, making it even more difficult to establish whether the issues experienced are specific to a browser version or just the mode being used.

For these reasons, I would urge developers not to rely strictly on document or browser mode for their cross-version testing needs.

Accurate IE Testing Solutions

Option 1: Virtual Machines + IECollection

We’ve reviewed a number of solutions that don’t provide an accurate means of testing multiple versions of IE. So, what does work? The answer lies in using dedicated virtual machines (VMs) and a tool called IE Collection1510.

screenshot

Out of the box, Utilu’s IE Collection offers the following:

  • A tested collection of standalone versions of IE, including versions 1 through 9;
  • Confirmed support for accurate rendering when compared against the original IE executables;
  • Confirmed support (to date) for the correct IE JavaScript engine implementations that shipped with the originals;
  • Confirmed support for the correct UA strings being detectable (not that you should be UA-string testing in the first place — however, if needed, it’s there);
  • Support for both the 32-bit and 64-bit versions of Windows XP, 2003, Vista and 7;
  • Access to the IE Developer Toolbar, which comes with the standard IE Collection set-up; this is compatible with IE 5 and above, but you also have the option to install Firebug Lite if you prefer that.

Although the majority of IE versions are supported and function reliably within the collection, there are known issues with versions of IE 7 when used under Windows Vista or Windows 7 (as noted on Utilu’s website). I’ll discuss how we handle this limitation shortly, but let’s first briefly go over virtual machines.

screenshot

A virtual machine (VM) is a sandboxed “guest” operating system that run within a normal “host” operating system. It effectively enables you to install and run a dedicated copy of almost any other operating system on your current one and to share a configurable set of memory and disk resources between the two. For example, a Windows 7 VM could easily be run on Mac OS X, as could an Ubuntu VM be run on Windows, and so on.

You’ll need two things to get started using VMs: a virtualization tool to create VMs on your OS, and a CD or image of the operating system that you wish to run as a virtual machine. Your company should be able to furnish you with Windows licenses relatively easily (… famous last words). If you’re a college student, you could probably get licenses for Windows XP SP3, Vista and 7 for free through the MSDN Academic Alliance11 program.

You could set up a number of different configurations to test Windows and IE, but if you’re using IE Collection, I would generally recommend the following:

  • Windows 7 for IE 6, 8, 9, 10;
  • Windows XP (SP3) for IE 7.

You shouldn’t need to support anything below IE 6 nowadays, but IE 4 to 5.5 also work fine on XP SP3.

At AOL, we use a combination of VirtualBox12, Parallels13 and VMware14 for our virtualization needs. I prefer Sun’s VirtualBox because it has the following advantages:

  • It’s free, providing a lower barrier to entry for developers who cannot afford commercial licenses for Parallels or VMware;
  • In my experience, it tends to perform more optimally than Parallels or VMware under the same configuration settings;
  • It has a marginally better UI than the alternatives.

That said, VMware is considered better if you’re working with a standalone server. At the end of the day, all three options are equally valid, and both Parallels and VMware come with trial versions that you can easily evaluate. Choose whichever option best suits your needs.

Pro tip: Remember to configure your VM to use the minimim amount of memory needed to test, unless you are using a dedicated testing box. By default, not all virtualization tools are configured to do this. This will simply ensure that the rest of your OS runs more smoothly.

Here are the three steps to setting this up:

  1. If you’re a Mac user, I recommend using a 32-bit Windows 7 Pro or Ultimate image (or installation CD), with a minimum of 17 GB of disk space allocated for the installation. 512 MB of virtualized RAM (which can be adjusted in the settings panel) should work fine for testing on a non-dedicated testing machine, but use more if feasible. You can create this as a new VM using VirtualBox, which I have configured to run at a resolution of between 800 × 600 and 1024 × 768 pixels. Windows users have two options. If you’re already running a version of Windows 7, skip to step 2, where we’ll begin installing IE Collection. If you’re running a different version of Windows and would like to run a virtualized copy of Windows 7 on top of it, then follow the steps above for Mac users. Mac and Windows users should boot up their VM or desktop Windows installation before proceeding to step 2.
  2. Next, download IE Collection1510 from Utilu’s website, and then install Windows 7 on either your VM or your primary system, depending on your set-up. This will give you access to IE 6, 8 and 9. I usually install IE 10 for testing purposes separately, but bear in mind that it is only in the platform preview phase at the time of writing. Ideally, install only the bare essentials to test on any VM in order to avoid excessive start-up times.
  3. Mac and Windows users: to address the issue of IE 7 not working on Windows 7, you’ll have to create another virtual machine with Windows XP installed (ideally with SP3). Because no reliability issues have been reported for running either IE Collection’s IE 7 or the original IE 716 on this OS, your bases should be covered here. As stated in my previous instructions, obtain a Windows XP image or installation CD, and create a new VM using VirtualBox. Boot it up, install IE7 through the link above, and you should be good to go.

That’s it really! Some developers prefer to run IE 6 and 7 on XP (step 3) rather than on Windows 7 or Vista, because XP accounts for the majority of users who have yet to upgrade to IE 8 or 9. This, too, is completely valid; the set-up is flexible enough to support a little personal preference.

Next, for the sake of being thorough, let’s look at some alternatives to the set-up I’ve recommended above.

Option 2: Dedicated VMs for Each Version of IE

Pragmatic developers might wish to maintain a dedicated VM for each version of IE being tested. On these VMs, rather than using any third-party tools, they would install the original, official IEs instead.

screenshot

The reasoning behind this is that it potentially minimizes the risk of individual browser configurations or overrides from interfering with one another, and it ensures that you get only the exact version of IE that you wish to test (and, in turn, the corresponding JavaScript engine in that version).

If you’re setting up dedicated VMs for individual releases of IE, I recommend the following set-up:

  • Windows XP (SP3) for IE 6,
  • Windows Vista (SP2) for IE 7,
  • Windows Vista (SP2) for IE 8,
  • Windows 7 for IE 9 and 10.

Knowing now about dedicated VMs, you might be wondering why we don’t use this set-up at AOL. I can only speak for my team, but we moved away from this approach because running three to four VMs on the same system and still using it efficiently for other tasks can become unmanageable.

With a minimum configuration of 512 MB of RAM per VM, we would be looking at around 2 GB of RAM (if not more) just for testing purposes. This set-up might work if you currently have 8 GB or more of system memory. But if not, there’s still hope.

We could always get a dedicated box just for testing purposes, and run all of our virtual machines directly on there instead. I recently acquired a second box at work for this very purpose. But at the end of the day, both options 1 and 2 are set-ups you can rely on.

Option 3: VNC or RDP + Dedicated Testing Boxes

screenshot

A few companies (such as ZURB) prefer to use a form of remote desktop access to allow designers and developers to log into a dedicated testing box or set of boxes for cross-browser testing, over a network. The benefit of this is reduced demand on memory for each client. And this can be implemented in a few ways; VNC and RDP are the first that come to mind.

Virtual network computing (VNC) is a platform-independent desktop sharing system that allows you to remotely control another computer. It transmits both keyboard and mouse events from one system to another, relaying screen updates back in the other direction over a network. Similarly, RDP allows users to access applications over a network, but it is a proprietary protocol developed by Microsoft. If you happen to be running Mac OS X, VNC is already built in, but a number of VNC clients are also available in the wild, such as RealVNC17. On the RDP side of things, Mac users can use Apple’s own Remote Desktop 18 and Remote Desktop Connection19 by Microsoft.

You might be wondering what the differences are between VNC and RDP. RDP is semantic and is more aware of controls, fonts and other graphical primitives, while VNC is a little “dumber” in this regard. VNC, though, permits a session to be shared on the target machine, which might be useful for multi-user demos; whereas RDP does not, as far as I’m aware.

With the decreased memory requirements, you might then wonder why we don’t opt to use remote desktops for testing at AOL. The primary reason is that, in some cases, testing over a network can result in packets (effectively, frames) being dropped, which can affect our ability to test highly interactive applications or websites. Also, websites that make use of rich-media experiences based on canvas, SVG or otherwise aren’t reliably tested over networks for this reason. But don’t let our experience put you off. I know of teams in other companies that haven’t run into these issues, and remote testing might be suitable for some networks more than others.

Option 4: Remote Testing Through a Third-Party Service

If your team is unable to purchase or rent dedicated boxes for remote testing due to cost, there are third-party services you can use which which readily host host multiple versions of Windows, along with historic versions of IE and other browsers out of the box for a low subscription fee.

BrowserStack.com20 and Browserling21 are two such options. You might be surprised to know that both actually support local testing of your servers and files through ssh tunneling and as long as your company are comfortable with content being tested in this manner, it’s an adequate and affordable way of getting reliable cross-browser testing into your business.

If I had to personally recommend one of the services over the other, it would definitely be BrowserStack. They offer a wider range of browser versions and OS versions than Browserling and also include debugging and developer tools as a part of their service.

There is however one downside to the services recommended above: at the time of writing, neither currently appear to support the Firefox nightlies/builds for FF7,8,9, Opera.next nor the dev/beta/canary builds of Google Chrome.

I personally like the flexibility to test how my projects might render and function in upcoming browsers as this provides you with a level of comfort about a release – eg. if FF7 gets released next week, I’d rather be able to patch projects before everyone starts updating rather than after. That said, this isn’t a concern thats shared by a lot of developers or designers, so feel free to try out BrowserStack or Browserling if Options 1-3 aren’t for you.

Another Alternative For Developers Wishing To Avoid Purchasing Multiple Licenses

Some developers may be familiar with Microsofts virtual machine disk images which I mentioned earlier. It can be quite a challenging task getting these images to work with anything other than Microsofts VirtualPC, however a recent project called IEVMS22 (thanks to Paul Irish for the heads up) attempts to solve this problem. IEVMS offers a set of scripts that facilitate using the images with VirtualBox on Linux or OSX and with a single command it’s possible to have IE6 – IE9 running in separate virtual machines.

The project also offers a clean snapshot mode, potentially enabling developers to roll-back these VMs so that its possible to have them working indefinitely. This project does (at the time of writing) suffer from a few minor bugs that are being worked out, but is certainly worth looking at if it sounds more appealing than the alternatives. Bear in mind that this script will still need to download the 10-12GB of disk images required for the VMs and I’m not sure if IEVMS breaks the EULA, so ensure this won’t be an issue before launching the scripts.

Conclusion

I hope this guide has come in handy. Please feel free to share your own stories and experiences of cross-version testing IE, because they may help other readers decide which option to try out first.

In part 2 of this post, we’ll look at cross-browser testing modern browsers. See you then!

(al)

↑ Back to topShare on Twitter

Addy Osmani is a Developer Programs Engineer on the Chrome team at Google. A passionate JavaScript developer, he has written open-source books like 'Learning JavaScript Design Patterns' and 'Developing Backbone Applications', having also contributed to open-source projects like Modernizr and jQuery. He is currently working on 'Yeoman' - an opinionated workflow for building beautiful applications.

  1. 1

    Great article! This has been a major headache for me for a while. Good to finally have a solution.

    0
  2. 2

    Virtual machines are the way to go when it comes to proper cross browser testing. I use a single XP VM with standalone IE versions from the Spoon.net collection.

    However a service that may have been overlooked by this article is BrowserStack – it seems to be new to the market but has great potential, which can avoid the need for that heavy VM installation.

    0
    • 3

      Nicolas Gallagher

      September 4, 2011 7:17 am

      Gareth, for IE testing on an OS X host, that is pretty much what I do too.

      * Windows XP VM with IE6, the Spoon.net IE7 and IE8 exe’s, and a bunch of legacy Firefox installations (2.0 – 3.6).

      * Windows 7 VM with IE9 and IE10 preview, modern Firefox, Chrome, Safari, Opera.

      Clean snapshot for each VM, and another snapshot after all the browsers are installed. I should really set up an Ubuntu VM too.

      0
  3. 4

    You forgot to mention using VM snapshots as you upgrade from IE7 to 8 to 9.

    0
  4. 5

    Sometimes this little document also comes in very handy, when testing and debugging in IE9:

    http://techfiles.de/dmelanchthon/images/dc1f60588300_13A29/How-IE9-Determines-Document-Mode.png

    0
  5. 6

    I don’t know if it’s still the case as it’s been a while since I used it, but Browsercam used to have a remote access service, so you were in fact able to test interactions.

    Another alternative that you haven’t mentioned that I’m doing is to use Windows 7’s XP Mode http://ieblog.members.winisp.net/images/InstallingXPMode.htm

    0
    • 7

      Browsercam allows remote desktop sessions into preconfigured os/browser combinations for testing functionality

      0
  6. 8

    @Addy We have launched an online browser testing tool, Browserstack.com in beta. You don’t need VM’s or local installations anymore.

    you can even test local server or html designs.

    0
  7. 9

    How about stylizer? It’s an application to test CSS in live veiewing mode and comes bundled with all IE versions and other major browsers. I find the app is prone to crashing a lot but it is still useful for the cross browser testing. Did anyone find out how accurate is the testing in this app?

    0
    • 10

      Stylizer suffers from a number of issues including regularly crashing when tested with sites that support media queries. In addition to this, as mentioned in the post, I would be hesitant to trust any all-in-one apps claiming to offer 1:1 experiences compared to what you would get with VM-testing.

      For full disclosure: I haven’t thoroughly put it through it’s paces as much as IETester, but I personally wouldn’t trust it if you intend on sticking something tested with it up on production. It likely uses a similar setup to IETester/MultiIE so I’ll have to retain my position on the recommendations in the post being more reliable.

      My two cents :)

      0
  8. 11

    I am using Windows 7 with IE9.
    IE9 has a developer toolbar which makes it very easy to view your site in an IE7&8 mode. So why all the hassle?

    0
  9. 17

    This is an awesome post! Thanks for your guide, this has been an issue for a while and I knew IETester was not the answer!

    0
  10. 18

    I’m so glad to see a realistic view of browser testing – too often do I see blog posts pushing apps like IE tester.
    Not only are many of the so-called cross browser test platforms utterly unreliable they also cannot mimic what happens in the real world – particularly those sites or apps that rely heavily on javascript for functionality.
    I work in eCommerce web design and we make it a priority to have 4-5 PCs available to enable us to cover testing for all browsers we support (for our customers).

    0
  11. 19

    I’ve tried using http://browserling.com or http://saucelabs.com I believe they’re using a cloud type VM solution so they allow Javascript interactions too. Give it a shot but as always – YMMV.

    0
  12. 20

    Didn’t we already have a funeral for IE6 ?!

    http://ie6funeral.com/

    The bastard came back to life ! !

    0
  13. 21

    IE = useless awful evil browser in all its version that I won’t ever test anymore.

    But Thanks anyway, it’s always interesting to read about some of us going on supporting that s***.

    People just have to switch, so just let explain them and let IE die .

    0
    • 22

      It is true that there are enough reasons for IE to die. But the choice is users’. You have to respect that.

      0
    • 23

      Totally agree. Sucks so many gave the thumbs down. We don’t need to ‘respect’ the fact that ‘they’ are using a totally obsolete technology. We need to educate those on the evils of IE(6)* and help them evolve. Keeping quiet and saying ‘fine, no problem’ only propogates the legacy of terror. Take a stand, my peers, and fight EVIL!!! =)

      M

      *The evils spreads beyond 6, but we need some sort of witch to burn. =P

      0
  14. 24

    Ultimate IE Collection has been a godsend for me. I have OSX lion running Parallels with Windows XP and Windows 7, IE collection on both. Rendering and DOM behavior are true to IE. It should be noted, that it’s best to get your hands on an old copy of Windows XP service pack 2. I suggest this because the native IE browser is version 6, IE Collection works best on this platform, and the native application files are the truest-performing under this version of windows.

    0
  15. 25

    One other thing that recently tripped me up in doing IE testing was that group policies can also cause odd browser issues. My particular problem was that group policy was forcing IE8 into compatibility mode (and thus importing an incorrect stylesheet) when browsing a site internal to the network. Just one more thing to examine when you want pages just right in the IE landscape.

    Take a look at Microsoft’s 1300 group policies for IE8: http://technet.microsoft.com/en-us/library/cc985351.aspx

    0
  16. 26

    Honestly this is stupid. You shouldn’t have to *test* for IE. Are we in 2006? Why are you even wasting your time? The only people that need to test for IE are those that do complicated, and often times stupid show-off-y things with CSS, and they’re asking for it. I never test for IE and my sites look fine. Just my 0.02, I know this is a dev blog so this is a good article, I’m just saying that testing for IE is like an elitist thing to do, and you shouldn’t do it, because if you code properly, you sites will look fine in IE, without “hacks”. Just learn to code better.

    0
    • 27

      You mention if you have Windows 7 installed, you can simply skip to step 2 – what about if you already have Windows XP installed, and I want a Virtualbox installed with the same Windows XP? Do I need an additional licence and/or installation disk/image for this?

      0
    • 28

      How do you know your sites look fine if you haven’t tested them? I know I can make mistakes and I prefer to send forward work I _know_ looks and works like it should, so I test.

      0
    • 29

      So you’re saying to not take advantage of the new features in HTML5 and CSS3 just so you don’t have to worry about testing IE? If that’s the case then yes, you are still in 2006.

      0
    • 30

      dude, it’s an elitist thing to do to not test for IE, just the things you say are pretentious. Stupid show-off-y things? Like what, you don’t even have a fallback for rounded corners like css pie , ah I forgot ,rounded corners are showy…. I bet you don’t even use a shim or a shiv for HTML5 elements.Think about all the little small town libraries that still use IE6 or the third-world countries where their only escape from life is surfing on the net to think about a better life. They also deserve to have the best experience. Stop pretending to be
      a progress evangelist when you’re just frustrated and lazy.

      0
    • 31

      Prove it then dude, show us some of your sites and we can judge how well they hold up in IE.

      0
    • 32

      I can’t stand self-centered “developers” such as yourself.

      Get over yourself and realise there is much more to the world than you.

      0
  17. 33

    You mention if you have Windows 7 installed, you can simply skip to step 2 – what about if you already have Windows XP installed, and I want a Virtualbox installed with the same Windows XP? Do I need an additional licence and/or installation disk/image for this?

    0
    • 34

      If you already have WindowsXP installed (I’m assuming SP2 or SP3), you can go ahead and install IECollection for the earlier versions of IE. You can then proceed to install VirtualBox and use it along with a more recent version of Windows (7 or Vista) for IE9+10. For either of these you *will* need an additional license and install disk/image.

      0
      • 35

        Ok thanks – I didn’t get the fact that you don’t necessarily need a virtual machine at all to run the multiple versions of IE with IECollection…just if you need a version not supported by your current OS.

        0
  18. 37

    Excellent article. I actually just upgraded my Windows 7 Home Premium to Ultimate so I could install Virtual PC to test IE6-9. In my Windows 7 I have shortcuts to IE6-9 according to instructions found at http://www.sitepoint.com/ie6-ie7-ie8-win7-xp-mode/. Works like a charm. I’m going to try IE Collection at work where we’re still running Windows XP.

    0
  19. 39

    Excelent post.

    However, as far as I can tell the crossbrowsertesting.com site is more or less the same as browsercam, i.e. offering Screenshots and VNC sessions for all relevant OSes and Browsers.

    Why do you say that you can’t test interaction with BrowserCam if they also offer VNC? Mistake or have they recently changed something?

    0
    • 40

      For one thing, If you are working in a development environment and your dev site isn’t accessible outside that environment (and I’m going to make the guess that most corporate sites are set up this way) then crossbrowsertesting.com can’t even see your site.

      0
  20. 41

    Great article(!) and I think some very fair points regarding Microsoft not really planning for (or knowing) how the future would be. The tool I am playing about slightly more often, is Adobe BrowserLab. Looking forward to the next edition of your write-up.

    0
  21. 42

    Great post. I’m glad to see the method I’ve been using (VM + IECollection) is the suggested one. It’s unfortunate we have to jump through such hoops for IE…..as usual.

    0
  22. 43

    There are also cloud-based saas services such as browser mob.com, testrunner.com, saucelabs.com – any experiences or opinions as to this approach? Pros are not having to setup all the vm’s & hardware – cons??? Are they up to snuff yet?

    0
    • 44

      A friend just pointed me at saucelabs.com. I spent about 20 minutes testing it out and my feedback would be that it’s quite similar to crossbrowsertesting and a few others services like it – the testing setup would work fine if you’re working with primarily ‘static/flat’ sites, but anything highly dynamic or which requires testing of artifacts like animation/canvas etc, may suffer due to lag. Feel free to evaluate and use best judgement on these.

      0
  23. 45

    lol you had me at “I’m a user-interface developer at AOL (yes, we’re not dead yet!)”

    Great article and thanks! Although I have VM on my Mac, I’ve been using CrossBrowserTesting.com but its not cheap and its laggy. I had no idea about IE Colllection, so thanks a bunch. Now I can cancel my account and buy some pizza! :-)

    0
  24. 47

    This was hugely helpful – thanks for taking the time and posting such a detailed breakdown of our options.

    0
  25. 48

    Nice read. I’ve been using the IECollection you describe in option one forever. I don’t use it on a virtual machine though.

    0
  26. 49

    Smashing Magazine follower from the get-go. Solid article! I appreciate having more articles focused on development, rather than just design.

    0
  27. 50

    thanks so much, great article. helps me to get along through the browser-compatibility jungle!

    0
  28. 51

    Looks like the first link at “Expression Web SuperPreview” gives a 404?

    Good article by the way.

    0
  29. 53

    Yes, testing for old IE is a nightmare. I really appriciate you point out switchable document and browser modes are unreliable. We are using VMs with stand alone installs.

    0
  30. 54

    Some weeks ago xdissent pushed a nice shellscript to github which allows you to run IE6-9 in Virtual-Box-VMs with one single command in the bash (works on Linux and OSX).
    I really love it, so i think this would be helpful for you guys here too :-)

    https://github.com/xdissent/ievms

    0
  31. 55

    In my opinion, webdevelopment and usage of new technologies can only go seriously forward, if the browsing device is kept up to date.
    I don’t see any use or reason to support more then one version number back the current.
    Everyone want’s IE6 to be dead … but most of them still “optimizes” their stuff for it. That’s ridiculous, really. Why put time and effort and money into something, just because users don’t get their ass up and upgrade?
    If one wants a modern website, then he has to use a modern browser. If he don’t use a modern browser on modern websites, he has to suffer a few to a lot of problems from it. It’s that simple.

    I just can’t take an old VW Bug and go win a NASCAR Race …

    We as developer have it in our hands to bring this forward. As developer you have to have the balls to tell your clients: “If you want this, you have to use that, otherwise it won’t work. Period.”. If you don’t tell and explain it to the clients, they will expect “all the old stuff” to work, because often they don’t even know it won’t.
    The developer has to make old behaviours of clients go away, and done right, he has modernized another company (I say this, because I had already a few of such successful “transformations”).

    You just have to do this and not wasting time with “crossbrowser testing” … :)

    0
    • 56

      (I should point out that IE6 coverage is only mentioned for the sake of being thorough, but I’ll address your points about older IE versions regardless)

      Although we (as designers and developers) would love for all our customers and clients to be using modern browsers, the reality is that it’s unfortunately not always as straight-forward as saying ‘upgrade if you want to use our site’.

      E.g. if you’re a company like Amazon and you discover that 25% of your users are still using IE7 (for whatever reason) you *are* going to try making sure these users have an experience that works and is very well tested. To not do so might run the risk of impacting a significant portion of revenue. You don’t want your users jumping ship elsewhere because of a broken looking site, right?

      Could we as an industry get together and say ‘None of us are going to support IE7 or IE8′ the same way we did with IE6?, Sure. Of course we could and it probably will happen eventually. Remember however that there are a number of groups who aren’t as easy to force upgrades upon:

      1) Government offices often have locked down rules regarding operating systems and browsers that can be used. In many cases this comes down to Windows XP/Vista + some variation of IE6-8
      2) The elderly

      There will be a day when older versions of IE are no longer an issue, but it’s going to take time for that to happen. Until then, if anyone needs to test across older versions of IE for whatever reason, this post will hopefully come in useful :)

      0
      • 57

        There are other groups, too, which were already mentioned, like those in countries where it’s difficult to afford new machines and upgrades. Also, some people just don’t care or understand. They just want their computer to always behave the way it did when they bought it.

        Each project will have its own testing requirements. If you are a developer who takes on a variety of projects or if you represent one website, you just need to keep in mind the demographic of users for the site in question. Some sites will cater more to the affluent/knowledgeable and others will have a demographic of the less affluent/those who don’t know or care. If your clients reside say, mostly in less affluent countries, then, you will need to test legacy browsers. If you are a high-end brand, you may not need to. Just pay attention to your stats and decide how much of a percent of your site visitors you are willing to let down. It comes down to that. Also, having graceful degradation plans in place is a good idea, too.

        0
  32. 58

    The one big problem, especially for us who don’t use Windows natively is the expense and inconvenience involved in getting Windows 7 for the virtual machines. I don’t know what the Windows DRM status is these days, but I can’t imagine it’s improved since the XP days. I don’t want to give any money to Microsoft for both idealistic reasons (I think their products are horrible), or for pragmatic reasons (How much does a Windows 7 installer that I can use as many times as I want cost? If there is no such thing, how many virtual machines can I build before I hit my install limit, and what happens next?) so what I ideally need is a way to run IE9 and/or 10 without Windows at all. Other than Crossover/WINE (which I’ve always found awkward and not very reliable) are there any other options? I would even consider buying Win7 for use in a VM if I was sure I could build as many VMs as I wanted without having to beg MS for more installs.

    0
    • 59

      Hey Gordon. That’s completely understandable. You didn’t mention if closed-network testing was a requirement:

      If you do need closed-network testing support, I’d recommend using IEVMS – https://github.com/xdissent/ievms. This allows you to install the Microsoft IE testing images automatically so that it can run on OSX/Linux without issue. It also claims to come with a ‘clean’ state option that lets you reset your VMs every time they expire (so they last forever), but I haven’t personally confirmed this as yet.

      If you would prefer not to download 10-12GB of images and don’t need support for closed-network testing, I would recommend considering options like BrowserStack or CrossBrowserTesting if you would prefer to avoid the expense of multiple Windows licenses. They’re relatively affordable and you don’t have to worry about owning personal licenses for any of the O/Ss being accessed.

      Hope that helps!

      0
  33. 60

    Check out http://browserling.com/
    Cross-Browser testing from the browser!

    0
    • 61

      Looks like Browserling uses IETester for IE6 and 7. It worked pretty well for me on most browsers, although there was lag time. Interesting!

      0
  34. 62

    A word of caution about virtual machines: I’ve encountered several false-positives when trying to test Internet Explorer 6 using VirtualBox *and* Virtual PC. I’ve then replicated completely different false positives using IETester. Whilst I’ve never endured a false _negative_ using a VM, a false positive can still waste a lot of time and effort as the developers discover they can’t replicate the bug QA raise.

    We’re not sure what causes the bugs in question. The devs think it’s to do with the rendering engine and the way it expects its threads to be handled – the VMs don’t quite behave like a real machine in this regard, so the browser trips over. However, those sorts of discussions are out of my field of expertise. All I know is: VirtualBox IE6 testing has been very frustrating to me and my fellow QA staff.

    0
    • 63

      I got into a big discussion at work today when I was told the virtual machines I loaded up for testing IE6, IE7 and IE8 would not help us due to the false positives that you can experience on an idealized VM. Items that will render properly on the VM but not actually in the browser running on a native host. This blows my mind and really I haven’t come up with an explanation for this anomaly nor have I personally experienced this, but I also haven’t done much if any IE compatibility testing and the folks I work with have. I would like to see an article or follow up that explores this issue more. I am replying to Jimmy who confirmed this in his comments, wondering if you have any more details on how this could arise in a VM environment when we have VMs that use native execution … Thanks!

      0
  35. 64

    Great article Addy :)

    0
  36. 65

    This article was very thorough and helpful. I used to use Multiple IE, but I have found it to be unreliable for the reasons discussed. Though I don’t test in IE6 anymore (stopped in 2010, and will NOT do so anymore unless a client explicitly requests it), I still need to test in IE7, 8 and 9. I think it’s a bit early for 10, but that part of the discussion is moot if a solution you mention here can allow for however many versions one needs.

    Really curious to see what the next part discuss, especially since most front-end developers I know of just test in the latest versions of all the non-IE browsers and leave it at that.

    0
  37. 66
  38. 67

    You forgot to mention that the Microsoft-supplied Virtual PC solution is completely free, while your own solution requires purchasing a new copy of Windows for each VM required.

    0
    • 68

      Thanks. You’re indeed correct, but similar to IETester and MultiIE, I guess I assumed that most developers are aware that the IE images are free. We’re going to update the post to reflect this and offer an additional tip or two around the MS IE images for anyone wishing to try these out with VirtualBox on the Mac or on Linux.

      0
  39. 69
  40. 70

    Very useful, thanks Addy!

    I didn’t know about IE collection. I’m using Windows 7 as my primary OS and test IE9, Chrome, Safari, Opera, Firefox on it, and use 2 VMs both have Windows XP installed, one have IE7 and one have IE8. I’ll try IE collection after reading this post.

    Thanks again :)

    0
  41. 71

    Great Article. Looking forward to the rest of the series.

    0
  42. 72

    I downloaded Utilu IE Collection (from their website) and Comodo Firewall detected a trojan, and removed IE6.
    Don’t know the specifics about whether it was infact a trojan, but better to be safe than sorry.

    Be warned, I guess.

    0
    • 73

      I had the same problem with IE Collection. Detected a trojan and removed IE6.

      0
    • 74

      That’s a false positive.

      You can check the files on Virus Total. Most scanners will tell you there is nothing wrong with them.

      In any way, there is no trojan in it.

      0
  43. 75

    What does the writer think of BrowserStack?

    0
  44. 77

    Good call on not supporting IE6 anymore. Even Microsoft are discouraging its use now and figures speak for themselves when it comes to the cost-effectiveness of supporting it:
    http://www.ie6countdown.com/

    0
  45. 78

    In some cases, you will have to support IE6 (as a customer requirement for example).
    For those who don’t want to spend some time installing vm, i found a really cool website that allows you to test your site on the major browsers : https://saucelabs.com/
    No word about it in this article, but you really should give it a try !!

    0
  46. 79

    Ironically whilst trying to read this article on a mac that I’m updating using safari 3, it consistently crashes the browser.

    0
  47. 80

    Nice and comprehensive article, thank you.

    I take my chance to promote an article I’ve written a year ago which is pretty similar to this, but also includes a section about performance testing. Performance – as we all know – isn’t the best possible in Internet Explorer: http://samuli.hakoniemi.net/debugging-and-testing-in-internet-explorer-made-easy/

    0
  48. 81

    I think the best way to reliable test IS setting up separate VMs for each IE, unfortunately, that’s a resource hogger. I found IE default developer tools to be the most helpful; however, they are not available for Mac. :/

    0
  49. 82

    Hi guys,

    lovely post on cross browser techniques. I’ve personally used testing boxes in the past. I’ve grown a tad tired of using BrowserStack as I often find it gives me inaccurate results and I have to deal with problems when a site comes back with dodgy layouts or errors. I’d love BrowserStack to compensate for the problems we come across as designers as it is a problem that is yet to be solved. Anyone come across with many browser issues using HTML5?

    James Scott,
    tristarwebdesign.co.uk

    0
    • 83

      @James, We were in beta then. We have grown a lot since then. We are almost on verge of coming out of beta. Could you give browserstack another spin?

      0
  50. 84

    Thanks for such a good article. We have faced this problem in past what we did was to find and download portable editions of IE from version 6 to 9 and use them.

    There are many floating on the web.

    If I remember correctly once upon a time spoon.net visualization solution (that allows one to run desktop app in a browser without having to install them) hosted all version of IE but later they were forced to remove them.

    0
    • 85

      The Spoon.net virtualization service was a godsend when it first came out and at that time, many of us relied on their virtualized IE6-8 to avoid running multiple VMs. Unfortunately, as you know, MS issued a takedown notice for their IE images and it all fell apart from there. There *are* still some Spoon images out there in the wild, but for anything which requires installation on corporate/business machines, I recommend any of the options mentioned in this post.

      0
  51. 86

    so to test in various versions of IE you propose 3 virtual machines and 3 operating systems. hilarious, and ridiculous. you think my company is going to buy windows xp, windows vista, and windows 7, just to test IE??

    and how do i get a legal copy of windows xp? it’s not made anymore!

    this joke article should be re-titled: “Dowload illegal torrents of all operating systems to test IE”

    0
    • 87

      I can completely sympathize with smaller companies who might not be able to afford multiple licenses to Windows for this setup (btw, XP can still be purchased). Whilst dedicated VMs are what I consider to be the perfect testing configuration you can get by paying for something a lot more affordable like BrowserStack or CrossBrowserTesting. They do have their limitations, but again, this might work better for your budget.

      An alternative (possibly better option than BrowserStack) mentioned in the post are Microsofts time-bombed IE testing images that are free, but require a re-download every couple of months once they expire. We’ll be updating the post shortly with some additional tips around this and a script called IEVMS that you might be interested in.

      0
  52. 88

    Reliable Cross-Browser Testing, Part 2: Mozilla Firefox

    Utilu Mozilla Firefox Collection: http://utilu.com/UtiluMFC/

    Utilu Mozilla Firefox Collection contains multiple standalone versions of Mozilla Firefox plus Firebug and Web Developer add-ons.

    0
  53. 89

    As a rule of thumb I only test as far back as the current version of the browser and 2 versions prior. So let’s say IE 9 is the latest. I would test IE 9, 8 and 7. This applies to other browsers as well.

    Theres no point of testing IE 6. Microsoft doesn’t support it and is actively encouraging users to upgrade, google ditched it before then and it’s just completely broken. It’s time to move the web forward, W3C standards are the way to go!

    0
    • 90

      Agreed. I mentioned this in another comment but IE6 is really just covered for the sake of being thorough as at Aol, IE7 is the baseline we work from.

      My personal rule of thumb when it comes to testing other browsers is current -2 (and in the case of browsers like Firefox and Chrome with accelerated release cycles/dev snapshots), current +2 as well.

      0
  54. 91

    Good stuff Addy. I was wondering if you’ve had any experience with Multi-Browser Viewer (multibrowserviewer.com/)?

    0
    • 92

      Multi-browser viewer is an excellent solution. It provides all of the virtualization without the pain. I’ve used it for over a year and love it.

      0
  55. 93

    So which of the 3 alternatives you guys use at AOL?

    0
  56. 94

    so which of the 3 alternative you guys use at AOL?

    0
    • 95

      The team I’m on use the first option outlined, however I know of others that use option 3.

      0
  57. 96

    I have IE6-9 thinApped on my box — it makes really easy to test against different versions and you don’t need to maintain a separate vm layer.

    I run the same solution on my MBP but in a w7 vm (under parallels)

    This setup is the best and least messy that I’ve tested
    so far.

    0
  58. 97

    Take a look at Microsoft Expression Web 4 Super Preview. They have a new Remote Browsers service which is still in beta. They have Internet Explorer 6 – 9 ; Safari 4-5, Firefox 3-5; and Chrome.

    Also, there is an interaction mode.

    0
  59. 98

    All this just to test for the worst browser family available. Don’t get me wrong this is a fantastic article. I’ve struggled with proper IE testing like everyone else and I’ve tried several of the methods mentioned above. As I was reading this however I couldn’t fight the feeling of an overwhelming cost/benefit disparity in doing all this. I’ve reached the point where I test everything in all the latest browsers and if the browser type used to visit my site is an old version of IE I display an information box with upgrade paths for my users.

    At the end of the day developing for old IE browsers is simply too time consuming and most clients (who don’t understand why this is time consuming) aren’t willing to pay enough money for a site to make that extra testing worthwhile. Naturally if a client insists I negotiate a fair price or I don’t take the job but … oh man I just hate IE so much.

    0
  60. 99
  61. 100

    Thank you so much for this article.

    0
  62. 101

    After a lot of experimenting and comparing I can only really recommend the “one machine/vm for every version of IE” that needs to be tested. Maybe with a quad core it might be possible to run up to 3 vms simultaneously… Don’t throw away your old machines!

    0
  63. 102

    Awesome post.

    A few additions and corrections though:

    1) Third-party tools: make sure to keep Spoon (formerly Xenocode) in your bookmarks, watchlist, and test-drives: http://spoon.net/Browsers — former Xenocode allowed you to virtualize any kind of browser/application as a local, stand-alone, single-file executable without installation through some terrific black magic. Xenocode redirects to Spoon now, and looks like their goal is to move that magic to the next level…

    2) Option #1: VMware provides VMware Player on http://www.vmware.com/products/player/ for free. It is perfectly sufficient and suitable for cross-browser testing (and more). It has many more features than Microsoft’s Virtual PC, and also comes with a feature called “Unity” that allows you to run an application from inside the VM in your parent system.

    3) Option #3: RDP natively supports to access and share sessions with multiple users, but the feature is not available in desktop systems.

    Anyway, thanks for writing up this summary! Wish I would’ve known about these options and pitfalls years ago… :)

    P.S.: *Love* the final “in part 2 [...] we’ll look at cross-browser testing MODERN BROWSERS.” LOL! So true… :-D

    0
  64. 103

    I’m just trying this new tool to test in different browsers: http://ow.ly/6xz6D I find it very useful.

    0
  65. 104

    You have suggested to use XP for IE7 testing with IECollection in “Option 1″, and to use Vista for the same task in “Option 2″.

    Can you clarify what is the difference?

    0
  66. 105

    Win7 comes with multiple winxp instances, unfortunately i can not run all at the same time, even I have 8G RAM, the xp multiple instances just takes all the memory.

    0
  67. 106

    Very broad and honest article. I currently use VMware Fusion virtualising Windows 7 with IE Collection. It does the trick.

    Thanks for letting us know about all of the other options out there!

    0
  68. 107

    i cannot get IE7 and IE8 to work on windows 7 running on parallels. it simply is greyed-out in the installation window. so i cannot tick their boxes. pretty useless for me then.

    0
    • 108

      Same Problem here. Can’t check IE8.

      0
      • 109

        I also had this same problem with IE Collection. Installed Windows7 Pro through Virtual Box on a Mac. Downloaded IE Collection, but I could only install IE6. IE7 (which I wasn’t expecting to work because of Addy’s comments) and IE8 were greyed out in the installation window. And on top of that, both versions of IE6 crash when I try to use them. I can’t find an explanation on the web as to why IE8 won’t install.

        btw – great article.

        0
        • 110

          yeah, i’m in exactly the same camp as you.

          running Win 7 on parallels, can’t install 7 or 8, and when I run 6 it doesn’t work properly at all. I remembered having tried IE Collection a few years back and deciding it wasn’t quite up to the task, now i think I remember why…

          0
          • 111

            Multi-Browser Viewer has a standalone versions of all IE versions, including IE8 and is included in the free trial and it runs in parallels.

            0
    • 112

      Yep, 3 months later, and this is still the case. I’m running Win 7 in VMware and IE Collection has proven fruitless. I think it’s fairly safe to say this isn’t an option for cross-browser testing anymore.

      0
  69. 113

    Great article!! It was extremely helpful, and helped solve some of the problems that I have been having with IE 9! Thank you for writing/sharing this!

    0
  70. 114

    I think IETester gets an unjustifiable amount of bad-press, when, for IE6 emulation at least, it’s an easy to set up, resource-light and mostly accurate solution. Yes, there are some inconsistencies with the real IE6 browser, but these are actually few and far between; occasionally annoying, but certainly not widespread enough to give it a resounding thumbs down

    Within any team with a dedicated tester as long as *they* have a more robust copy of IE6 (eg on a VM) then IETester suffices the vast majority of the time for your developers. In about 4 years of front-end development I’ve only come across an IE6 bug not-reproducible in IETester (or the opposite, a false bug thrown in IETester) a handful of times.

    PS – one tiny correction/addition – IE8 also has multiple browser modes, and reproducing IE7 from within IE8 generally works a lot better than trying to do it from within IE9

    0
  71. 115

    Awesome article! Started looking at cross-browser tests last week and this was a priceless help for me. Looking forward to part 2.
    Regarding cross-browser script tests, do you have any recommendations?

    Thanks,
    Ricardo

    0
  72. 116

    Different VM instances for each verion of IE 6,7,8,9…
    Then FF 3.6..
    =) the way to go!

    0
  73. 117

    Competition for browser market share between Chrome and IE is going to be more fierce. The advent of mobile computing will add more complexity. In this situation you want an end to end cross-browser solution. eggPlant from TestPlant is a perfect tool to try.

    0
  74. 118

    This article is excellent and very detailed, thank you!
    Will give spoon.net a try after reading about it in the comments!

    0
  75. 119

    IE 7 and IE 8 are greyed out in window 7. make this software rather pointless.

    0
  76. 120

    I believe this site contains some very excellent information for everyone . “The test of every religious, political, or educational system is the man that it forms.” by Henri Frdric Amiel.

    0
  77. 121

    Hi, I have been using Virtualbox to perform all my IE testing from like 2 years now. That includes all the mayor browsers like firefox, chrome, safari and opera.

    But recently, one guy in my company started to said than IExplorer acts different from a virtual machine than a real pc with Windows.

    I want to know if this is even real. I mean real facts, not just some-rare-cases that all we got sometimes around IE. We can’t trust 100% on what we’re seeing on IExplorer running on virtuals vs real machines?

    Thanks in advance.
    - Mario

    0
  78. 122

    Just want to mention that even if VM are probably the way to go, there are some situations where you cannot use that. At this time I’m working on a big application where every developper has already a virtual machine for local development. Unless you hace a monster computer, you won’t be able to run two virtual machines :(

    0
  79. 123

    All you need is VirtualBox software (free) and a developer copy of Windows8 (free). Follow this tutorial. Worked for me. https://centroy.com/blog/testing-internet-explorer-mac-its-actually-quite-easy

    0

↑ Back to top