Cross-Browser Testing: A Detailed Review Of Tools And Services

Advertisement

As you probably know, cross-browser testing is an important part of any developer’s routine. As the number of browsers increase, and they certainly have in recent years, the need for automatic tools that can assist us in the process becomes ever greater. In this article, we present an overview of different cross-browser testing applications and services. Surely, you are already familiar with some of them, and you may have even stumbled across another overview article, but this one takes a different approach.

This is not just a list of available tools, but rather a comprehensive analysis based on my experience with each of them. For the impatient among you, a summary table is at the end summarizing key metrics and unique features for each service. But if you’re interested in my personal experience with these tools, then read on.

Probably the most important metric of these services is the capture delay, which I measured for the URL stackoverflow1, with the following browsers enabled: Firefox, IE, Chrome and Safari.

BrowserShots

BrowserShots2 is the oldest and best known free online multi-browser screenshot service. It supports the largest number of browsers: a total of 61 different browser versions and operating systems, which is great, but I can hardly imagine anyone wanting to test their website under Kazahakase 0.5 running on BSD Unix. Feature-wise, it allows you to enable and disable Javascript, Java and Flash and change the screen size. I find the latter very useful, especially nowadays when one has to take into account smartphone browsers with non-standard resolutions.

Screenshot3

The interface is not very user-friendly. Selecting the browsers and options you want takes time, and because it is a Web service you have to do it over every time you want to take a screenshot. When (and if) you finally get your screenshots, there is no easy way to compare different captures in order to find rendering inconsistencies. HTTP redirect is not fully automated: BrowserShots displays the URL you are being redirected to, but you have to start the screenshot again manually.

The biggest disadvantage of BrowserShots—which, in my opinion, makes it practically unusable for a professional developer — is the response time. In our test scenario, it was more than 45 minutes. Note that a screenshot expires in 30 minutes, unless you manually extend it. As you can see from the shot below, BrowserShots has serious bugs with scrolling (see MSIE 8.0 screenshot) and at least one browser screenshot failed, even though it said the operation was successful.

4

Conclusion

Unless you need only a single test on a particular browser, this service is not for you. Even then, by the way, it would probably take less time to install that browser, test the page and then uninstall it.

Unique features: None.

Disadvantages: Painfully slow.

BrowserCam

BrowserCam5 is another well-known screenshot service. Unlike BrowserShots, this is a commercial service. The cheapest plan cost $159.80 a year and provides access for five users. The interface is nice. It allows you to create a project and specify the URL and browsers you want to capture, so that you do not have to do it all over again to re-test the page. But because it is a non-AJAX Web-based interface, its response time is not comparable to that of a native application, which is a bit annoying.

Screenshot6

Browser support is slightly more limited than that of BrowserShots, but it is good enough for practical purposes; it supports multiple versions of IE, Firefox, Safari, Opera and Chrome, as well as some older browsers on OS X, Linux and multiple versions of Windows. Capture speed is decent: it took about two minutes to take a screenshot of our test scenario.

BrowserCam supports multiple resolutions and has window and full-page capture, which means scroll bar support. Another nice feature is mobile device capture: it supports Blackberry, iPhone, Android and Windows Mobile devices. Note that mobile capture support is not part of the browser capture plan and costs $999.95 extra annually. It also has an email capture service, which in my opinion is of limited use, and remote access, which can be useful for troubleshooting rendering inconsistencies that are detected from a screen capture. Both services cost extra. The screenshot below is of a BrowserCam results window.

Remote access packages allow you to connect using VNC to your choice of Linux, Windows and Mac machines with different browser versions. This can be a good option for debugging on hardware that you do not have, such as Mac. But the price of $499.95 a year is not far from the price of Mac mini, and because the VNC protocol is not terribly efficient, extensive remote debugging via VNC can be daunting.

7

Conclusion

A very good professional service with advanced features and thoughtful interface.

Unique features: Mobile device support, remote access.

Disadvantages: Expensive.

Adobe BrowserLab

BrowserLab8 is a new offering from Adobe and was previously known as Meer-Meer. It is written in Flash and as such has the advantage of being cross-platform compatible and of having the look, feel and (most importantly) response time of an application. It is currently offered free of charge in preview mode while Adobe “is monitoring the performance.” Because it will monitor it for more than one year, one wonders whether it has other reasons for this. According to Adobe, it will charge $10 to $20 per month for this service starting in 2011.

Screenshot9

The interface is attractive, polished and easy to use, as you can see from the screenshot below. You can inspect captures one by one or view two captures side by side, which is more useful. The much lauded “onion skin” option is not very practical: most of the time, browsers will not render a page identically pixel by pixel, but the page might still look the same.

Browser support is modest compared to the competition. At the time of writing, BrowserLab supports only Chrome, Firefox, IE and Safari: a total of 12 browsers and OS version combinations. But it looks like the quality of the product is still at beta level; in two captures, it actually cut the image horizontally. Scroll bar support is buggy, too.

Screenshot speed is very good. Our test scenario did it in less than one minute.

10

Conclusion

A very nice interface, and free till the end of 2010.

Unique features: None.

Disadvantages: Modest browser support, minor bugs.

Microsoft Expression Web SuperPreview

SuperPreview3411 is a new addition to Microsoft’s Expression Web WYSIWYG development environment. This is the standalone version, limited to Internet Explorer and available for download free of charge. Browser support is limited. The standalone version supports only IE 6, 7 and 8, while the full version has support for Firefox and Safari. The user experience, on the other hand, is very impressive.

Screenshot12

Because it is an application that runs on your PC, the response time and screenshot delay are among the best in class. In our test scenario, it loaded the website in a matter of seconds. Please note, though, that because SuperPreview works with only two browsers at a time and does not support Chrome, this test was not identical to that of other services.

SuperPreview cannot be purchased without the Expression Web, whose retail price is $149.

13

Conclusion

The interface is extremely easy to use, and the speed is incredible. But browser support is very limited, and being part of the Expression Web package, it’s almost unusable.

Unique features: None.

Disadvantages: Limited browser support, expensive.

BrowserSeal

BrowserSeal14 is a new tool. Similar to SuperPreview, it is an application rather than a Web service, and as such its capture speed is very good. BrowserSeal finished our test case in less than one minute. Browser support is quite broad. With multiple versions of IE, Firefox, Safari, Opera and Chrome, it pretty much covers every browser anyone would want to test. It has two good features that set it apart from the competition: standalone browser support and a command-line interface for automation scripting.

Screenshot15

All browsers supported by BrowserSeal can be launched manually, which means that once you’ve found a rendering inconsistency in a browser, you can actually run the problem browser and troubleshoot the issue. This is something most other services do not offer.

The price is very competitive, too: the standard version sells for $49. There is also an automation version with a command-line interface that lets you capture multiple URLs from a script or batch file. The interface is attractive and easy to use. The optional tabbed interface makes it easy to spot even the slightest rendering difference when switching from one capture to another.

16

Conclusion

A very nice tool, with comprehensive browser support. The interface is easy to use, the capture speed is great, and the price is competitive.

Unique features:: Comes with standalone versions for all major browsers; has command-line mode for automation scripts.

Disadvantages:: Runs on Windows only.

Litmus

Litmus3617 is another Web-based screenshot service. Its browser support is impressive, with 23 browser versions and operating system combinations, including IE, Firefox, Chrome, Safari, Opera, Flock, Camino, SeaMonkey and Netscape. Capture speed is okay but not comparable to that of native applications: our test took five minutes.

Screenshot18

The interface is clear and simple but lacks some features. For instance, there is no easy way to compare capture results. All you can do is view them one by one or download them to your PC. The app, though, does support projects, so you don’t have to enter URLs and change browser settings every time you want to take a screenshot, but this is pretty much all it does.

Litmus does not support scrolling; that is, it captures only the top of long pages, which is a major drawback. The price is a bit high for a service that has such basic features: a single-user license costs $588 annually.

19

Conclusion

Good browser support, and average capture speed, which is probably good enough for most users. But very few features.

Unique features:: None.

Disadvantages: Does not support scrolling, and lacks other standard features found in competing products.

Multi-Browser Viewer

Multi-Browser Viewer20 is an application but relies on a server farm for browser rendering; in other words, the application is just a graphical interface, so it is as easy to use as an application but suffers the delays of a typical Web-based service.

Screenshot21

One interesting feature is that it comes with standalone browsers that can be used for debugging. But note that these are not the same browsers used for screen capture. Multi-Browser Viewer has standalone browsers that can be used for debugging, and it has a rendering farm with many more browsers that can be used for screen capture.

Browser support is impressive, with 54 browser and OS version combinations (out of which 17 are available in standalone versions), including IE, Firefox, Chrome, Opera, Safari, Camino, Konqueror. The price is reasonable: a single-user license costs $129.95 annually.

Feature-wise, it does lag significantly behind the competition: there is no support for authentication or capture delay. Scroll bar support is buggy; in our test case, it worked for IE, Firefox and Safari, but not for Opera.

22

Conclusion

A good interface and impressive browser support.

Unique features: Standalone versions of some (but not all) browsers.

Disadvantages: Lacks many features of competing products, buggy scroll bar support, runs on Windows only.

Browsera

Browsera3823 is a Web-based screenshot service. Browser support is limited compared to that of most competitors: only IE, Firefox and Safari are supported. The standard plan costs $588 annually. The interface is attractive, fast and clean. You can conveniently organize your screenshot sessions into projects.

Screenshot24

Browsera supports authentication, scroll bars and page crawling (i.e. you can ask Browsera to crawl your website recursively and take a screenshot of every page). The screenshot response time is very fast for a Web-based service; it completed our test in three minutes.

25

Conclusion

A professional service with a good interface and interesting features, but limited browser support.

Unique features: Recursive website crawling.

Disadvantages: Limited browser support, expensive.

Browser Packs

If all you need is to test your website in specific browsers with and you are willing to perform the tests manually, there are a few free services and applications that could help:

At first glance, Spoon looks convenient because it is a Web service, which relieves you from having to install many browsers locally. But I had some stability problems with this service.

Meanwhile, both the IE Collection and BrowserSeal.BrowserPack (offered free of charge, separate from the BrowserSeal commercial screenshot service) work very reliably. I did not have any issues with browsers installed by these packs. The IE Collection has every IE version you could think of. BrowserSeal.BrowserPack, which relies on the IE Collection for IE support, also supports two Firefox, three Opera and two Safari versions.

Conclusion

The following table summarizes services that were tested and analyzed in the article. You can use the separate page for the full table30 for a better overview. I have included some metrics for each service to make it easier for you to choose the best one based on price, features and performance trade-offs.

Supported Browsers Capture speed Price (1 year) Interface Authentication Capture delay Scroll bars Special features
BrowserShots31 IE, Firefox, Chrome, Opera, Safari, Dilo, SeaMonkey, Minefield, Epiphany, Flock, Galeon, Konqueror, K-Meleon, Avant, Netscape, Shireteko, Kazehakase, Iceweasel 45 mins Free Bad No No No None
BrowserCam32 IE, Firefox, Chrome, Opera, Safari, Konqueror, Camino, Netscape, AOL 2 mins $999.95 Good Yes Yes Yes Mobile browsers support, remote access service
BrowserLab33 IE, Firefox, Chrome and Safari 1 min Free (till end of 2010) Good No Yes Buggy None
SuperPreview3411 IE, Firefox and Safari 1 min $149 Good No No Yes None
BrowserSeal35 IE, Firefox, Chrome, Opera and Safari 1 min $49 Good Yes Yes Yes Standalone browser versions, support for automation scripts
Litmus3617 IE, Firefox, Chrome, Opera, Safari, Flock, Camino, SeaMonkey, Netscape 5 mins $588 Basic Yes No No None
Multi Browser Viewer37 IE, Firefox, Chrome, Opera, Safari, Flock, SeaMonkey, Netscape, K-Meleon, Camino, Konqueror, Epiphany, Kazehakase 2 mins $129.95 Good No No Buggy Standalone browser versions
Browsera3823 IE, Firefox, Safari 3 mins $588 Good Yes No Yes Recursive crawling

Obviously, we have no clear winner. Each service has its advantages and disadvantages, and you are left to decide what is the best trade-off for your case. Professional developers would likely not use BrowserShots because of the unreasonably long response time. SuperPreview and Browsera are probably also impractical because of their very limited browser support.

BrowserLab will probably remain popular as long as it is free. Once Adobe starts charging about $20 per month for it, one would hardly have reason to use it, unless you worked in Dreamweaver, which has a BrowserLab extension, because there are much better alternatives.

When choosing a tool, one of the most important factors in your decision will be whether to use a Web service or application. Some people prefer Web-based tools because they do not require installation. Personally, I prefer applications, at least for the development tools that I use frequently. They generally have a better interface and faster response time; they never have outages, and they can be used to debug locally (i.e. on my hard drive or company intranet — although some Web-based services offer a workaround for this issue).

BrowserCam, BrowserSeal, Litmus and Multi-Browser Viewer are all very good choices. But they do vary significantly in price. If you need to test mobile browsers, BrowserCam is probably your only option. For everyone else, I would recommend either BrowserSeal or Multi-Browser Viewer; both come with standalone browser versions that are extremely important for testing. Unfortunately, both of them are Windows only, so Mac users will probably have to go with BrowserLab or BrowserCam. If automatic testing is important to you, then the BrowserSeal automation edition is your best bet.

(al)

Footnotes

  1. 1 http://www.stackoverflow.com
  2. 2 http://www.browsershots.org
  3. 3 http://www.browsershots.org
  4. 4 http://www.smashingmagazine.com/wp-content/uploads/2010/05/browsershots1.jpg
  5. 5 http://www.browsercam.com
  6. 6 http://www.browsercam.com
  7. 7 http://www.smashingmagazine.com/wp-content/uploads/2010/05/browsercam.jpg
  8. 8 http://browserlab.adobe.com
  9. 9 http://browserlab.adobe.com
  10. 10 http://www.smashingmagazine.com/wp-content/uploads/2010/05/browserlab1.jpg
  11. 11 http://expression.microsoft.com/en-us/cc136529.aspx
  12. 12 http://expression.microsoft.com/en-us/cc136529.aspx
  13. 13 http://www.smashingmagazine.com/wp-content/uploads/2010/05/superpreview1.jpg
  14. 14 http://www.browserseal.com
  15. 15 http://www.browserseal.com
  16. 16 http://www.smashingmagazine.com/wp-content/uploads/2010/05/browserseal1.jpg
  17. 17 http://litmusapp.com/
  18. 18 http://litmusapp.com/
  19. 19 http://www.smashingmagazine.com/wp-content/uploads/2010/05/litmusapp1.jpg
  20. 20 http://www.multibrowserviewer.com
  21. 21 http://www.multibrowserviewer.com
  22. 22 http://www.smashingmagazine.com/wp-content/uploads/2010/05/multibrowserviewer1.jpg
  23. 23 http://www.browsera.com/
  24. 24 http://www.browsera.com/
  25. 25 http://www.smashingmagazine.com/wp-content/uploads/2010/05/browsera1.jpg
  26. 26 http://spoon.net/browsers/
  27. 27 http://www.browserseal.com/?option=com_content&view=article&id=35
  28. 28 http://utilu.com/IECollection/
  29. 29 http://www.my-debugbar.com/wiki/IETester/HomePage
  30. 30 http://www.smashingmagazine.com/wp-content/uploads/2010/06/cross-browser-testing-table.html
  31. 31 http://www.browsershots.org/
  32. 32 http://www.browsercam.com/
  33. 33 http://browserlab.adobe.com/
  34. 34 http://expression.microsoft.com/en-us/cc136529.aspx
  35. 35 http://www.browserseal.com/
  36. 36 http://litmusapp.com/
  37. 37 http://www.multibrowserviewer.com/
  38. 38 http://www.browsera.com/

↑ Back to top Tweet itShare on Facebook

Demiurg is a web developer, consultant and author with over 10 years' experience, currently working on a new stealth-mode internet project.

Advertising
  1. 1

    Nothing that claims to run more than one version of IE one the same machine can ever be entirely trusted.

    0
  2. 102

    It’s a totally different type of testing but using Selenium can have a huge learning curve but it makes things quick/easy/automated.

    I ran across this site and am currently trying them out for web-sourced screenshots: https://saucelabs.com/

    You can get back a textfile that says whether it passed or not and see a video of it all happening. Very, very nice =)

    0
  3. 203

    Is it possible to install mac. as virtual OS ?

    0
  4. 304

    You can only run MacOS X on a Mac legally.

    0
  5. 405

    Mohammad Ashour

    June 20, 2010 1:10 am

    I think some of these services are catering to developers who want to test every browser under the sun. While I understand that, because it gives you control over your site to a great degree, it’s also very impractical. Look at Yahoo! They guarantee support for a small subset of available browsers for CSS and JavaScript: http://developer.yahoo.com/yui/articles/gbs/

    I like the Yahoo! approach. I don’t think cross-browser means supporting every browser. I think it means starting with a common-sense subset, going live, and then analyzing the traffic on the site to see what browsers people actually use when they visit. Then you have real data and can adjust your browser support list and retest.

    0
  6. 506

    Interesting. Thank you.

    0
  7. 607

    I could not agree more. I believe that automatic testing tools with “good enough” browser support, such as BrowserSeal, are the way to go.

    0
  8. 708

    Excellent summary of the field and good comparison of the various services. I’d love to hear what you think about Mogotest (http://mogotest.com/). We’re a new player in the space and offer several unique features.

    Thanks,
    Kevin

    0
  9. 809

    One more screenshot service!? Jesus…

    Are you really sure that there is market for all these tools :) ?

    0
  10. 910

    Always used a WM for the task and always preferred tools over services. Now it seems that a few more cool tools have came to the picture and I’d definitely try them. I downloaded a Browserseal trial because never heard about them and it looks very promising. It manages standalone installations of real browsers which is very smart and reliable solution. Reliability was my concern. Does it look the same on a REAL browser? Does a service provide same results after upgrade? Here you have them all, standalone and real… I’ll play more before I have final conclusions but anyways, it’s very nice to hear about the progress in the area. And a big-big 10x to the author for great review!

    0
  11. 1011

    Great article! Thanks

    0
  12. 1112

    Really a unique & comprehensive research in this growing field. Got me thinking.. thanks.

    0
  13. 1213

    This is “normal”. I also experienced outages from time to time with BrowserLab.

    0
  14. 1314

    Thanks. Will check it.

    0
  15. 1415

    You are right in that there are currently no good tools for testing complex staff such as hover states on multiple browsers.

    As for testing local sites, some of the above tools such as BrowserSeal can do the job

    0
  16. 1516

    Very promising tool!
    May be the same work must be done for smart phones.

    0
  17. 1617

    Thank you, very usefull!

    0
  18. 1718

    I understand that it is based on Selenium. Is it just a “web interface” to Selenium Grid of different browsers and operating systems or is there more than that ?

    0
  19. 1819

    This is something I could not figure myself too. I can hardly believe that there is market for so many services and tools.

    0
  20. 1920

    While we do make use of Selenium Grid (I’m currently the maintainer of that project), there’s much more to it than that. We provide screenshot comparison tools over full canvas captures, so you can quickly see how two pages look relative to one another. Beyond the screenshot component we detect a wide variety of problems as we process your site such as redirect loops, 404s, 500s, empty content documents, and CSS & HTML validation results. We’ve also customized the CSS validator to handle commonly used, yet technically invalid, browser extensions so you can focus on true errors.

    And we have an API available for whatever integration you might need. We will shortly be releasing an open-sourced Firebug plugin that uses the API and will provide you with renders in different browsers as you interactively develop.

    We’re actively working on a bunch of other things too, like client approval tools. But, I’ve begun rambling, so sorry for that. If you want to check it out, feel free to signup for the alpha at http://mogotest.com/ or email me at kevin@mogotest.com. I’d love to hear any feedback, good or bad.

    Thanks,
    Kevin

    0
  21. 2021

    Thanks for covering BrowserSeal. Please note that we just released a new 1.6 version which improves stability and solves some issues, mainly for IE6, IE7 and IE8. More info is available on our web site.

    0
  22. 2122

    A very good collection of cross browser testing websites.
    I personally have only used browsershots which is very useful as it tests most browsers, however can be very tedious and time consuming to use.
    Also I’ve found that using browsershots on pc and mac give different resulting screenshots which isn’t helpful.
    It would be helpful if there was a browser testing application that worked well on mac computers.

    0
  23. 2223

    Very informative article, thank you.

    0
  24. 2324

    I got inspired and created BrowserBox, an appliance with 22 bundled browsers you can download and run in VMWare Player, VirtualBox or as a Live CD.

    0
  25. 2425

    Why all the emphasis on screen testing. Although I found the onion skin feature of multibrowserviewer great the real test is to use the site in the different browsers.

    For response, nothing beets running the browser and I love how multi-browser viewer didn’t clog my machine yet allowed me to test 23 browsers and 5 mobile browsers. Especially lived the fact ther were all just stand alone exe’s. No all,s registry entries etc. Using the iPhone emulator to develop a mobile site was a breeze :-)

    0
  26. 2526

    Problem with the IE pack is that I found the rendering is not the same as the default installed IE’s. This goes for the IE pack used by Browserseal as well. The only solution above that has actual standalone browsers is Multi-Browser Viewer.

    0
  27. 2627

    Nice article, thank you. Never heard about many of these services. Will check BrowserSeal and maybe some others

    0
  28. 2728

    Install IE9 and press F12. Then you can select browser mode and rendering mode from IE7 upwards. On the same machine I have Chrome, Safari and Firefox. I have IE6 and an older version of Firefox running in Virtual PC which is free.

    0
  29. 2829

    Would be great to have an update on this article. A few of the tools now seem to have different features or have changed direction (litmus for example seems to be an email testing suite)

    0
  30. 2930

    Is there any good mobile browser testing tool available??

    Please send the response at v ivekgarg 79 @ gmail.com

    0
  31. 3031

    If you are interested in mobile browser testing you may want to take a look at BrowserSeal 2 (this version was released after this article was published), which now comes with Opera Mini support, including Opera Mini version for PC.

    0
  32. 3132

    spoon.net has stopped its service for IE with the request of Microsoft. Now we can preview our html in ie6.

    0
  33. 3233

    spoon.net has stopped its service for IE with the request of Microsoft. Now we can not preview our html in ie6.

    0
  34. 3334

    One option you hadn’t mentioned is a Windows package called Utilu IE Collection. It allows you to run stand-alone, independent, versions of IE5,6,7,8 & even Beta-9.

    For more info I’d suggest checking out the review I’ve written on it:

    http://www.kevinleary.net/testing-websites-multiple-versions-internet-explorer-ie6-ie7-ie8/

    Depending on your situation it may or may not be the best browser testing option for you.

    0
  35. 3435

    You can also try this: http://tredosoft.com/Multiple_IE

    0
  36. 3536

    I’m looking for an affordable solution for cross browser testing as well. Via CrossBrowserTesting.com, are you able to test internal test sites? Like local host or sites only available within your domain?

    Thanks,
    John

    0
  37. 3637

    Did you actually read the article ? IE collection is mentioned…

    0
  38. 3738

    I have alot had great luck working with crossbrowsertesting.com. Now they support mobile browsers.

    0
  39. 3839
  40. 3940

    http://browshot/ is a new screenshot service with 15+ mobile browsers (iPhone, iPad, Nook, etc.) and desktop browsers. You can try out the service with a free account.

    0
  41. 4041

    Alas, BrowserCam is discontinued as of February 1, 2013.
    http://www.browsercam.com/important-message/

    0
  42. 4142

    Quite a few of these screenshot services are no longer around, however I have used GrabzIt (http://grabz.it) to provide screenshots in the past.

    0
  43. 4243

    Spoon is nice, but unfortunately it does not work flawlessly on all platforms.

    0
  44. 4344

    Gomez=BrowserCam

    0
  45. 4445

    Selenium IDE does not need that step of a learning curve.
    The bigger problem with Selenium is browser support. It pretty much works reliably with IE and FF only.

    0
  46. 4546

    Check BrowserCam device capture service

    0

↑ Back to top