Menu Search
Jump to the content X X
Smashing Conf Barcelona

You know, we use ad-blockers as well. We gotta keep those servers running though. Did you know that we publish useful books and run friendly conferences — crafted for pros like yourself? E.g. our upcoming SmashingConf Barcelona, dedicated to smart front-end techniques and design patterns.

Prioritizing Devices: Testing And Responsive Web Design

My Android Galaxy smartphone is so sweet. It plays games, has a lovely screen and lets me check all of my favorite websites while I’m commuting to and from work. And my new iPad is even better; it’s all I use at home when I’m relaxing in the living room, cooking in the kitchen or toileting on the toilet. As a consumer of electronic gadgets, I’m happier than Angelina Jolie in an orphanage with all of the devices with which I can use to access the Internet.

As a developer, I hate it.

Have you seen how many browsers1 and devices we have to test now? I remember when Internet Explorer (IE) 8 came out and we were annoyed that we had to start testing six browsers. Now, we’re testing at least 15! Back then, when every home had broadband and before anyone had a smartphone, we were living in the Golden Age of web development. We never knew how easy our jobs were.

Further Reading on SmashingMag: Link

Because of all the things we have to support now, testing has become really, really, really difficult and also super-expensive. But imagine having to support and test all of these browser and device combinations while also responding to the news. When you work in a busy environment like the BBC News, where much of what you make is pegged to a news story or breaking event and everything is made using responsive web design, the job becomes really tricky and occasionally very stressful.

In the Golden Age of web development, we had to support only five desktop browsers on a broadband connection.

Unfortunately, we can never move the deadlines that news events present to us. We can’t contact the UK government and ask them to postpone an election by a few more days while we get the new responsive timeline carousel working in IE 8. And during the flooding in the UK earlier this year, we couldn’t contact God to ask him to pause the heavy rain long enough for my team to get the interactive flooding map working in Android 2.3.

Before responsive web design, we had to test five web browsers on a desktop computer. Now with responsive web design, we have at least 15 browsers working on a myriad of different-sized devices, with many different input types, multiple pixel resolutions and hugely varying connection speeds.

Your boss might have once dictated your testing strategy as such: “This website has to work perfectly in all of the browsers on this list.” If you tried to carry out that strategy with the vast number of browser and device combinations that now visit your website, you would quickly spend a fortune and get sucked into a testing black hole.

There must be a better way to deal with the problem that responsive design has created for testing. So, let’s step back from this testing singularity for a moment and think about the problem.

In this article, we’ll devise a testing strategy so that you don’t have to test every device every time you want to update a live website. I’ll draw on my experience as a developer working on the BBC News’ website.

Devising A New Testing Strategy Link

With unlimited time, money and human resources, you’d be able to test your entire website on every device, every day of the week. In the real world, even large organizations like the BBC don’t have the resources to attempt this perfect solution.

Nevertheless, as developers, we’d expect JavaScript that works in Chrome to probably work in Safari or Firefox, too. We’d also be confident that the majority of our CSS would render correctly. However, we’d be less certain that a responsive layout would work perfectly on multiple devices. And when we test the layout on an old Nokia device, we’d be more nervous than a startup founder about to meet a venture capitalist.

We can take advantage of similarities between browsers and devices. If some code works on a Samsung Galaxy S4 running Chrome, then it probably also works on an Android tablet or iPhone 5S. If IE 9 throws no JavaScript errors, then we’d be able to test IE 10 and 11 with much more confidence. But some browsers are so old and wicked and unlike anything else we have to deal with — yes, IE 8, I’m looking at you — that we should always test on them.

IE 8 is throwing more errors on this page than Miley Cyrus manages to twerk in a concert.

While you should try to support as large a proportion of your audience as possible, your analytics will show that a small group of browsers and devices make up the majority of your traffic, while the rest are a long tail of obscure browsers and devices. Take advantage of this.

By prioritizing which devices and browsers you test first, you can maximize your testing time. Don’t think of testing as a single task; break it up into groups of browsers according to how important they are to you. You could — dare I say — take shortcuts. Do 50% of your browser testing and then release, knowing that the majority of your users won’t see bugs, and then test and fix the long tail.

This new strategy could be carried out in different ways:

  • Making a major new version of your website? Test all browser groups.
  • Making a minor change that needs to be released very quickly? Test your primary and secondary browser groups, release, and then test the rest and release again if need be.

In the industry, running a segment of tests is called “smoke testing.” Smoke testing is a quick process that gives you confidence that you haven’t broken anything fundamental in your product. By sometimes choosing to test only your primary group of browsers, you’re effectively smoke testing for certain browser and device combinations.

Every Website Is Different Link

Before you present the top of this article to your boss and start testing only iPhones, consider that every website is different. Websites have different objectives and audiences and, thus, will be visited by different browser and device combinations.

With any new strategy, first consider the audience of your product:

  • Age and class
    Young people and less affluent people are more likely to use non-desktop devices. As a developer who’s been in the industry for 15 years, I struggle to accept that my desktop-centric mental model of the web is now in a minority.
  • Global distribution
    Different form factors are popular in different regions of the world. High-end smartphones are most popular in the West. “Phablets” (i.e. really big phones) are popular in the East because the larger screen size makes it easier to type text messages with Eastern character sets. Phones with long battery life are popular in rural India and Africa.
  • Engagement time
    In the UK, desktop browsers now make up less than 50% of a typical website’s traffic. This number drops dramatically on the weekend and rises dramatically during workday lunchtime (especially for legacy versions of IE installed in offices).

Testing All The Things Link

Setting up this photo caused a sudden spike in traffic from old Nokia devices to our website.

The point of grouping is to get the most out of testing as few devices as possible. So, for your first group, test as many different properties as possible. I call this a “test matrix”:

  • Screen size
    Designing responsively means being agnostic to device type. So, get a good idea of how your website works on a wide range of screen sizes.
  • Connection speed
    Treat connection speed the same way. Unfortunately, there is no correlation between screen size and connection speed. While a given device might be able to connect to Wi-Fi as well as to various mobile connection types that offer different Internet speeds, mobile devices generally limit the number of concurrent downloads to two. This means that even when an iPhone has a Wi-Fi connection, it still won’t be as fast as a MacBook. You’ll have to account for this.
  • Pixel density
    Pixel density refers to the number of pixels that make up a screen. The higher the pixel density, the clearer the picture on the screen will be. Differences in pixel density can cause issues that are subtle and hard to debug. The biggest issue I’ve come across is browsers that calculate subpixel values differently (which happens when widths are defined with a floating point percentage, such as 33.3%), thus breaking the layout.
  • Interaction style
    Make sure to cover not only mouse-and-keyboard interaction and touch interaction, but also D-pad styles of interaction, which happens with the arrow keys on a phone keyboard or the “nipple” on a BlackBerry (honestly, it’s called a “nipple” — ask your mum if you don’t believe me). However, don’t prioritize D-pad interaction unless Nokia and BlackBerry phones make up a significant share of your traffic.
  • Similarity to other browsers
    If browser A is very similar to browser B, and browser A passes your test, then browser B has a lower chance of causing serious problems than other browsers. So, you can lower the priority of browser B and expect to deal with only minor layout issues later on. If browser C has a smaller user base than browser B but is much older and has more differences, then you should probably test browser C earlier in the development cycle to catch issues that are harder to solve than minor layout bugs.
  • Browser rendering mode
    Browsers have two ways of rendering a page:

    • the standard client-side rendering that we are all used to,
    • proxy rendering.

A proxy browser sends the requested URL to a server, which downloads and renders the page before serving an image of the page to the user. A proxy browser renders much faster than a typical web browser and so is popular among users who are sensitive about their data plans, but they come at a cost.

Because the user is looking at a screenshot of a web page, the design might not work quite as intended. While proxy browsers might sound like a niche case and not worth prioritizing, note that Nokia’s Ovi browser, Opera Mini and the Chrome mobile browser either are proxy browsers or have proxy rendering modes.

Understand Your Testing Options Link

We have quite a few options for testing in the industry now: emulators, virtual machines, testing services like BrowserStack7 and grassroots community efforts like Open Device Lab23168. While testing services are great at showing what your website looks like, they will never be as good as testing your code on actual devices. Emulators and testing services are more efficient than buying and maintaining your own device lab, and they really help you to understand how your code behaves on different platforms, but they don’t tell you everything you need to know about the UX.

Are loading times fast enough? Are the hit sizes big enough? Will the user’s hand get in the way of the screen and keep them from noticing a change? Does the time between pressing the screen and the resulting change feel adequate?

BrowserStack is great for seeing how your website renders across many devices, but it doesn’t give you any haptic feedback.

Testing services are good for finding logical issues with your code — JavaScript errors, timeouts, etc. — but the coverage is only as good as the test that you run. The services suffer from a positive bias, in that they are good for asserting that stuff works. They won’t tell you whether something feels wrong or looks strange. I always suggest eyeballs for testing; unfortunately, this doesn’t scale to large products, especially if you lack dedicated testers.

You need to make a judgment call, balancing the amount of testing required with the testing resources available. Can you split up automated and manual testing, focusing the latter on new features? Are you continually testing something that never changes? If you test frequently, could you test a proportion of each part of the product or randomly choose features to test?

Group Devices By Priority Link

Now that we’ve gotten the theory out of the way, let’s look at an example in practice. The device groups defined below are specific to the statistics for the BBC News, so adapt the list to your website according to the advice in the section above “Every Website Is Different.”

Group 1 Link

Group 1 is essentially the Chrome, Safari and Android browsers on different devices and the desktop (“desktop” here meaning both Mac and Windows). Group 1 generally covers just over 50% of an audience for a typical website and is the absolute minimum amount of testing I would recommend before any kind of release.

I strongly recommend that you purchase the following devices:

Device/Virtual machine/Emulator Browser Screen size Pixel density Interaction style Connection speed Browser similarity Rendering mode
your development machine Chrome latest large standard mouse/keyboard broadband other modern browsers typical
Samsung Android 4.* device Chrome latest small high touch broadband/mobile N/A typical
Apple device running iOS 7 Safari 6 small/medium high touch broadband/mobile N/A typical
Apple device running iOS 6 Safari 5 small/medium high touch broadband/mobile N/A typical
Samsung Android 4.* device Android browser small high touch broadband/mobile N/A typical
any Android 2.3 device Android browser very small standard touch broadband/mobile N/A typical

(Large view10)

This could easily cost you the best part of $2,500. I am really sorry about this, but I’m guessing that because you read Smashing Magazine, you’re a technologist, so you probably already have one of these devices in your pocket.

Android devices are much cheaper than iOS devices anyway, so spending some of your budget on two Android devices is worthwhile. These four devices plus Chrome on the desktop should be your primary testing group. Chrome is the most popular desktop browser — fortunately, too, because most developers prefer to work in it.

You’ll need to test both Chrome and Android’s default browser on a Samsung Galaxy device running Android 4.*. All top-end Android devices (made by HTC, Sony and Motorola) except for Samsung now ship by default with Chrome. Only Samsung uses a slightly modified version of the browser that ships with the Android Open Source Initiative — that browser is a flavor of WebKit.

Recommended buys:

  • Buy an iPad and an iPhone. Make sure they have different versions of iOS so that you can test Safari 5 and 6 in different resolutions.
  • Buy a top-end Samsung Galaxy Android 4.* device and a cheap Android 2.3 device. This will give you an idea of how your code runs on Android devices with different types of processors and memory sizes.

Cheaper alternative:

  • Download Apple’s Xcode IDE and use the iPhone virtual machine. This will give you access to all versions of iOS on all devices, but then you’d have to buy a Mac, so that would still cost you the best part of $2,500 anyway.
  • Download the Android IDE (Java, yuck) and use the Android virtual machine. This isn’t highly recommended, though, because you wouldn’t be able to see your website in action on low-end hardware.

Group 2 Link

The good news about group 2 is that there are no devices to buy (if you develop on a Mac). The bad news is that it requires downloading a lot of virtual machines. Group 2 consists mostly of desktop browsers:

Device/Virtual machine/Emulator Browser Screen size Pixel density Interaction style Connection speed Browser similarity Rendering mode
virtual machine IE 8 large standard mouse/keyboard broadband N/A typical
your development machine Safari 6/7 (see note below) large standard mouse/keyboard broadband other modern browsers typical
your development machine Firefox latest large standard mouse/keyboard broadband other modern browsers typical
virtual machine IE 11 large standard mouse/keyboard broadband other modern browsers typical
Windows Phone 8 Emulator IE 10 mobile small standard touch broadband other modern browsers typical

(Large view11)

Note: Depending on the version of OS X your MacBook is running, you will most likely have Safari 6 or 7. Mavericks is the latest and most popular version of OS X, so prioritizing Safari 7 is recommended.

IE 11 and 8 can be tested using virtual machines downloaded from modern.IE1712. Windows Phone Emulator13 can be downloaded from Microsoft. Notice that I’ve not mentioned IE 9 or 10. You can test those browsers with virtual machines, too, but the majority of those users would have upgraded to IE 11 by now, so their priority is lower.

IE 8 is a special case because there is no upgrade path for users stuck on Windows XP, and, being so old, it will be the browser that gives you the most problems. The BBC News’ statistics show almost no IE 8 users on weekends or on weekday mornings or evenings, but we get a sudden spike of IE 8 activity during workday lunchtimes. This indicates that people now mostly use IE 8 when they are on their lunch break at work.

Recommended buys:

  • If your development machine is a Mac, then definitely include Safari 6 in this group. If not, then depending on the size of your audience, you might want to think about getting a Mac. Spending $2,500 for one device is expensive, and your budget would be better spent on other devices, so this is recommended only if you have a large budget for devices.

Cheaper alternative:

  • If your development team is PC-centric, then you could look to BrowserStack to test Safari 6. BrowserStack can be used instead of all of these virtual machines, too, and might be preferable if your machine is old. Virtual machines require at least 1 GB of memory to themselves to run smoothly. I’d recommend a minimum of 8 GB to run virtual machines because you’ll want a couple of them up at the same time.

Expensive alternative:

  • Buy a Nokia Windows Phone. Devices are always more preferable than emulators.

Group 3 Link

A third group consisting of Opera Mini and Nokia’s Ovi browser is also worth considering. These are “proxy browsers” that behave differently than standard browsers (see above for a description of how they work):

Device/Virtual machine/Emulator Browser Screen size Pixel density Interaction style Connection speed Browser similarity Rendering mode
Apple device running iOS 7 Opera Mini latest small/medium high touch broadband/mobile N/A proxy
any Nokia Ovi device Ovi browser very small very low D-pad/touch broadband/mobile N/A proxy

(Large view14)

Download and test Opera Mini on your iOS 7 device. If you have an international audience, then a Nokia Ovi device is a mandatory purchase; we have a Nokia C5 in our device drawer. While Nokia budget phones are not insanely popular in the UK or US, they are more popular in markets that value low prices, battery life and robustness (dropped your iPhone recently?). If you have a global audience, then definitely allocate part of your budget to a Nokia Ovi.

A few years ago, there was a clear distinction between the new range of smartphones that were appearing on the market and traditional “feature” phones. Unfortunately today, that distinction is not clear at all. The definition of a smartphone used to be that it supports Wi-Fi, has a modern browser, has a touchscreen and supports apps. The Nokia C5 has all of these features. Every phone is a smartphone now.

The big difference between the top and bottom ranges of phones is hardware capability. Manufacturers differentiate and compete by varying the amount of processing power and RAM available in each phone. This means that, while the difference in browser (read “software”) capabilities between, say, an HTC One and a Sony Experia Typo is nil, the difference in capacity to process your website and in reaction times to user input (read “hardware”) is huge. You can’t detect a device’s processing power or memory, so understanding how your website works with limited hardware is important.

Recommended buys:

  • Nokia Ovi phone

Group 4 Link

Group 4 is the long tail of browser support. The previous three groups had distinct characteristics (popularity, desktop, proxy rendering). Group 4’s characteristic is randomness! This is where your testing gets more varied and where the value of spending time on edge-case bugs decreases.

Device/Virtual machine/Emulator Browser Screen size Pixel density Interaction style Connection speed Browser similarity Rendering mode
virtual machine IE 10 large standard mouse/keyboard broadband other modern browsers typical
virtual machine IE 9 large standard mouse/keyboard broadband other modern browsers typical
your development machine Firefox desktop version 4 large standard mouse/keyboard broadband N/A typical
your development machine Opera desktop latest large standard mouse/keyboard broadband N/A typical
Any Android device Firefox Mobile latest small/medium high touch broadband/mobile other modern browsers typical
Any Android device Opera Mobile latest small/medium high touch broadband/mobile other modern browsers typical
BlackBerry device running BB10 default browser small high touch broadband/mobile other modern browsers typical
BlackBerry device running version 6 default browser small standard D-pad/touch broadband/mobile N/A typical
BrowserStack Safari desktop 6 large standard mouse/keyboard broadband other modern browsers typical
BrowserStack Safari desktop 5 large standard mouse/keyboard broadband other modern browsers typical

(Large view15)

  • Firefox desktop version 4 is the last version of Firefox that doesn’t auto-update; it’s the only old version of Firefox still trending in our statistics. It’s big enough for you to consider testing; fortunately, though, despite its age, it’s still a decent browser, so unless you are using advanced CSS3 features, you shouldn’t find many problems.
  • Firefox Mobile and Opera Mobile are excellent modern web browsers. They are a low priority because they render pages so well that you can expect your website to work in them with only minor layout issues by the time you make it down to the fourth group of browsers.
  • IE 9 and 10 are quickly disappearing from our statistics. We think they will be completely gone by the end of the year. Check your own statistics before deciding whether to support them.
  • BlackBerry devices are still worth testing. From OS 6, BlackBerry’s default browser has actually been very modern and feature-rich. The difference between this and other devices, though, is its form. Many BlackBerry owners use the trackball (D-pad) to move a cursor around the small landscape-oriented screen. This screen makes a mockery of people who design for the “fold,” because the fold on these devices is 200 pixels tall.
  • Safari 6 is the version of Safari that comes with Apple OS 10.7. While 10.7 isn’t the latest version of Apple’s operating system, it’s still relatively popular. Safari 6 is very similar to Safari 7, so it’s not worth the large investment required for an additional desktop machine.

Recommended buys:

  • As always, I recommend buying the devices if your budget allows. Buy a BlackBerry 10 and a BlackBerry Bold running OS 6. BlackBerry Bold should still be available on eBay, while the BlackBerry Z10 (running BlackBerry OS 10) should be available in most stores.

Cheaper alternative:

  • Test Safari 6 using BrowserStack. It’s still available to test via Mac OS X Lion.
  • Test Safari 5 using BrowserStack. It’s still available to test via Mac OS X Snow Leopard.
  • Unfortunately, BrowserStack doesn’t have BlackBerry devices to test; however, you can download and run simulators from BlackBerry’s website. You can run a Playbook simulator as a virtual machine on a PC or Mac, but the Playbook was never popular, so the simulator isn’t very useful. The phone emulators are much more useful; you can download exact model and OS versions, but they only work on a PC, so you will need a second machine if you are a Mac user.
  • An Open Device Lab23168 is an even cheaper alternative to BrowserStack, but the devices available to you will depend on what is available in the lab in your area. You will also need to account for the cost and convenience of getting to the device lab.

Expensive alternative:

  • Buy a Mac running 10.7 to test Safari 6. This might be the best choice if your development machine is a PC.
  • Buy a Mac running 10.6.8 to test Safari 5 — but only if you are seriously loaded!

Conclusion Link

Testing (or, to use the proper term, quality assurance) is a profession unto its own. As designers, we shouldn’t try to simplify the testing process; rather, we should understand the different testing strategies well enough to solve the challenges we now face in trying to support browsers.

Although the explosion of devices that started with the iPhone has dramatically increased the number of browsers we have to support, remember that a large proportion of your audience use only a handful of devices. The rest of your audience consumes the web via a massively diverse long tail of devices. Test your product on as many devices as possible, but also prioritize which devices to test in order to maximize your time.

Test hard, but test smart. Nobody can test everything all the time.

Further Reading Link

(al, ml)

Footnotes Link

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
  23. 23
  24. 24
  25. 25

↑ Back to top Tweet itShare on Facebook

Tom Maslen is a web developer working in the Visual Journalism team for BBC News. A keen Spicy Monster Munch fan, he's dynamite between the sheets and most people's best friend. Every Sunday he does his Nan's weekly big shop for her. He's an angel.

  1. 1

    I’m going to get in here early and give a shout out to the Responsinator –

    It’s a very cost effective tool for those who don’t have the finances.

    It doesn’t have all of the devices out there, but I use it on every build alongside an iPhone and iPad.

    • 2

      same thing can be accomplished with chrome dev tools emulator, but it is not alternative to testing on real devices.

      • 3

        You’re right, it can. But seeing the site in context helps a lot.

        It might not be as good as the real thing, but it’s a worthy tool to have in your locker. Not everybody has a drawer full of devices to use or a desk big enough to fit them all on.

      • 4

        Andrej Galuf

        July 14, 2014 4:45 pm

        You have no idea how many websites I’ve seen that were built using chrome tools and where the designer simply didn’t forsee the kinks of the real device. For instance, the site might look perfectly fine with that fixed header and footer in chrome, but when previewed on a landscape iphone, where you get an extra built-in header and footer while scrolling, it can turn into a disaster. There’s no way to predict where the device’s innate coding might turn your “carefully” selected font into chinese characters or that android 2.3 just won’t care about those percentage based radiuses.

        Much like Andy says below, it’s impossible to test everything. That’s why I’ve built myself a small set of devices and browsers that cover as wide a field as possible. Hopefully between them, bugs are relatively rare.

        • 5

          Man, you’re right. Might I add one thing though: it is infinitely better to test the site on one mobile device than on a bunch of emulators. And dare I say that most of us have at least that. So say, you can test it on Windows&Mac on your MacBook and iPhone. Android is a huge marketshare (depending on your visitors of course) but it will be somewhat similar to most webkit browsers (although there are some serious differences in how they render JS, CSS seems to be on par). Do this first, then the emulators. Depending on how many people you want to reach and your audience you can launch now and fix one by one (starting with the most popular ones) using a real lab.

          I’m pretty much rephrasing the article here and giving a word of support to testing on real hardware :)

  2. 6

    Andy Parker

    July 14, 2014 3:13 pm

    Nothing beats testing real physical input devices, whether it is a phone, tablet, watch, whatever because then you are testing how that device interacts as well as browser rendering. Issues like distance for the thumb to the primary action, not block scroll paths built into the device causing janky movement are things you cannot see with a virtual instance.

    It is impossible to test everything and anything, I think this concept of blocking things into groups is a good idea, we have a device lab which is becoming quickly historic but that’s also at the fault of manufacturers.

    Try and get a development kit from any of the top phone manufacturers for example to just test their browser – it doesn’t happen. Perhaps if you have enough clout or notoriety you may be able to schmooze someone enough to get something out of them, but even when working on a project for a major telephone manufacturer recently I was unable to obtain one of their handsets in order to test their site or web apps.

  3. 7

    Tárcio Zemel

    July 14, 2014 3:43 pm

    Great article, great strategy proposal, congrats!

    Just one thing about “‘desktop’ here meaning both Mac and Windows”. There’s a entire world beyond Mac and Windows, pal.

  4. 8
  5. 10

    David Scanu

    July 14, 2014 3:46 pm

    Great article.

    I would recommence Browser-Sync for syncing the same web page on all those devices.

    Codekit 2 has a great server feature too.

  6. 11

    Nice article. thanks!

    I would also recommend to find and fix cross-mobile & browser issues in seconds.
    Site provides the testing of the following areas:
    – visual rendering of site across 25+ devices and browser
    – site performance (server side + client scripts)
    – device compatibility
    – content analysis (broken links, spell checking etc.)

    You also receive a detailed report and step-by-step instruction to address any found issue.

    Our web-design studio has been using this service for a while to check development sites in progress and it has been a killing time-saver to identify website issues across devices.

  7. 12

    Marija Zaric

    July 14, 2014 7:21 pm

    I suggest for testing responsive websites and others as well:
    resizer, amIresponsive…. With so many devices we can’t test the responsive websites on all of them but one thing for sure is that with the responsive approach the potential users have a choice of various devices they can choose and get the best user experience.

  8. 13

    Jonny Firstson

    July 14, 2014 11:38 pm

    Hi Marija Zaric,

    have you noted that iPadpeek renders content in your local Chrome/IE/Opera browser and it also renders Flash? :)
    It doesn’t look that they are emulating iPad at all, but simply render the content using your browser inside the iPad picture, pretending that it’s the real iPad :)

    Not cool

  9. 14

    Richard Powell

    July 15, 2014 5:58 am

    I encountered this challenge for the first time in 2010 when rebuilding a 50,000 page website in HTML5 with responsive design. I knew we would need to do a lot of extra testing.
    The fundamentals of this article are good, but virtual machines, emulators and browserstack are a huge time-sink when a team of ten people are setting up, administering and upgrading them on their desktop computers. Only one member of my team was a experienced system administrator – the rest had a steep learning curve.
    I got so concerned about the productivity decline that I looked for a new solution.
    So instead, I bought a team subscription to a cloud service to access browser testing environments and banned VM’s and Browserstack.
    The pros and cons:
    The team instantly had access to a much broader range of testing platforms.
    We gained the ability to run automated tests (where we wanted to).
    Using tunnelling we could test any site that the developer could access from their desktop.
    This saved a significant amount of time.
    Costs were much cheaper than parallels plus other software plus windows licences for the macs.
    No queuing for a computer with a specific config to test.
    The cons:
    Lack of developer and inspection tools on the browsers.
    Not quite as good as physical devices in the hand. But one provider used physical tablets and phones in their testing platform!
    The cloud solution would have been cheaper. It was a shame I couldn’t get refunds for all the unnecessary licenses already purchased for desktop solutions.

    Finally I worked in a large organisation, so I found other teams in a similar position to mine, and we collectively bought one example of the most popular kind of physical devices e.g. iPad. These were booked out when required e.g. for final testing, or for internal demonstration of a prototype. This was a useful fallback, which we rarely needed to use.

  10. 15

    Richard Powell

    July 15, 2014 6:38 am

    Responsinator and Testize are interesting tools. Note that these that they just change the viewport dimensions to match those of the target device.
    If you have adaptive designs, or use the native input fields available on the devices, or use the useragent string to customise any features, they won’t give an accurate rendering of the screen appearance.
    If media queries are the only mobile design element you’re using, then they’ll be fine.

    • 16

      Richard, we have contacted Testize recently inquiring on the authority of their tests. They are actually using devices to run tests and not mocking the viewports for browsers

      • 17

        Richard Powell

        July 16, 2014 7:14 am

        Interesting. I tested an adaptive site with Testize and it failed to recognise the device and render for it. However when I tested the same site with a physical device, it recognised and loaded correctly.
        Either something is not representative in the environment, or the site has a hard to reproduce bug.

        • 18

          Hi Richard,
          I’m Amy from Testize team. Could you let me know which site have you tried to test, so I could validate what happened?
          Btw, we do process responsive designs correctly, for example:
          Thanks in advance.

  11. 19

    “I strongly recommend that you purchase the following devices: … ”

    Funny guy ;) Let me borrow your credit card.

  12. 20

    I’d suggest to add screen readers as part of your quality assurance procedure & budget. It would be cheaper and save time to fix the problem early than try to fix it later after the site is deployed and find out certain critical function doesn’t work for screen reader users. One can use a screen reader emulator for desktop, active the Voice Over (for Apple product) or the Talk Back on Android devices. Better yet, enlist an actual screen reader from your community of users.

    • 21

      Hi Ranti,

      That’s a good point, I didn’t mean to deliberately omit screen readers. One of the strategies in the article is to catch rendering issues early so I would probably de-prioritise screen reader testing. But this is because I’m a seasoned developer who writes semantic HTML and makes JavaScript interactions accessible by default so maybe it would be more useful for less experienced developers.

      A benefit to screen reader testing is that it shows you how SEO friendly your page is going to be, so it’s definitely something people should take seriously.


  13. 22

    “And during the flooding in the UK earlier this year, we couldn’t contact God to ask him to pause the heavy rain long enough for my team to get the interactive flooding map working in Android 2.3.”

    The bible says your wrong.

  14. 23

    “While you should try to support as large a proportion of your audience as possible, your analytics will show that a small group of browsers and devices make up the majority of your traffic, while the rest are a long tail of obscure browsers and devices. ”

    You ever been to a site that just plain didn’t work on your device; at least not in a usable way? Did you go back..?

    All those “obscure” browsers you didn’t fix for… aren’t coming back. That’s why it’s such a low number all the time; because they didn’t come back.

  15. 24

    Instead of putting the onus on the developers, we need to put it on the software creators. Everyone NEEDS to adhere to standards. They don’t. They never have. They need to start. Every browser has to render things EXACTLY the same. Sure, you can make your browser faster if you want. Add features here and there. But, the web page rendering needs to be exact. If this doesn’t happen (and soon) we will never be done testing anything…ever.

  16. 25

    I strongly recommend to check how is your responsive website works…

  17. 27

    Vadim Makeev

    July 15, 2014 8:51 pm

    > Download and test Opera Mini on your iOS 7 device

    There’s important note on it: the latest version of Opera Mini 8 for iOS using Turbo mode by default and will render your pages using built-in WebView engine e.g. WebKit (with some server-side compression) which is totally different from Mini mode that could be enabled in browser options. This will be the real Mini mode including server-side rendering using Presto engine, less powerful JS, etc.

    There’s another option to get real Opera Mini for testing: to use Microemulator that could open JAR file of Opera Mini app which could be downloaded here. I hope you’ll mention some of that information in your article.

  18. 29

    Aaron Martone

    July 15, 2014 9:10 pm

    Cross-browser testing and performance is a particularly aggravating subject for perfectionists like myself (FYI, perfectionists don’t think they are perfect, far from it, we just attempt to achieve perfection in everything we do). The problem here, is that due to a lack of standardization, perfection is impossible. And like asking someone with OCD to simply NOT have it, it’s never that simple.

    As such, like with video games and software, we end up with a list of “minimum system requirements” with our project. A scope of the most popular hardware/software combinations that we feel warrant the majority of users who access our services. If your device falls outside that range, we simply cannot guarantee that things will work as expected. It isn’t feasible; and I’m coming to terms with that realization. As long as devices continue to be different from one another, we cannot hold onto the idealism that we can fix everything; we have to be more realistic than that.

  19. 31

    If you develop on a PC instead of a Mac, you can get VMWare Player for free and copies of various Windows operating systems + OS’s as well as Mac OSX images for free. Easy install, they run faster than on the mac, and you can laugh at mac users for having to go through a lot more problems than a PC user for this type of testing.

  20. 32

    I recommend getting real WindowsPhone devices. I bought a Nokia Lumia 520 (WP8/IE10) for 150€ 6 mounth ago and last week a Lumia 630 (WP8.1/IE11) for 140€. No money for a company or agency.

  21. 33

    Victor Brito

    July 17, 2014 9:55 am

    Some remarks about this excellent article.

    First, two mistakes. I do not think Opera Mobile and Firefox Mobile are available for Apple devices: they are for Android ones. And Windows Vista users with SP 2 installed can upgrade to IE 9.

    Then, I would include desktop browsers like Firefox latest, Safari latest and Opera latest in group 1, just to remember that there is not only Google Chrome (let us avoid reproducing what developers did in late 1990’s and early 2000’s with IE, don’t we?). And I would consider Opera desktop latest similar to other modern browsers (even Opera Desktop 12.1, the latest version running Presto).

    As far as group 2 is concerned, I would replace the Windows Phone emulator by a real Windows Phone device: as Sven said, Windows Phone devices are not very expensive, especially the Nokia Lumia ones.

    Finally, we must keep in mind that these groups have to be updated as new browser versions come: just think of the Nokia Lumia 630, which runs on Windows Phone 8.1, whose default browser is IE Mobile 11.

    • 34

      Hi Victor,

      Yes you’re absolutely correct, Vista users can upgrade to IE9. The firefox and Opera Mobile browsers stated in group 4 should have been for Android too. I’ll update the article.


  22. 35

    Matthew Trow

    July 17, 2014 2:04 pm

    A great insight into the complexity of modern web design!

    One thing I’ve learned along the way, for new projects, test often on as many platforms as possible.

    Each time you create a major feature / layout, test it to destruction.
    The time spent in early development will save 2, 3 or 4 times as much time as trying to test it all toward the end.

    You can do quick tests using emulators – the new Chrome developer tools are excellent for this. Firefox responsive view is also fantastic. Then you haul out the devices at key points of development.

    Due to budget constraints, I’ve settled on doing more emulator testing for layout than individual device testing. I test on 5 mobile devices – ipad, iphone, android tablet, android phone & old android phone. For desktop, the major browsers on windows & mac and ie down to version 8.

    Finally, low level tests on a text only browser, along with accessibility testing.

    Mobile first for wireframing and design is *essential* – and deciding on breakpoints equally so. The key breakpoints *have* to offer a good measure of fluidity – a flexible grid system, capable of handling the difficult breakpoints between 480 & 1280 without resorting to a single column is also essential.

    Test often, test early, adapt, change, commit often.

  23. 36

    How about IE6 & 7, so sad front developer in China :(

    • 37

      Hi Riant,

      The list of devices/browsers is prioritised based on the stats taken from my own companies website. If you wanted to include IE6 and 7 then take a copy of this list and add them in where you think they should go.

      I personally would test IE7 once I’ve tested IE8. These two browsers are quite similar (IE7 has more CSS bugs in it) so if IE8 worked I’d have more confidence to then test IE7.

      If you find that trying to support browsers as old as IE 6 and 7 are causing you huge amounts of issues I would strongly recommend using a technique like “cutting the mustard”


  24. 38

    Mateusz Papiernik

    July 21, 2014 3:40 pm

    Thanks for this useful article. Having said that, I can’t stress enough how important is to take advantage of the analytics you have access to. Results can be surprising. At my current job I consider IE 8 and Android 2.3 already extinct, and for most of our customers we rarely have to test IE9 either. On the other hand, we have to be precise with IE testing on Windows Mobile devices, and they can prove tricky at times. I’ve many times encountered strange rendering errors that I wouldn’t foresee and that wouldn’t happen on any other mobile, nor on desktop IE.

  25. 39

    Great content, but the terrible Miley Cyrus reference was unneccessary. I’m guessing it’s a joke but it’s out of place and kinda creepy.

  26. 41

    Also Angelina Jolie in an Orphanage? Please stop trying to be funny and just stick to the content writing, the jokes don’t work.

  27. 42

    Responsive design has been becoming trend that each and every web site needs to be Responsive. Google also track and rank website according to it’s supports for multiple devices. To test if your site is responsive on not. use this link

  28. 43

    Great read. And here I was thinking RWD was supposed to make my life easier!

    Hey, instead of buying any real devices, why not just go with a real-device mobile testing service like Keynote?

    There’s a free version with a limited device list over wifi only and then a free-trial to paid version with a bunch more over real carrier networks.

  29. 44

    Just to add the value to the content, here is the tool which might also help you:


↑ Back to top