Menu Search
Jump to the content X X
Smashing Conf New York

We use ad-blockers as well, you know. We gotta keep those servers running though. Did you know that we publish useful books and run friendly conferences — crafted for pros like yourself? E.g. upcoming SmashingConf Barcelona, dedicated to smart front-end techniques and design patterns.

How To Create Your Own Front-End Website Testing Plan

So, your designers and developers have created a fantastic front-end design, which the client is delighted with, and your job now is to test it. Your heart begins to sink: Think of all the browsers, all the devices and all of these web pages you’ve got to test, not to mention the iterations and bug fixes. You need a front-end testing plan.

This article shows you what to consider when creating a front-end testing plan and how to test efficiently accross browsers, devices and web pages.

Benefits Of A Front-End Testing Plan Link

  • Clarity on project’s scope
    Knowing which browsers and devices are within the project’s specification will help you to focus and will reduce development time and costs.
  • Reduced client friction
    This is done by showing detailed reports of completed test plans.
  • Confidence in deploying the project
    This comes from knowing you’ve thoroughly tested the project.

In my experience, front-end website mockups are not extensively tested before they are handed over to the back-end development team, the reason being time and budget constraints. They’re usually tested in the front-end developer’s browser of choice and maybe on their smartphone. So, when we came up against a monster of a project, Dad Info1, which has a ton of responsive page types, we needed a plan to methodically and efficiently test functionality and performance across platforms and across devices.

Dad Info is a not-for-profit website, built on Joomla, that aims to help fathers in need, whatever their situation. Last year, it helped over 500,000 dads and delivered 220,000 pages of information every month. I will be using Dad Info as a case study throughout to demonstrate how to create and complete your own front-end testing plan.

Let’s Tool Up Link

Before getting started, you’re going to need to tool up. In this article, I’ll use the following toolkit to complete the testing process:

  • Asana2
    Used for bug-tracking and team management
  • Chrome Developer Tools3
    Used for inspecting, debugging and profiling
  • Windows’ Snipping Tool (or Shift + Command + 4 on a Mac)
  • BrowserStack4
    Used to test cross-browser functionality on multiple virtual machines and emulators
  • Devices
    Ideally, you’ll want real devices. We have iPhone 4, 5 and 6; HTC One M8; Samsung Galaxy S5; Nokia Lumia 1520; Google Nexus 5; BlackBerry Curve; iPad 2; and Asus VivoTab Smart. If you don’t have these, you can use BrowserStack’s emulators.
  • Google’s PageSpeed Insights5
  • Pingdom Website Speed Test6
  • Screenr7
    Web-based screen recording, with sharing capabilities

What Are We Testing? Link

Because we’re front-end testers, our job is to know exactly what we are testing. We might not always find bugs per se; rather, we might find that something isn’t working as expected or that the developer has misunderstood the functionality requirements. Having a detailed specification up front that all stakeholders have agreed too will help to avoid some of these problems entirely. Later on in the Dad Info case study, we will go through a front-end specification to test the home page.

Who Are We Testing For? Link

First of all, you will need to understand your audience and how they will be consuming the website. Here are a few quick questions you should always ask:

  • What are the most popular devices your audience uses?
  • What operating system and browser combinations are most popular among your audience?
  • What connection speeds do they have (3G, 4G, broadband/fibre)?
  • How tech-savvy are they? We can make a judgment call here based on the topic of the website, their devices and their demographics.

For our Dad Info case study, the answers would be as follows:

  • iPhone 5, iPad 2+, desktop (1024 pixels and up)
  • Windows 7 Chrome, Windows 8 Chrome, Windows 8 Internet Explorer 10, OS X Safari, iOS 6 Safari
  • Broadband and 4G (a lot of city workers)
  • Our audience is mainly men between 18 and 35 years of age, fairly tech-savvy, with a smartphone in their pocket and an understanding of social media applications such as Facebook and Twitter.

So, how does this help us carry out a front-end test plan?

Armed with this information, we can instantly break down our huge to-do list into segments that are relevant to our audience and that prioritize our testing methodology. For functionality, we know which devices and browser to test in; for performance, we know what connections to test on; and for usability, we know that our audience uses social media applications, so we can include interface elements that they would be familiar with.

Know Your Limits Link

Know the limits of your project. At some point, you’re going to have a “That’s good enough” conversation with yourself. Projects limits are usually controlled by a few factors:

  • Budget
    Remember that you should charge for testing. A lot of designers don’t, which is crazy. Testing is time-consuming — and, being designers and developers, our product is time.
  • Timeline
    Include testing in the project’s timeline. It’s often left off the list and, therefore, rushed.
  • Scope
    Not every website needs to work on hundreds of devices. Figure out the main use case, and focus on fulfilling that audience’s requirements.

Browser and Device Support Levels Link

To set the scope of browser and device support easily with clients and to avoid those “bad conversations,” we’ve found that being up front about our “levels of support” really helps. Below are some simple definitions that you can apply to each page type you test.

Support level 1: fully supported browsers and devices

  • All content must be readable.
  • All functionality must work.
  • Deviation from approved graphic design must be minimized.

Support level 2: partially supported browsers and devices

  • All content must be readable.
  • Navigation must work.
  • Business login functionality must degrade gracefully.
  • Any degradation to presentation must not obscure content.

Support level 3: unsupported browsers and devices

  • No support or testing is required.

Performance Support Levels Link

You might also want to agree with your client on a performance target. The quick and dirty method here is to agree on a score to reach in Google’s PageSpeed Insights and Pingdom’s Website Speed Test. Normally, we aim for at least 85 out of 100.

Tools For Managing Your Test Plan Link

It doesn’t matter what tools you use. I use Asana8 and BugHerd9; you could use a simple spreadsheet. It comes down to what works best for you. At a minimum, your tool should be able to do the following:

  • add bugs, issues and tasks in an ordered and segmented list, with the ability to tag (“priority, system critical, etc.);
  • assign bugs, issues and tasks to members of your team (or to yourself), with due dates;
  • comment on bugs, issues and tasks, with a date-ordered history thread;
  • upload screenshots, videos and documents related to bugs, issues and tasks;
  • mark a bug, issue or task as resolved or completed;
  • report on completed versus remaining bugs, issues and tasks.

How To Describe Bugs And Issues Link

Ever got a bug report from your client that stated, “I clicked on the blog and it doesn’t work”? Useless, right! So, what does a well-written bug report look like?

  • Be specific
    Without being wordy, clearly summarize the problem. Do not combine multiple bugs into one report; rather, submit a separate report for each issue.
  • Show how to replicate
    Detail step by step exactly what you did and what issue occurred as a result.
  • Limit pronouns
    Descriptions like “I clicked it and the window didn’t appear” are very unclear. Instead, “I clicked the ‘Submit’ button and the window marked ‘Registration’ did not load” tells the developer exactly what you did and what happened.
  • Read what you’ve written
    Does it make sense? Do you think it’s clear? Can you replicate the bug by following your own steps?

Setting Up Your Front-End Test Plan Link

So far, you have collected a bunch of useful information and data, but you’re going to need a proper testing plan to succeed as a front-end tester. Without one, you’re shooting in the dark. So, what exactly does a front-end testing plan look like? In its simplest form, it’s a to-do list, with a bunch of tasks for testing each of your web page types against a set of agreed criteria. The best way to explain this is through a case study.

Case Study: Front-End Test Plan For Dad Info’s Home Page Link

Test Plan Documentation Link

Here, we lay out an overview to give the tester some context about the project. And we let them know what we want tested, on which devices and browsers and how long they have to do it.

Budget:

  • total of 10 days for testing
  • use one and a half days for front-end testing of home page

Timeline:

  • complete initial testing within one day, with feedback to front-end developers completed the same day
  • three days for fixing bugs
  • a further half day for retesting bugs

Scope:

  • Support level 1 (browsers):
    • Windows 8: IE 10+, Chrome (latest), Firefox (latest), Safari (latest)
    • Mac OS X Mavericks: Chrome (latest), Firefox (latest), Safari (latest)
  • Support level 1 (devices):
    • iPhone 4 / 5, iPad 2, Asus VivoTab Smart
  • Support level 2:
    • Windows 7: IE 9+, Chrome (latest), Firefox (latest), Safari (latest)
    • Windows XP: IE 8, Chrome (latest), Firefox (latest), Safari (latest)
  • Support level 3:
    • anything else

For this project, we require three reports to assure the client that the page has undergone and passed the testing process:

  • browser and device report: support level 1 and 2
  • responsive report: support level 1
  • performance report: minimum 85 out of 100

Original Approved Design Link

Having a visual representation of what you are working towards is important to ensuring that graceful degradation is within acceptable limits and that the presentation doesn’t change much between browsers. We’ve added this image to the test plan documentation:

Homepage design of Dad Info10
Home page mock-up of Dad Info (Image: New Edge Media) (View large version11)

Details of Page Functionality Link

In the home page design, I’ve highlighted all of the functionality that needs to be tested, highlighting them with block overlays. This helps everyone involved to know exactly what to look for and puts us all on the same page. I’ve also added this to the test plan documentation.

Home page graphic design mock up with testing area overlays of DAD.12

Home page mock-up, with overlays for testing areas (Image: New Edge Media) (View large version13)

Based on these overlays, we can produce a full list of functionality.

Search form Link
  • Click or touch in the search box and then press the search icon to submit the form.
  • Hover over navigation item to display white highlight.
  • Hover over navigation item “Family” to show drop-down mega menu.
  • Press the up and down arrows to browse the slides.
  • Press the pagination elements to skip to a particular slide.
  • Swipe through the slides on touch devices.
News feed: Link
  • Hover over the headings to change their color.
  • Press the up and down arrows to browse the slides.
  • Press the pagination elements to skip to a particular slide.
  • Swipe through the slides on touch devices.
Call-to-action block 1: Link
  • Hover over title to change it
  • Press the up and down arrows to browse the slides.
Twitter: Link
  • No special front-end functionality
Forum: Link
  • No special front-end functionality
Topics: Link
  • Hover over a support topic to reveal a picture to the right, with a description. Click “More” to go to a new page.
  • Hover over an icon to change its opacity.
Newsletter: Link
  • Clicking in or touching the “Enter email address” box should work.
  • Press “Subscribe” to submit the form.
  • No special front-end functionality

Browser and Device Report Link

Whether you decide to use a tool like Asana, BugHerd or Trello, your job as a tester is essentially to collect the following information to relay back to your front-end developers (or to use yourself if you’re solo). To quickly go through all of the browsers, I use BrowserStack, setting up virtual machines that run the OS and browser combinations that I require.

Test Item Browser/Device Pass/Fail Bug/Issue Description
search form
1.a
Windows 8 (IE 10) pass
navigation
1.a
Windows 8 (IE 10) pass
navigation
1.b
Windows 8 (IE 10) fail Cannot move mouse to mega menu area without it disappearing. See this Screenr recording14.
image carousel
1 – 3.a
Windows 8 (IE 10) fail The left and right arrows in the yellow box do not scroll slide 4 back to slide 1.

It’s a methodical job of working through all of the browser and device combinations until you have completed the same tests for each one.

Real Bug From Dad Info Link

  • item: navigation’s “Join” button
  • browser/device: Windows 8, IE 10, 1024-pixel screen width
  • pass/fail: fail
  • bug description: The “Join” button overlaps the Twitter button, highlighted in blue. See attached screenshot.
03-issuetracking-opt-small15

Screenshot of “Join” button bug (Image: New Edge Media) (View large version16)

Responsive Report Link

In the report on responsiveness, we are specifically testing elements and functionality that change as the screen’s canvas reduces in size. This includes navigation, page layout and images.

We have a few possible testing methods:

  • Resize the browser’s window
    This is the quick and dirty method, grabbing your browser’s window by the corner and dragging across the break points to see what happens. This is a great way to quickly scan the overall responsiveness of a website to see which elements change.
  • Use emulators
    BrowserStack emulates the majority of popular devices, and in my experience it is good enough for testing.
  • Use real devices
    The expensive option! This entails thoroughly testing on actual devices in your hand. Getting a screenshot usually requires capturing a photo, emailing it to yourself and then annotating in Photoshop. There are some options for screen recording, including UX Recorder17 for the iPhone.

For the Dad Info project, we used a mixture of all three. The front-end developers resized their browser to get the gist of responsive elements, while the quality assurance team used emulators and real devices to complete the testing process to the client’s satisfaction.

Real Bug From Dad Info Link

  • Item: image carousel 1
  • Browser/device: iPhone 5, iOS 7
  • Pass/fail: fail
  • Bug description: The margins set at the bottom of the carousel push the “News support your child…” title for the featured article too far down. Featured article title 1 should be 20 pixels below image carousel 1. See attached screenshot.
04-iphone-opt-small18

Screenshot of iPhone bug (Image: New Edge Media) (View large version19)

Performance Report Link

In the performance report, we are looking to score a minimum of 85 out of 100 in Google’s PageSpeed Insights. To get the client to sign off on the report, we’ve included a screenshot of the page speed analysis. Of course, if it doesn’t pass the agreed upon standard, then we provide feedback to the front-end development team using the bug-tracking report template.

Tip: We use boilerplates (stored in GitHub repositories, which we fork) for our content management system and Magento builds, whose performance is already optimized. This saves us a bunch of time.

05-google-desktop-opt-small20

The desktop version passed Google’s PageSpeed test. (Image: New Edge Media) (View large version21)
06-google-mobile-opt-small22

The mobile version passed Google’s PageSpeed test. (Image: Google2623)(View large version24)
07-pingtest-opt-small25

The website passed Pingdom’s speed test. (Image: Google2623) (View large version27)

The reporting is complete. We have just a few issues to send back to the front-end developers to wrap up the front-end build. Once it gets to the quality assurance team, we will retest the elements that kept the project from getting a clean bill of health.

Automated testing tools for visual regression Link

One thing to consider is the need for visual regression testing post go live, allowing you to compare editions of pages to see if your new feature, css update or class rename has caused any problems.

Visual regression tests essentially do a DIFF on the two versions and outline (in varying ways) the differences between them, highlighting potential issues.

Here are some great resources to get you started:

Summing Up Link

Testing is a critical process that developers should integrate into their workflow to minimize the number of bugs that get caught in the quality assurance phase. Front-end testing also needs to be budgeted for — with time, resources and money. The process will appeal to methodical types because they don’t need to be creatively skilled to carry it out. Tools are out there to make your life a little easier, but they won’t do the work for you. Whichever tool you pick, stick with it, define a process and put the effort in. The result will be a better website, with significantly fewer bugs, which your client will love and which will reduce the number of “Why isn’t this working?” phone calls and emails on Sunday night.

Your Action Plan Link

Reading what you should do is one thing; actually doing it quite another. So, I suggest carrying out the following actions right now:

  1. List the devices you have on hand, or check out Open Device Lab33 to find devices near you.
  2. Create support levels for your chosen browsers and devices.
  3. Make time for testing in your timelines and quotations.
  4. Select a management tool (Asana, BugHerd, etc.), or set up a spreadsheet to track bugs, issues and tasks.
  5. Select the first project to apply your test plan to.
  6. Go do it!

Front-end testing will give you and your client confidence in the finished project. You’ll know the website has been thoroughly tested for bugs and is ready for the world to see.

(da, ml, al)

Footnotes Link

  1. 1 http://www.dad.info/
  2. 2 http://asana.com
  3. 3 https://developer.chrome.com/devtools
  4. 4 http://www.browserstack.com
  5. 5 https://developers.google.com/speed/pagespeed/insights/
  6. 6 http://tools.pingdom.com
  7. 7 http://www.screenr.com
  8. 8 https://asana.com
  9. 9 http://bugherd.com
  10. 10 https://www.smashingmagazine.com/wp-content/uploads/2014/10/01-dad-homepage-opt.jpg
  11. 11 https://www.smashingmagazine.com/wp-content/uploads/2014/10/01-dad-homepage-opt.jpg
  12. 12 https://www.smashingmagazine.com/wp-content/uploads/2014/10/02-homepage-overlay-opt.jpg
  13. 13 https://www.smashingmagazine.com/wp-content/uploads/2014/10/02-homepage-overlay-opt.jpg
  14. 14 http://screenr.com
  15. 15 https://www.smashingmagazine.com/wp-content/uploads/2014/10/03-issuetracking-opt.png
  16. 16 https://www.smashingmagazine.com/wp-content/uploads/2014/10/03-issuetracking-opt.png
  17. 17 http://www.uxrecorder.com/
  18. 18 https://www.smashingmagazine.com/wp-content/uploads/2014/10/04-iphone-opt.jpg
  19. 19 https://www.smashingmagazine.com/wp-content/uploads/2014/10/04-iphone-opt.jpg
  20. 20 https://www.smashingmagazine.com/wp-content/uploads/2014/10/05-google-desktop-opt.png
  21. 21 https://www.smashingmagazine.com/wp-content/uploads/2014/10/05-google-desktop-opt.png
  22. 22 https://www.smashingmagazine.com/wp-content/uploads/2014/10/06-google-mobile-opt.png
  23. 23 http://www.google.co.uk
  24. 24 https://www.smashingmagazine.com/wp-content/uploads/2014/10/06-google-mobile-opt.png
  25. 25 https://www.smashingmagazine.com/wp-content/uploads/2014/10/07-pingtest-opt.png
  26. 26 http://www.google.co.uk
  27. 27 https://www.smashingmagazine.com/wp-content/uploads/2014/10/07-pingtest-opt.png
  28. 28 https://github.com/BBC-News/wraith
  29. 29 https://github.com/Huddle/PhantomCSS
  30. 30 https://github.com/facebook/huxley
  31. 31 https://www.lullabot.com/blog/article/css-regression-testing-resemblejs
  32. 32 http://csste.st/tools/
  33. 33 http://opendevicelab.com
SmashingConf New York

Hold on, Tiger! Thank you for reading the article. Did you know that we also publish printed books and run friendly conferences – crafted for pros like you? Like SmashingConf Barcelona, on October 25–26, with smart design patterns and front-end techniques.

↑ Back to top Tweet itShare on Facebook

Lawrence Howlett has a real passion for user experience and lead generation. He owns and runs www.newedge.co.uk — a full service digital agency, started from his bedroom, that now operates websites and manages digital marketing campaigns for high street brands and house hold names. Lawrence also shares his expert knowledge through eLearning courses, eBooks and blogging - connect over at http://lawrencehowlett.co.uk/.

  1. 1

    Luckily the modern desktop browsers are nowadays so good and with the help of jQuery the desktop problems are nothing like they used to be back in the day.

    On the other hand, the multitude of mobile device-browser-platform combinations have multiplied the time needed for testing.

    I’m using browser resizing, Browserstack and a set of mobile devices over Adobe Edge Inspect to test responsiveness and browser rendition errors. The few bugs that leave my development desk flies back to me in Jira.

    3
  2. 3

    Great article. Lots of good information. In my 15+ years of website development experience testing is the one thing that is almost always overlooked and under-appreciated. One of the most critical things to do once you determine your testing strategy is to get the right tools in place to manage the feedback and information. Glad you outlined some options.

    Another good feedback and testing option is https://www.pageproofer.com, it works across browsers and devices to allow your to note issues and leave feedback directly on a website, no jumping between website and tracking tool, no emails or spreadsheets to worry about.

    3
  3. 4

    @Lawrence, Please check links that gives 404 ex. BugHerd link.
    Nice Article, and good job in Dad.info (y)

    0
  4. 6

    Nice post which gives the valuable information.

    -3
  5. 7

    Good article Lawrence Howlett. We have a lot of similar methods and I thought very usefull to start asking myself “Who are my public?”, “what devices do they use?”, “what network are they using?” Testing the speed is a process that I forget almost the time when I’m working alone on a freelancer project but I’ll pay attention from now on, and start using red overlays highlights on my layout to know what I’m going to test step by step, thanks!

    -1
  6. 9

    Good post – but enforcing a minimum page speed score sounds odd to me. Page speed is a useful tool, but much like other similar tools, its scores should be taken with a pinch of salt. For example – it will mark you down for loading typekit or modernizr in the head. Getting around that may not actually do anything much for your page load times.

    Enforcing front-end devs to meet a minimum score does not seem conducive to me. It would be more helpful to enforce a minimum page-load time – which is a real world figure that a user will actually experience. It would also place the trust back in your front-end devs that they know how far to take optimisation of their code for a project. Even then, that would still depend on the project.

    Also – a good alternative to Asana is Trello, which is a simple but very powerful tool for software project management.

    0
    • 10

      A minimum page speed is specific to the project, not global for all clients, so can reflect the client aspirations. It’s a reporting metric that is very useful to ensuring front-end developers, back-end developers and hosting infrastructure are all on the same page.

      We tried Trello but didn’t get on with the “Card” type layout, just seemed a bit disorganised. Much prefer lists, but that’s 100% personal choice.

      0
      • 11

        Ah I see, I wasn’t clear that was on a per project basis.

        I would encourage you to give Trello another shot. It’s really just a board – how you organise it is pretty much up to you and you can use lists with Trello. But if you fundamentally don’t get along with the card idea then it’s probably better to use a different tool. We all work differently, after all.

        0
  7. 12

    Michael Schlotfeldt

    November 25, 2014 6:15 pm

    How does everybody else test all the needed browsers? I like using virtual machines installed locally for Windows, and physical devices for Mac OS X plus tablets and phones.

    Recently been trying Adobe Edge Inspect, Ghostlab, and other cross-device testing tools. Been really excited about those, but have not been able to find a really good comparison. Anybody know of any great comparisons?

    0
    • 13

      I’m a fan of SauceLabs for cross browser/platform testing, although it’s not free :(

      0
    • 15

      The “almost free” way is using virutal machines. I personally have three running WinXP – one with ie6, one with ie7 and one with ie8, one running OS X and Safari, one running SUSE and two running Win7 – one with ie8 and one with ie9. The reasons I have two ie8 is because they behave somewhat differently on XP and 7…. The other thing is that the Safari on Windows and OS X, again behaves differently…

      As you can see this is a time-consuming way. In the past months I am using http://www.browserstack.com . It gives you the same virtual machines + it allows you to emulate to mobile devices. Since I have only a Nexus phone and tablet, It allows me to test on other devices as well.

      Yes, it’s paid… but:

      1) Gives you more options to test.
      2) Uses virtual machines rather than emulators.
      3) It reduces the strain on your PC (launching several vitual machines to mitigate an issues is often a costly task).

      0
  8. 16

    Good article with handy tips.

    0
  9. 17

    “They’re usually tested in the front-end developer’s browser of choice and maybe on their smartphone.”

    Perhaps you should hire better frontenders…

    0
    • 18

      Who should get better front-end developers? In my experience of taking over projects and redesigns of existing websites, that statement is true. That is why we have front-end test plans, to ensure this is absolutely not the case!

      0
  10. 19

    How to covert a non-responsive website to responsive website? Can you help me out please

    -3
  11. 21

    Davy Eggermont

    December 3, 2014 9:12 am

    Great article. I love the way you have described your support levels.

    0
  12. 22

    Excellent article :)

    0
  13. 23

    Lawrence, great article, any tips for helping to determine your audiences’ device list?

    1
  14. 24

    nice article :)

    0
  15. 25

    Kristi Guerrero

    February 16, 2015 12:12 pm

    To accomplish this process successfully you have to know 3 things
    1. Your requirements
    3. Projects requirement
    3. User’s Requirements
    Merge them up
    Fix the bugs.
    That’s all

    1
  16. 26

    Diti Trivedi

    April 22, 2015 6:56 am

    Hi Lawrence,

    Its a good read, I just wanted to recommend the readers TestingWhiz, which is a automated tool for testing websites and main benefits of automation tools is that it requires no programming skills or code development experience and even provide more accuracy by reducing human errors. TestingWhiz can also help you in cross browser testing and for database testing. To know more you can visit http://www.testing-whiz.com
    For free demo of the software you can message me at dktrivedi@cygnet-infotech.com

    0
  17. 27

    Martin Wilberts

    June 4, 2015 5:35 am

    Excellent example test plan and a good case study. The problem we face being in a large financial institution is that we need to do as much as possible testing internally first and due to company policy we aren’t allowed VPN access into third party testing tools like saucelabs or browserstack and this causes a problem. We have been using a tool called Multibrowser, which is a downloadable software from http://MultiBrowser.com. It has both mobile browser emulators and a responsive tool that runs locally and can be automated (record and playback). Only then, once we feel the site is ready locally would we start testing on actual devices. Before go live we would have the site available on a IP staging site and test pagespeed etc.

    0
  18. 28

    Ishani Sachinthani

    August 20, 2015 12:58 pm

    Excellent article :)

    0
  19. 29

    About the testing apps, i create one free here: http://equaltozero.ro/demo/plugins/respons/ it is a app that it can test any kind of web site or shop or game, etc. You can take a look and test your web pages/app! ;-)

    1

↑ Back to top