Analyzing Network Characteristics Using JavaScript And The DOM, Part 2

Advertisement

In Part 151 of this series, we had a look at how the underlying protocols of the Web work, and how we can use JavaScript to estimate their performance characteristics. In this second part, we’ll look at DNS, IPv6 and the new W3C specification for the NavigationTiming2 API.

DNS Explained

Every device attached to the Internet is identified by a numeric address known as an IP address. The two forms of IP addresses seen on the open Internet are IPv4, which is a 32-bit number often represented as a series of four decimal numbers separated by dots, e.g. 80.72.139.101, and IPv6 which is a 128-bit number represented as a series of multiple hexadecimal numbers separated by colons, e.g. 2607:f298:1:103::c8c:a407.

These addresses are good for computers to understand; they take up a fixed number of bytes and can easily be processed, but they’re hard for humans to remember. They’re also not very good for branding, and are often tied to a geographic location or an infrastructure service provider (like an ISP or a hosting provider).

To get around these shortcomings, the “Domain Name System” was invented. At its simplest form, DNS creates a mapping between a human readable name, like “www.smashingmagazine.com” and its machine readable address (80.72.139.101). DNS can hold much more information, but this is all that’s important for this article.

For now, we’ll focus on DNS latency, and how we can measure it using JavaScript from the browser. DNS latency is important because the browser needs to do a DNS lookup for every unique hostname that it needs to download resources from — even if multiple hostnames map to the same IP address.

3
Larger view4

Measuring DNS Lookup Times

The simple way to measure DNS lookup time from JavaScript would be to first measure the latency to a host using its IP address, and then measure it again using its hostname. The difference between the two should give us DNS lookup time. We use the methods developed in Part 151 to measure latency.

The problem with this approach is that if the browser has already done a DNS lookup on this hostname, then that lookup will be cached, and we won’t really get a difference. What we need, is a wildcard DNS record6, and a Web server listening on it. Carlos Bueno did a great write-up about this on the YDN blog a few years ago7, and built the code that boomerang uses.

Before we look at the code, let’s take a quick look at how DNS lookups work with the following (simplified) diagram:

DNS Lookup path from Client to Authoritative Server8
From left to right: The client here is the browser, the DNS server is (typically) the user’s ISP, the Root name server knows where to look for most domains (or who to ask if it doesn’t know about them), and finally the authoritative server which is the DNS server of the website owner.

Each of these layers has their own cache, and that cache generally sticks around for as long as the authoritative server’s TTL (known as “Time To Live”) says it should; but not all servers follow the spec (and that’s a complete topic for itself).

Now, let’s look at the code:

var dns_time;

function start() {
    var gen_url, img,
        t_start, t_dns, t_http,
        random = Math.floor(Math.random()*(2147483647)).toString(36);

    // 1. Create a random hostname within our domain
    gen_url = "http://*.foo.com".replace(/*/, random);

    var A_loaded = function() {
        t_dns = new Date().getTime() - t_start;

        // 3. Load another image from the same host (see step 2 below)
        img = new Image();
        img.onload = B_loaded;

        t_start = new Date().getTime();
        img.src = gen_url + "image-l.gif?t=" + (new Date().getTime()) + Math.random();
    };

    var B_loaded = function() {
        t_http = new Date().getTime() - t_start;

        img = null;

        // 4. DNS time is the time to load the image with uncached DNS
        //    minus the time to load the image with cached DNS

        dns_time = t_dns - t_http;
    };

    // 2. Load an image from the random hostname
    img = new Image();
    img.onload = A_loaded;

    t_start = new Date().getTime();
    img.src = gen_url + "image-l.gif?t=" + (new Date().getTime()) + Math.random();

}

Let’s step through the code quickly. What we’ve done here is the following:

  1. Create a random hostname prefixed to our wildcard domain. This makes sure that the hostname lookup isn’t cached by anyone.
  2. Load an image from this host and measure the time it takes for it to load.
  3. Load another image from the same host and measure the time it takes to load.
  4. Calculate the difference between the two measured times.

The first measured time period includes DNS lookup time, TCP handshake time and network latency. The second time measured includes network latency.

There are two downsides to this approach though. First, it measures the worst case DNS lookup time, i.e. the time it takes to do a DNS lookup if your hostname isn’t cached by any intermediate DNS server. In practice, this isn’t always the case. There isn’t an easy way to get around that without the help of browsers, and we’ll get to that later at the end of this article.

Also, what probably makes it hard for most people to implement is setting up a wildcard DNS record. This isn’t always possible if you don’t control your DNS servers. Many shared hosting providers won’t let you set up a wildcard DNS record. The only thing you can do in this case is to move hosting providers, or at least DNS providers.

Measuring IPv6 Support And Latency

Technically, measuring IPv6 shouldn’t really be a separate topic, however, even a decade after its introduction, IPv6 adoption is still fairly low. ISPs have been holding back because not too many websites offer IPv6 support, and website owners have been holding back because not too many of their users have IPv6 support, and they’re not sure how it will impact performance or user experience.

The IPv6 test in boomerang helps you determine if your users have IPv6 support and how their IPv6 latency compares to IPv4 latency. It doesn’t check to see if their IPv6 support is broken or not (but see Google’s IPv6 test page229 if you’d like to know that).

There are two parts to the IPv6 test:

  1. First, we check to see if we can connect to a host using its IPv6 address, and if we can, we measure how long it takes.
  2. Next, we try to connect to a hostname that only resolves to an IPv6 address.

The first test tells us if the user’s network can make IPv6 connections. The second tells us if their DNS server can lookup AAAA records. We need to run the test in this order because we’d be unable to correctly test DNS if connections at the IP level fail.

The code is very similar to the DNS test, except we don’t need a wildcard DNS record:

var ipv6_url = "http://[2600:1234::d155]/image-l.gif",
    host_url = "http://ipv6.foo.com/image-l.gif",
    timeout: 1200,

    ipv6_latency='NA', dnsv6_latency='NA',

    timers: {
        ipv6: { start: null, end: null },
        host: { start: null, end: null }
    };

var img,
    rnd = "?t=" + (new Date().getTime()) + Math.random(),
    timer=0, error = null;

img = new Image();

function HOST_loaded() {
    // 4. When image loads, record its time
    timers['host'].end = new Date().getTime();
    clearTimeout(timer);
    img.onload = img.onerror = null;
    img = null;

    // 5. Calculate latency
    done();
}

function error(which) {
    // 6. If any image fails to load or times out, terminate the test immediately
    timers[which].supported = false;
    clearTimeout(timer);
    img.onload = img.onerror = null;
    img = null;

    done();
}

function done() {
    if(timers['ipv6'].end !== null) {
        ipv6_latency = timers.ipv6.end - timers.ipv6.start;
    }
    if(timers['host'].end !== null) {
        dnsv6_latency = timers.host.end - timers.host.start;
    }
}

img.onload = function() {
    // 2. When image loads, record its time
    timers['ipv6'].end = new Date().getTime();
    clearTimeout(timer);

    // 3. Then load image with hostname that only resolves to ipv6 address
    img = new Image();
    img.onload = HOST_loaded;
    img.onerror = function() { error('host') };

    timer = setTimeout(function() { error('host') }, timeout);

    timers['host].start = new Date().getTime();
    img.src = host_url + rnd;
};

img.onerror = function() { error('ipv6') };
timer = setTimeout(function() { error('ipv6') }, timeout);
this.timers['ipv6'].start = new Date().getTime();
// 1. Load image with ipv6 address
img.src = ipv6_url + rnd;

Yes, this code can be refactored to make it smaller, but that would make it harder to explain. This is what we do:

  1. We first load an image from a host using its IPv6 address. This checks to see that we can make a network connection to an IPv6 address. If your network, browser or OS don’t support IPv6, this will fail and the onerror event fires.
  2. If the image loads up, we know that IPv6 connections are supported. We record the time that we’ll use to measure latency later.
  3. Then we try to load up an image using a hostname that only resolves to an IPv6 address. It’s important that this hostname does not resolve to an IPv4 address or this test might pass even if the DNS server cannot handle IPv6.
  4. If this succeeds, we know that our DNS server can look up and return AAAA (the IPv6 equivalent of A) records. We record the time.
  5. And then go ahead and calculate the latency. We can compare this with our IPv4 latency and DNS latency. This would also be an appropriate place to call any callback function to say that the test has completed.
  6. If any of the image loads fired an onerror event or if they timed out, we terminate the test immediately. In that case, any tests that haven’t run have their corresponding variable (ipv6_latency or dnsv6_latency) set to “NA”, indicating no support.

There are other ways to test IPv6 support with help from the server side, for example, have your server set a cookie stating whether it was loaded via IPv4 or IPv6. This only works well if your testing page and your image page are on the same domain.

The NavigationTiming API

The NavigationTiming API10 is an interface provided by many modern browsers that gives JavaScript developers detailed information about the time the browser spent in the various states of downloading a page. The specification is still in a draft state, but as of the date of this article, Internet Explorer, Chrome and Firefox support it. Safari and Opera do not currently support the API.

JavaScript developers get access to the NavigationTiming object through window.performance.timing. Try this now. If you’re using Chrome, IE 9+ or Firefox 8+, open a Web console and inspect the contents of window.performance.timing.

The diagram below explains the order of events whose time shows up in the object. Let’s look at a few of them:

Navigation Timing Overview diagram from the W3C11
Larger view12 | Image source13

Now the items we’ve been interested in measuring are:

  1. Page Load Time
    We get the full page load time by taking the difference between loadEventEnd and navigationStart. The latter tells us when the user initiated the page load, either by clicking a link, or entering it into their browser’s URL bar. The latter tells us when the onload event finished. If we’re not interested in the execution time of the onload event, we could use loadEventStart instead.
  2. Network/Application Latency
    Network latency is the time from the browser initiating download to the time the first byte showed up. Now, part of this latency could be attributed to the application doing something before sending out bytes, but there’s no way to know that from the client side. We use the difference between requestStart and responseStart.
  3. TCP Connect Time
    TCP connect time is the difference between connectStart and connectEnd, however, if the connection is over SSL, then this includes the time to negotiate an SSL handshake. You’d need to take that into account, and use secureConnectionStart instead of connectEnd if it exists, and if you care about the difference.
  4. DNS Latency
    DNS latency is the difference between domainLookupStart and domainLookupEnd.

Important: We use a combination of times from window.performance.timing to determine each one of these.

While this looks good for the most part, and really tells you what your users experience, there are a few caveats to be aware of. If DNS is already cached, then DNS latency will be 0. Similarly, if the browser uses a persistent TCP connection, then TCP connect time will be 0. If the document is read out of cache, then network latency will be 0. Keep these points in mind and use them to determine what fraction of your users makes effective use of available application caches.

The Navigation Timing interface provides us with many more timers, but several of them are restricted by the browser’s same origin policy. These include details about redirects and unloading of the previous page. Other timers related to the DOM already have equivalent JavaScript events, namely the readystatechange, DOMComplete and load events.

The Network Information API

Another interesting network related API is the Network Information API2514. While not strictly performance related, it does help make guesses at expected network performance. This API is currently only supported by Android devices, and is exposed via the navigator.connection.type object. In particular, it tells you whether the device is currently using Ethernet, Wi-Fi, 2G or 3G.

An article I highly recommend reading would be David Calhoun’s piece that shows some good examples on Optimizing Based On Connection Speed15. Both the article and the comments are useful reading.

Summary

While the Navigation Timing API provides easy access to accurate page timing information, it is still insufficient to draw a complete picture. There is still some benefit to estimating various performance characteristics using the techniques mentioned earlier in this series.

Whether we need to support browsers that do not currently implement the Navigation Timing or get information about resources not included in the current page, be sure to find out more about the user’s network bandwidth or whether their support for IPv6 is better or worse than their support for IPv4 — a combination of methods gives us the best all-round picture.

All of the techniques presented here were developed while writing Boomerang16 though not all of them made it into the code yet.

References

The following links helped in writing this article and may be referred to for more information on specific topics:

(Credits of image on frontpage: Vlasta Juricek27)

(il) (ea)

Footnotes

  1. 1 http://www.smashingmagazine.com/2011/11/14/analyzing-network-characteristics-using-javascript-and-the-dom-part-1/
  2. 2 https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html
  3. 3 http://www.smashingmagazine.com/wp-content/uploads/2013/08/smashing-dns-waterfall-large_mini.png
  4. 4 http://www.smashingmagazine.com/wp-content/uploads/2013/08/smashing-dns-waterfall-large_mini.png
  5. 5 http://www.smashingmagazine.com/2011/11/14/analyzing-network-characteristics-using-javascript-and-the-dom-part-1/
  6. 6 http://en.wikipedia.org/wiki/Wildcard_DNS_record
  7. 7 http://developer.yahoo.com/blogs/ydn/posts/2009/11/guide_to_dns/
  8. 8 http://www.smashingmagazine.com/wp-content/uploads/2013/08/dns-lookup_mini.png
  9. 9 http://ipv6test.google.com/
  10. 10 https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html
  11. 11 http://www.smashingmagazine.com/wp-content/uploads/2013/08/timing-overview-large_mini.png
  12. 12 http://www.smashingmagazine.com/wp-content/uploads/2013/08/timing-overview-large_mini.png
  13. 13 https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html
  14. 14 http://www.w3.org/TR/netinfo-api/
  15. 15 http://davidbcalhoun.com/2010/using-navigator-connection-android
  16. 16 http://github.com/yahoo/boomerang
  17. 17 http://en.wikipedia.org/wiki/Domain_Name_System
  18. 18 http://tools.ietf.org/html/rfc1035
  19. 19 http://en.wikipedia.org/wiki/Wildcard_DNS_record
  20. 20 http://en.wikipedia.org/wiki/IPv4
  21. 21 http://en.wikipedia.org/wiki/IPv6
  22. 22 http://ipv6test.google.com/
  23. 23 http://tunnelbroker.net/
  24. 24 https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html
  25. 25 http://www.w3.org/TR/netinfo-api/
  26. 26 http://davidbcalhoun.com/2010/using-navigator-connection-android
  27. 27 http://www.flickr.com/photos/vlastula/300102949

↑ Back to topShare on Twitter

Philip Tellis is a geek, speedfreak and Chief RUM Distiller at SOASTA where he works on the mPulse product. mPulse helps site owners measure the real user perceived performance of their sites and help visualize correlations between performance and business metrics like conversions, sales, lolcats and more. You can use mPulse for free at http://www.soasta.com/free.

Advertising
  1. 1

    I have been using the Navigation Timing and Network Information APIs to estimate connection speed in order to serve more bandwidth appropriate assets. As part of that I did some research into how long an HTTP request takes so that I could estimate the request time for borwsers that don’t support the Navigation Timing API.

    0
    • 2

      Hi Orde, that’s interesting information, and in line with research that I’ve been doing as well. I’ve been collecting data using boomerang that was written for this purpose. Would love to have your opinion on it as well as some code patches. I read through your network speed article as well, and would like to know what you think about the method used in boomerang.

      0
  2. 3

    hi Philip,

    nice article, as always ;-)

    One comment: you write “DNS server is (typically) the user’s ISP” … would it not be better to write “DNS resolver” instead of “DNS server”?

    0
  3. 5

    I’ve been waiting 21 months for part 2 and finally it’s here!

    0
  4. 7

    But whats the point of DNS speed info?

    We will get it more or less after page is loaded, so where we can use it? To disable facebook widgets on slow network?

    0
    • 8

      Lukas,

      It helps in a few cases. For example, if you’re evaluating different DNS providers to decide which to go with, then this is a good technique for that. You could also use it to pick different DNS providers for different markets that you run your business in.

      Secondly, it can tell you whether to increase the number of domains you serve content from (by increasing parallel downloads) or whether the DNS latency outweighs the gains from parallelisation.

      I wouldn’t use it to test third party widgets because, first, you can’t host the DNS test image on the third party’s server, and secondly, with third parties, it’s an all or nothing situation. You either pay the price for all of their infrastructure choices (DNS + TCP + HTTP + Bandwidth + Concurrency) or you don’t (either they are good at not making you pay the price (I have a blog post on that), or you take them off your site).

      0
  5. 9

    The problem with the DNS latency measurement technique is the second image load can be from local browser cache not the actual download from the server. Even if that is the case, how do you know that you are not reusing the same connection when downloading the second image? Because if you reuse the connection, you are excluding the connection time (i.e 3-way handshake) and that would messes up with the whole estimation.

    0

Leave a Comment

Yay! You've decided to leave a comment. That's fantastic! Please keep in mind that comments are moderated and rel="nofollow" is in use. So, please do not use a spammy keyword or a domain as your name, or else it will be deleted. Let's have a personal and meaningful conversation instead. Thanks for dropping by!

↑ Back to top