Analyzing Network Characteristics Using JavaScript And The DOM, Part 1

Advertisement

As Web developers, we have an affinity for developing with JavaScript. Whatever the language used in the back end, JavaScript and the browser are the primary language-platform combination available at the user’s end. It has many uses, ranging from silly to experience-enhancing.


(Image: Viktor Hertz)

In this article, we’ll look at some methods of manipulating JavaScript to determine various network characteristics from within the browser — characteristics that were previously available only to applications that directly interface with the operating system. Much of this was discovered while building the Boomerang project to measure real user performance.

What’s In A Network Anyway?

The network has many layers, but the Web developers among us care most about HTTP, which runs over TCP and IP (otherwise known jointly as the Internet protocol suite). Several layers are below that, but for the most part, whether it runs on copper, fiber or homing pigeons does not affect the layers or the characteristics that we care about.

Network Latency

Network latency is typically the time it takes to send a signal across the network and get a response. It’s also often called roundtrip time or ping time because it’s the time reported by the ping command. While this is interesting to network engineers who are diagnosing network problems, Web developers care more about the time it takes to make an HTTP request and get a response. Therefore, we’ll define HTTP latency as the time it takes to make the smallest HTTP request possible, and to get a response with insignificant server-processing time (i.e. the only thing the server does is send a response).

Cool tip: Light and electricity travel through fiber and copper at 66% the speed of light in a vacuum, or 20 × 108 kilometres per second. A good approximation of network latency between points A and B is four times the time it takes light or electricity to travel the distance. Greg’s Cable Map is a good resource to find out the length and bandwidth of undersea network cables. I’ll leave it to you to put these pieces together.

Network Throughput

Network throughput tells us how well a network is being utilized. We may have a 3-megabit network connection but are effectively using only 2 megabits because the network has a lot of idle time.

DNS

DNS is a little different from everything else we care about. It works over UDP and typically happens at a layer that is transparent to JavaScript. We’ll see how best to ascertain the time it takes to do a DNS lookup.

There is, of course, much more to the network, but determining these characteristics through JavaScript in the browser gets increasingly harder.

Measuring Network Latency With JavaScript

HTTP Get Request

My first instinct was that measuring latency simply entailed sending one packet each way and timing it. It’s fairly easy to do this in JavaScript:

var ts, rtt, img = new Image;
img.onload=function() { rtt=(+new Date - ts) };
ts = +new Date;
img.src="/1x1.gif";

We start a timer, then load a 1 × 1 pixel GIF and measure when its onload event fires. The GIF itself is 35 bytes in size and so fits in a single TCP packet even with HTTP headers added in.

This kinda sorta works, but has inconsistent results. In particular, the first time you load an image, it will take a little longer than subsequent loads — even if we make sure the image isn’t cached. Looking at the TCP packets that go across the network explains what’s happening, as we’ll see in the following section.

TCP Handshake and HTTP Keep-Alive

TCP handshake: SYN-ACK/SYN-ACK

When loading a Web page or image or any other Web resource, a browser opens a TCP connection to the specified Web server, and then makes an HTTP GET request over this connection. The details of the TCP connection and HTTP request are hidden from users and from Web developers as well. They are important, though, if we need to analyze the network’s characteristics.

The first time a TCP connection is opened between two hosts (the browser and the server, in our case), they need to “handshake.” This takes place by sending three packets between the two hosts. The host that initiates the connection (the browser in our case) first sends a SYN packet, which kind of means, “Let’s SYNc up. I’d like to talk to you. Are you ready to talk to me?” If the other host (the server in our case) is ready, it responds with an ACK, which means, “I ACKnowledge your SYN.” And it also sends a SYN of its own, which means, “I’d like to SYNc up, too. Are you ready?” The Web browser then completes the handshake with its own ACK, and the connection is established. The connection could fail, but the process behind a connection failure is beyond the scope of this article.

Once the connection is established, it remains open until both ends decide to close it, by going through a similar handshake.

When we throw HTTP over TCP, we now have an HTTP client (typically a browser) that initiates the TCP connection and sends the first data packet (a GET request, for example). If we’re using HTTP/1.1 (which almost everyone does today), then the default will be to use HTTP keep-alive (Connection: keep-alive). This means that several HTTP requests may take place over the same TCP connection. This is good, because it means that we reduce the overhead of the handshake (three extra packets).

Now, unless we have HTTP pipelining turned on (and most browsers and servers turn it off), these requests will happen serially.

HTTP keep-alive

We can now modify our code a bit to take the time of the TCP handshake into account, and measure latency accordingly.

var t=[], n=2, tcp, rtt;
var ld = function() {
   t.push(+new Date);
   if(t.length > n)
     done();
   else {
     var img = new Image;
     img.onload = ld;
     img.src="/1x1.gif?" + Math.random()
                         + '=' + new Date;
   }
};
var done = function() {
  rtt=t[2]-t[1];
  tcp=t[1]-t[0]-rtt;
};
ld();

With this code, we can measure both latency and the TCP handshake time. There is a chance that a TCP connection was already active and that the first request went through on that connection. In this case, the two times will be very close to each other. In all other cases, rtt, which requires two packets, should be approximately 66% of tcp, which requires three packets. Note that I say “approximately,” because network jitter and different routes at the IP layer can make two packets in the same TCP connection take different
lengths of time to get through.

You’ll notice here that we’ve ignored the fact that the first image might have also required a DNS lookup. We’ll look at that in part 2.

Measuring Network Throughput With JavaScript

Again, our first instinct with this test was just to download a large image and measure how long it takes. Then size/time should tell us the throughput.

For the purpose of this code, let’s assume we have a global object called image, with details of the image’s URL and size in bits.

// Assume global object
// image={ url: …, size: … }
var ts, rtt, bw, img = new Image;
img.onload=function() {
   rtt=(+new Date - ts);
   bw = image.size*1000/rtt;    // rtt is in ms
};
ts = +new Date;
img.src=image.url;

Once this code has completed executing, we should have the network throughput in kilobits per second stored in bw.

Unfortunately, it isn’t that simple, because of something called TCP slow-start.

Slow-Start

In order to avoid network congestion, both ends of a TCP connection will start sending data slowly and wait for an acknowledgement (an ACK packet). Remember than an ACK packet means, “I ACKnowledge what you just sent me.” Every time it receives an ACK without timing out, it assumes that the other end can operate faster and will send out more packets before waiting for the next ACK. If an ACK doesn’t come through in the expected timeframe, it assumes that the other end cannot operate fast enough and so backs off.

TCP window sizes for slow-start

This means that our throughput test above would have been fine as long as our image is small enough to fit within the current TCP window, which at the start is set to 2. While this is fine for slow networks, a fast network really wouldn’t be taxed by so small an image.

Instead, we’ll try by sending across images of increasing size and measuring the time each takes to download.

For the purpose of the code, the global image object is now an array with the following structure:

var image = [
	{url: …, size: … }
];

An array makes it easy to iterate over the list of images, and we can easily add large images to the end of the array to test faster network connections.

var i=0;
var ld = function() {
   if(i > 0)
      image[i-1].end = +new Date;
   if(i >= image.length)
      done();
   else {
      var img = new Image;
      img.onload = ld;
      image[i].start = +new Date;
      img.src=image[i].url;
   }
   i++;
};

Unfortunately, this breaks down when a very slow connection hits one of the bigger images; so, instead, we add a timeout value for each image, designed so that we hit upon common network connection speeds quickly. Details of the image sizes and timeout values are listed in this spreadsheet.

Our code now looks like this:

var i=0;
var ld = function() {
   if(i > 0) {
      image[i-1].end = +new Date;
      clearTimeout(image[i-1].timer);
   }
   if(i >= image.length ||
         (i > 0 && image[i-1].expired))
      done();
   else {
      var img = new Image;
      img.onload = ld;
      image[i].start = +new Date;
      image[i].timer =
            setTimeout(function() {
                       image[i].expired=true
                    },
                    image[i].timeout);
      img.src=image[i].url;
   }
   i++;
};

This looks much better — and works much better, too. But we’d see much variance between multiple runs. The only way to reduce the error in measurement is to run the test multiple times and take a summary value, such as the median. It’s a tradeoff between how accurate you need to be and how long you want the user to wait before the test completes. Getting network throughput to an order of magnitude is often as close as you need to be. Knowing whether the user’s connection is around 64 Kbps or 2 Mbps is useful, but determining whether it’s exactly 2048 or 2500 Kbps is much less useful.

Summary And References

That’s it for part 1 of this series. We’ve looked at how the packets that make up a Web request get through between browser and server, how this changes over time, and how we can use JavaScript and a little knowledge of statistics to make educated guesses at the characteristics of the network that we’re working with.

In the next part, we’ll look at DNS and the difference between IPv6 and IPv4 and the WebTiming API. We’d love to know what you think of this article and what you’d like to see in part 2, so let us know in a comment.

Until then, here’s a list of links to resources that were helpful in compiling this document.

(al)

↑ Back to top

Philip Tellis is a geek, speedfreak and Chief RUM Distiller at SOASTA where he works on the mPulse product. mPulse helps site owners measure the real user perceived performance of their sites and help visualize correlations between performance and business metrics like conversions, sales, lolcats and more. You can use mPulse for free at http://www.soasta.com/free.

  1. 1

    I like the logic behind this code, though I don’t quite understand the purpose. Do you have some specific use-cases for this technique?

    0
    • 2

      Hi Stijnd,

      We’ve used these techniques at Yahoo! (see the boomerang project) to measure real user experience (with respect to network performance) while they browse a site. Once you start measuring, you can identify important bottlenecks (ie, those that affect user retention/sales/etc.) and also quantify any improvements/regressions that get in.

      Philip

      0
  2. 3

    This is a bizarre topic … not sure I understand the purpose.

    0
  3. 4

    The purpose should be for example – definition of SLA. Based on SLA you should open different type of IP services.

    0
  4. 5

    Karl Dubost (Opera)

    November 14, 2011 8:46 am

    Note that Opera browsers have HTTP Pipelining on even on Mobile for a very long time.

    0
  5. 6

    Awesome post, thank you for sharing

    0
  6. 7

    love love love this…thanks…makes me think this might be useful as a wordpress plugin as well…when you login…you can get a user report

    0
    • 8

      Email me. Someone’s already built a drupal plugin using boomerang, and I’m working on the data collection and analysis framework.

      0
  7. 9

    This was a fun post to read, and I look forward to the other ones.

    It is a little weird, mostly because the first question you ask yourself is ‘why would you use Javascript to do this?” though I understand how it could be useful in measuring real-life user experience (given that you have the resources to pursue such a thing).

    0
  8. 10

    Awesome post.I love this post.Thank you.

    0
  9. 11

    Very interesting read. Certainly now that front-end performance is becoming more and more a hot topic.

    0
  10. 12

    Could you explain the “+new Date” construct that you are using with your code snippets? Is “+new” a custom function, or is this a normal JavaScript construct that I have just not seen yet?

    0
    • 13

      The + just coerces the Date to a number. It’s equivalent to getTime(), but much shorter.

      0
      • 14

        You’re writing overly “clever” code, not readable code (I was going to ask the same question, thought it was a typo in the first snippet, but then kept seeing it).

        Without an initial comment explaining what it does, it doesn’t help. Contrast that with new Date().getTime();

        Shorter does not always equal better.

        0
        • 15

          You’re right, of course. I’ve just gotten into the habit of writing it this way. I believe I initially got it in to make boomerang fit into one less TCP packet, and it just stuck with me.

          That said, it isn’t uncommon to see this construct in node.js modules, perhaps because with node you know which JS engine you’re running under, and don’t have to worry about it not working in a different engine.

          0
  11. 16

    Great work phillip.
    I will be using your idea to measure up performance in a sophisticated db architecture in our company.
    Keep up the good work.

    0
    • 17

      you can use the boomerang JavaScript library to do the front-end measurement and beacon that back to your back end infrastructure. If you need help building the back end, I can suggest a few options.

      0
  12. 18

    Thanks, very interesting! You could also see performance info with the new window.performance api (Chrome, FF), http://www.html5rocks.com/en/tutorials/webperformance/basics/

    0
    • 19

      yeah, I’ll be covering that in part 2 of this article. boomerang already supports the NavigationTiming API, but not all browsers support it, and we’ve noticed some bugs in the implementations of at least one browser. NavigationTiming also doesn’t tell you the user’s network throughput, or the difference between IPv4 and IPv6 latency.

      I believe that at this point in time, there’s still a place for a generic library that combines the best of all techniques to measure real user performance.

      0
  13. 20

    this would be better if you walked through each part of your code and broke down what was happening

    0
  14. 21

    Philip, I’m curious what one would use this for. I’ve seen people talking about how it would be nice to know a users connection speed to figure what type of assets/resources to send to them. Is that what this could be used for?

    0
    • 22

      Yes, that’s part of what it’s used for. At Yahoo! we also used it to measure how changes to code affected page load time and latency. These measurements only make sense when you compare users with similar network throughput values.

      0
  15. 23

    I’ve done something far not so precise before I stumbled your code and the boomerang project.

    I wanted to ping the server so I can make a visual loader of any image, instead of the spinners that say nothing about the progress. But didn’t work as expected.

    I’ll try with boomerang.

    0
  16. 24

    Hi Philip!
    If you want to play with execution times why don’t you suggest to use a software tool like Tor, Charles Proxy or Slowy ?
    Anyway your JS solution is more “scientific”. Good to know.

    0
    • 25

      Hi Alberto,

      These tools are good to use in development, but they tell you nothing about your user’s experience. You can’t ask your users to install Charles or YSlow. We actually built YSlow and boomerang as two complementary products, one to use in development and the other to use in production.

      0
  17. 26

    This article was fantastic and I can’t wait to read the next part. I learnt a lot!

    0
  18. 27

    wheres part two

    0
  19. 29

    JAMIE McDonnell

    June 13, 2013 11:44 pm

    Hi guys,

    this looks like it is exactly what I need but I’m not having much luck with it, or boomerang either to be honest.

    My boomerang test is here:
    http://www.ui-ux-design.com/ef2f/boomerang/

    from here:
    https://gist.github.com/alexyoung/458518

    Can anyone see what I’m doing wrong or why I see nothing in the console?

    I’ve tried using the most basic boomerang example with subscribe before_beacon, but I still get nothing ;(

    I’m using Chrome beta 28 on osx.

    Any help, suggestions, or a working test page I can build out from would be great!

    Cheers, and keep up the great work!

    Jamie

    0
    • 30

      Hi Jamie,

      You need to include one of the boomerang plugins as well to actually add something to the beacon and call the sendBeacon() method. boomerang on its own only provides the beaconing functionality. The rt.js or bw.js (or both) plugins make it work.

      You would typically create a combined + minified version of the code using the Makefile that’s included in the boomerang repo. Feel free to leave comments on the boomerang issue tracker as I follow that more closely.

      0
  20. 31

    Great read, I feel like I understand more about the process after reading the two articles.

    Posts like this are really refreshing. Being focused on client side development takes up all my reading time so it was good to take in some new information about the process itself.

    0

Leave a Comment

Yay! You've decided to leave a comment. That's fantastic! Please keep in mind that comments are moderated and rel="nofollow" is in use. So, please do not use a spammy keyword or a domain as your name, or else it will be deleted. Let's have a personal and meaningful conversation instead. Thanks for dropping by!

↑ Back to top