A non-geeky guide to understanding performance measurement terms

In our industry, there’s a lot of language around how we time website speed. We tend to assume that outsiders understand our language, but something I read recently indicates that the average person doesn’t. We need to fix that.

A couple of weeks ago, I came across this article written by Luxury Daily writer Rachel Lamb: Luxury marketers dramatically drop site loading times. Given our own research into web performance and luxury markets here at Strangeloop, my curiosity was piqued.

The article made this statement, based on a recent report by SmartBear:

The average luxury site’s load time went from 2.6281 seconds in the third quarter to 1.321 seconds in the fourth quarter of 2011.

Looking at the results made me laugh and cry:

Given that the ideal load time is 2 seconds or less, this is a good-looking set of numbers — too good-looking. I did a little research of my own via WebPagetest.* The three new columns are mine.

What do these numbers mean?

If you’re a relative newcomer to the performance scene and the tables above looks like numerical gibberish, it’s not your fault. I’ll get into the terminology later in this post. For now, suffice to say that there’s a lot of variance in these numbers.

So, you have response time, time to first byte, start render time, and load time. Which set of numbers do you rely on to answer the #1 question site owners ask:

How fast does my site load for real users?

The short answer is: None of them, totally.

The long answer is: It’s complicated. Keep reading.

If you can’t trust numbers, what can you trust?

Numbers in a spreadsheet are a good way to spot larger patterns and trends, but if you want to get a ground-zero look at your site’s performance, capturing videos and filmstrip views of your pages’ load times are one of the best ways to go.

To illustrate, let’s take a closer look at two of the top-performing sites, according to this article: Prada and Rolls-Royce.

Remember that, in the luxury website performance article, Prada was lauded as having one of the fastest sites, with a response time of 0.17 seconds? While the response time may have been quick, there’s a serious problem at the network level. If you view Prada’s page load as a filmstrip (a nifty WebPagetest feature that I don’t think gets talked about enough), you see that, from a user’s perspective, nothing happens on the page until around 10.5 seconds, which roughly correlates to the start render time of 10.8 seconds.

You can also output the filmstrip to a video:

If you view the filmstrip for Rolls-Royce, you see that nothing starts to happen until around 3 seconds, again correlating roughly to the start render time of 2.712 seconds. This might sound acceptable on paper, but note that the feature banner doesn’t load till after the 11-second mark. An eyetracking study by usability expert Jakob Nielsen found that delaying banner load by 8 seconds resulted in the banner being virtually ignored when it finally showed up.

Now let’s watch it as a video:

So how do we make sense of the fact that, according to the Luxury Daily article, the Roll-Royce website had a response time of 0.169 seconds, while my tests showed the page didn’t start to show up in the browser until almost 3 seconds, and didn’t fully load until more than 13 seconds had passed? We need to define our terms.

Four key performance measurement terms explained (so that normal people can understand them)

First, I want to be straight about the fact that I don’t think SmartBear was trying to mislead anyone with their numbers. Without knowing how SmartBear defined “response time” in their tests, it’s impossible to comment on their results. Because of this vagueness, I think Ms. Lamb has made two understandable mistakes — mistakes I encounter frequently when I talk about performance outside the geek zone:

• Using “response time” and “load time” interchangeably.
  
• Not realizing that “response time” can mean any number of completely different things.

There isn’t a lot of effort to educate the lay public — such as journalists, and even customers — about what these terms mean. To address this problem, here’s a simple guide to understanding fundamental website performance measurement terms, and when and why you should care about each.

Response time

What it means: Response time is incredibly tricky, and it causes a lot of the confusion I encounter. It can refer to any number of things, depending on whom you ask: server-side response time, end-user response time, HTML response time, time to last byte with no bandwidth/latency, and on and on. Long story short: There’s no single definition.

Caveats: If someone starts talking to you about response time, ask them to clarify which response time they mean. Be wary of anyone who tries to sell you on the idea that there’s only one definition. If user experience matters to you, ask how whatever type of response time you’re looking at relates to what the end user actually sees.

When it’s useful: Different types of response time measurements tell you different things, from the health of your back end to when content starts to populate the browser. As I’ve already said — and it bears repeating — you need to know what you’re measuring and why.

Time to first byte

What it means: Time to first byte is measured from the time the request is made to the host server to the time the first byte of the response is received by the browser.

Caveats: Time to first byte doesn’t really mean anything when it comes to understanding the user experience, because the user still isn’t seeing anything in the browser.

When it’s useful: For detecting back-end problems. If your website’s time to first byte is more than 100 milliseconds or so, it means you have back-end issues that need to be examined. (Web performance consultant Andrew King has written a good post about this, as has Google performance expert Pat Meenan.)

Start render

What it means: As its name suggests, “start render” indicates when content begins to display in the user’s browser. This term seems to have evolved as an alternative to “end-user response time”, but it’s not yet widely used outside of hardcore performance circles.

Caveats: Doesn’t indicate whether the first content to populate the browser is useful or important, or simply ads and widgets.

When it’s useful: When measuring large batches of pages, or performance of the same page over time, it’s good to keep an eye on this number. Ideally, visitors should start seeing usable content within 2 seconds. If your start render times are higher than this, you need to take a closer look.

Load time

What it means: The time it takes for all page resources to render in the browser — from those you can see, such as text and images, to those you can’t, such as third-party analytics scripts. (Geek version: “Load time” is also known as “document complete time” or “onLoad time”. It’s measured when the browser fires something called an “onLoad event” after all the page resources have fully loaded. No matter what you call it, it’s used as a primary measuring stick for site performance.)

Caveats: Needs to be taken with a grain of salt, because it isn’t an indicator of when a site begins to be interactive. A site with a load time of 10 seconds can be almost fully interactive in the first 5 seconds. That’s because load time can be inflated by third-party scripts, such as analytics, which users can’t even see.

When it’s useful: Load time is handy when measuring and analyzing large batches of websites, because it can give you a sense of larger performance trends.

Three things to remember:

1. There’s no single “right” way to measure performance. Each measurement tells you something meaningful about how your site performs.
2. You need to understand the different performance measurement terms so that you can interpret your own data. If you don’t, sad to say some people will take advantage of your ignorance to mislead you for their own benefit. (For example, it’s a little-known fact that some performance vendors have convinced site owners to tie bonuses for key employees to backbone test results, which do not measure real-world performance.)
3. As a matter of due course, you always need to gather large batches of data and rely on median numbers. But you also need to periodically get under the hood and take a real-world look at how your pages behave for real users.

*WebPagetest is a third-party tool that simulates how fast a site loads for real-world users using a variety of browsers. In this set of tests, I looked at how fast each site would load for a person using Internet Explorer 8 over a DSL connection via the WebPagetest server in Dulles, VA.

Related posts:

• Case Study: How Symantec is showing its visitors the wrong content first
• Waterfalls 101: How to understand your website’s performance via waterfall chart
• Why the performance measurement island you trust is sinking