Crowdsourced website performance tuning

Yesterday on Reddit:

Today on Reddit:

Today in Subway.com’s source code:

Makes me wonder what would happen if the performance community subjected a new site to this treatment every day.

Related posts:

TechCrunch: The slowest tech blog, or one of the fastest? Turns out, it’s both.

When I started writing this post, my intent was to do a bit of performance benchmarking for leading tech blogs. But what started off as a standard benchmarking exercise turned into a great opportunity to talk about another approach to understanding your performance numbers in a meaningful way, and why it’s important to look at your site from as many angles as possible.

Shortly after TechCrunch announced its acquisition by AOL yesterday, some folks in the performance community were quick to suggest that one of AOL’s first jobs should be to improve the site’s performance. People cited that the main page that takes more than 20 seconds to load, making TechCrunch a desperate candidate for performance tuning.

But TechCrunch isn’t the only tech blog out there that readers complain about. So I decided to run some page tests to see how they measure up.

Time to interact (aka “load time”)

When you do a test via Webpagetest, this is called “load time”. Up till recently I’ve been using this term, too, but lately I’ve started calling this “time to interact”. It refers to the amount of time before the document is complete and fully interactive. Whatever you call it, this is the number most people pay attention to.

Website Time to interact (first view)*
TechCrunch 30.168
GigaOM 19.556
LA Times: Technology Blog 19.448
Mashable 17.868
New York Times: Bits Blog 15.297
Technology Review 14.973
Average 19.551 seconds

This is one of the worst industry benchmarks I’ve ever encountered. In fact, it’s suspiciously bad. There’s no way sites this slow could have such dedicated readerships.

I wanted to take another view, so I used Webpagetest’s video function to create a set of side-by-side videos to show how these sites load in real time (I’ve slowed it down so you can appreciate it in all its incremental glory):

This video reveals a couple of perspectives that you would miss if you just focused on load time…

Time to fully load: Interesting but not necessarily important

This is the amount of time it takes for every page element to load.

Website Time to interact Time to fully load
TechCrunch 30.168 29.5
GigaOM 19.556 21.9
LA Times: Technology Blog 19.448 26.8
Mashable 17.868 39.6
New York Times: Bits Blog 15.297 29.1
Technology Review 14.973 17.2
Averages 19.551 27.35

These numbers aren’t a huge surprise, given that the number of calls to the server for these sites ranges between 129 (Technology Review) and 323 (TechCrunch).

Before I started this entire exercise, I made a blindfolded guess about the performance culprits for most of these sites: ads and other bits of third-party gak. But as I’ve discussed elsewhere, a decently optimized page can sort and prioritize third-party content so that visitors are served meaningful content first. So let’s add those numbers to our table…

Time to start drawing: Very important

I define “time to start drawing” as the amount of time it takes for meaningful content to appear on the screen. Let’s take a look:

Website Time to interact Time to fully load Time to start drawing
TechCrunch 30.168 29.5 5.8
GigaOM 19.556 21.9 6.4
LA Times: Technology Blog 19.448 26.8 6.1
Mashable 17.868 39.6 6.5
New York Times: Bits Blog 15.297 29.1 7.2
Technology Review 14.973 17.2 5.1
Averages 19.551 27.35 6.18

Now, I’m not saying that an average time of 6.18 seconds is good. There’s clearly room for improvement here. But it’s not outright terrible, and it’s lower than the Fortune 500 benchmark of around 7 seconds.

And check it out: at 5.8 seconds, TechCrunch is actually the second-fastest site surveyed, not the worst, as the initial page test results indicated.

This exercise serves as a reminder of three things:

  1. The danger of focusing on just one piece of performance data
  2. The value of benchmarking your site against your competitors (though I’m still a staunch proponent of the audacious goal)
  3. And most important: always making sure you look at your site from your visitors’ perspective

Edited 9/30/10 to add: I just realized that I accidentally published the wrong numbers in the “time to interact” columns. I had originally tested the sites on both IE7 and IE8, but decided in the end to focus on the results for IE7. However, I mistakenly entered the IE8 numbers in the “time to interact” column. Duly corrected now, with my apologies for any confusion. Note that the updated numbers still corroborate the thesis behind this post.

*All tests conducted on Webpagetest — IE7 on DSL via the server in Dulles, VA.

Related posts:

Shop.org Performance Index: Should you care how you rank?

I’m on my way to the Shop.org Annual Summit and looking forward to sitting in on some promising-sounding sessions. The directory of speakers is a who’s who of the internet retail community, and since I have some time to kill en route, I thought I’d see how their sites stack up performance-wise.

Here’s my unofficial Shop.org performance index:*

Website First view Repeat view
eBay 1.511 0.654
Guitar Center 2.341 1.883
Best Buy 2.923 0.568
Patagonia 2.979 1.331
Kenneth Cole 3.181 1.243
Crate & Barrel 3.214 1.311
Jones New York 3.344 1.788
Marks & Spencer 3.749 3.762
Vintage Tub & Bath 3.828 2.620
Groupon (San Francisco) 4.224 5.257
Shoe Buy 4.255 2.793
Like.com 4.513 0.914
The Gap 4.812 1.103
Whole Foods 4.851 0.944
Wine.com 5.014 2.747
REI 5.292 1.708
Office Depot 5.459 2.281
Shop.org performance index 5.914 2.324
The Vitamin Shoppe 6.399 2.276
Sears 6.449 2.546
Barney’s 6.758 1.328
Southwest Airlines 7.790 1.492
Staples 8.098 2.815
Crocs 9.773 3.510
Lacoste 10.400 6.520
House of Fraser** 10.409 1.425
The Wet Seal 10.866 5.090
Godiva Chocolatiers 17.236 2.849

Benchmarks: Helpful or not?

If you read last week’s post about setting audacious performance goals, right now you may be thinking, “Hey, Josh, what’s up with the flipflopping? Didn’t you just say that indices and benchmarks aren’t all that helpful?”

To which I will respond, “Not exactly.”

How benchmarks and performance indices mislead us

This little exercise serves to illustrate how averages and benchmarks can be misleading. In this case, a small number of slow-performing sites inflated the average page load time to almost 6 seconds. If someone were to focus on just this so-called benchmark and consider their site a performance success if it loaded in 5.914 seconds, they’d be wrong. Almost two-thirds of the sites tested performed faster than theirs.

But benchmarks are useful to a point

As I said last week, benchmarks give us a sense of our place in the world, and how we perform relative to others. They’re a good starting point for self-analysis. But to get the most out of a benchmark or an index, you need to look at the entire thing, not just a single number.

In this case, ignore the underwhelming average load time of 5.914. Instead, take a moment to appreciate how many of these sites loaded in 2 or 3 seconds. That’s where you want to be — among the pack leaders. Not average.

Now note the leader, eBay, which had a very respectable page load time of 1.511 seconds. This is to be expected — I mean, it’s eBay, after all. But look who ranks second: Guitar Center. Have you heard of Guitar Center? I hadn’t heard of Guitar Center till today. But despite not being an eBay or a Google or a Microsoft or an Amazon, Guitar Center’s home page loads in a very respectable 2.341 seconds. And I bet it got there because someone at Guitar Center had an audacious goal.

And that’s why benchmarks can be good, but audacious goals are better.

*All tests but one conducted using Webpagetest. Testing on DSL and IE8 via the server in Dulles, VA.
**House of Fraser is hosted in the UK, so was tested on DSL and IE7 via the server in Gloucester, UK.

Related posts:

Are your website’s performance goals audacious enough?

“We want you to be able to flick from one page to another as quickly as you can flick a page on a book. So we’re really aiming very, very high here… at something like 100 milliseconds.”

~Urs Hölzle, Senior VP Operations, Google

Urs said this at Velocity this past June (you can hear it at 3:45 of this video), and it’s resonated with me ever since.

I talk a lot about the business value of performance and why there’s no such thing as fast enough. But to my knowledge, Google is the only big company out there that has truly internalized this philosophy. In fact, I was down in Mountain View at the Googleplex a few weeks ago, and I heard on more than a few occasions the 100 ms goal spoken about in real terms — things like “This project will help us get closer to the goal.”

I’m not trying to imply that the folks at Google are the only people who care about performance. If you’re reading this blog, you clearly care. But the folks at Google are the only people I’ve met who have a clearly stated performance goal: all pages on their sites will load in 100 ms or less. In fact, they take it one step further and have the audacity to try and get all pages on the world wide web down to 100 ms.

This is a fascinating contrast to the rest of our industry, which tends to focus on benchmarks as indicators of success:

  • The average website in Keynote’s “Business Top 40″ takes 2.34 seconds to load.
  • The average web page loads in 4.9 seconds.
  • The average Fortune 500 website takes about 7 seconds.

So, do you consider your site successful if it loads in less than 7 seconds? Less than 4.9? 2.34? How do you decide where to take aim? Or do you take the attitude that you’ll just sort of chip away at load time when you have time, and content yourself with the knowledge that everyone else has the same attitude?

Benchmarks are useful tools. They give us a sense of our place in the world, and how we perform relative to others. They’re a good starting point for self-analysis. The problem with benchmarks is when they tempt us to focus solely on how we compare to others, who may be as flawed as we are. Google has wisely stepped outside the benchmark arena and has created a goal that has nothing to do with their competitors and everything to do with the actual people who use their products.

Google’s decision to aim at 100 ms makes sense from a human factors perspective. 100 ms gives us the illusion of instantaneous response. 100 ms makes a web experience feel real.

Is 100 ms possible? No, not right now. Will it be possible? Yes, most definitely, largely through the efforts of people who didn’t settle for benchmarks.

My challenge to you is to ignore benchmarks. Create an audacious goal for your site. Evangelize it in your organization. And take aim.

Related posts:

If children are the most demanding web users, why are their sites so slow?

I have two little boys. They aren’t school-age yet, but I’m already fascinated by everything I’m reading about how technology may be implemented in their future classrooms. Last week, I came across this article on how a pilot iPod Touch program in third-grade classrooms led to radically improved reading and math results on standardized tests.

When you think about it, these results make complete sense. Let’s look at two scenarios:

Scenario A: Standard-issue math class

You get your sheet of problems, which you work out as best you can, and then turn them in to your teacher without knowing which ones you got right and which ones are wrong. Your teacher marks them that evening, and you get your marked-up paper back the next day. The problem with this scenario is that your brain is, by now, a thousand miles away from the problems you got wrong. Most adults are incapable of internalizing delayed feedback efficiently. Why should we expect children be any different?

Scenario B: iPod-equipped math class

Your teacher gives you your math problems. You start working away. Each time you get a question right, you get positive feedback from your device before you’re taken to the next problem. When you make a mistake, you get immediate feedback, too. After receiving a few prompts and tips, if you’re still struggling with a problem, you’re directed to get a bit of immediate coaching from your teacher. He or she comes over, talks you over the hump, you nail the problem, and then move on to the next one.

Which scenario do you think would result in a more successful learning experience?

Since this is a performance blog, you can probably guess where I’m going with this…

When it comes to kids and website performance, how fast is fast enough?

If web- and app-based learning is the wave of the future, designers and developers will need to develop expertise in the unique online behaviour of children. There’s already a lot of data out there, about everything from preferred fonts (14 pt Arial, FYI) to the best kind of “close” button (a big red circle with an X in it).

What there’s not a lot of: data about children and site speed.

It’s surprisingly hard to find this information. Like the Victorians, our tendency seems to be to pretend that children are just miniature grownups, and to assume they have the same expectations and behaviours that we do. Are we as misguided as the Victorians?

I did find this recent usability study by user experience guru Jakob Nielsen, but I have to admit, I balked at shelling out $188 just to read the small section on page load times. Instead, I gleaned what I could from the summary, and then did more digging around the web. Here’s what I found:

  • Research consistently shows that most children under the age of five have an attention span of between 8 and 15 minutes. Many children have even less.
  • Like adults, children are quick to judge a site, and to leave if they deem it no good.
  • While adults have some (if limited) patience with waiting for pages to load, children want instant gratification. They expect to see some kind of picture right away when they hit a button.
  • A recurring theme in many studies is that children tend to wait for images to completely load on a page before navigating to another page. They don’t understand that a complete page load is not needed for the page to be functional. Not surprisingly, all this waiting makes them frustrated and likely to be discouraged from using a site.

So how do you design a fast site for kids?

The obvious challenge in creating a fast user experience for children is in the fact that kid-oriented sites tend to have more, not fewer, bells and whistles than sites for adults. Graphics, animation, third-party apps — how do you serve these nigh-instantaneously?

When I went digging for workarounds, I found an interesting comment on this O’Reilly Radar post from a reader named Preston Austin. He said:

Back when under a few seconds load time was still considered a “good” page and aol dial up mattered a lot and still wreaked havoc with everything (late 90s) I was developing lots of little activities for a very popular children’s web site. We saw traffic in excess of 5 million dynamic hits on a good afternoon and wrote our own stats package – so it was easy to measure fine changes in user behavior.

In an effort to increase kid’s satisfaction with the site, we started experimenting with simplifying pages to get some kind of page showing fast and then precaching subsequent content in certain non-game activities so that things would be gamelike – and happen near instantly on a click.

It was pretty interesting – we discovered that we could readily lengthen all of number of sessions, estimated repeat sessions, length of sessions, number of clicks per session, and most interestingly rate of clicks per session by pre-caching likely next moves while a child was viewing current content (we used various horrible hacks, and tools like Shockwave and later Flash to do this). To really make a difference, we had to get the response to clicks down under about 200ms.

Even though Preston is describing events from the late ’90s, I’m willing to bet that his conclusion — optimal page response time is under 200ms — holds true today. It makes sense given Jakob Nielsen’s persistent findings about how human beings react to page delays:

  • 0.1 seconds gives us the illusion of instantaneous response.
  • 1 second keeps our flow of thought seamless.

200ms doesn’t quite give the illusion of instantaneousness, but it’s well within the threshold for keeping the user’s flow of thought seamless.

Report card: How fast are kids’ sites today?

With that 200ms goal in mind, I ran some quick tests on a handful of well-known kids’ sites.* The results aren’t pretty.

Website Page load time (in seconds)
PBSKids/Games 14.275
Yahoo! Kids 11.140
Sports Illustrated Kids 9.571
Discovery Kids 5.342
Fact Monster 4.927
Average 9.051

Ouch. Clearly, the bar is low. If we want to deliver websites and web-based apps to children, we need to do better than this.

(On a related note, Google recently launched the Family Safety Center, a collection of tools and resources to help parents and educators figure out how to help their kids safely navigate the web. Definitely worth checking out.)

*All tests were conducted using Webpagetest: IE7 on DSL via the server in Dulles, VA.

Related posts: