I’m on my way to the Shop.org Annual Summit and looking forward to sitting in on some promising-sounding sessions. The directory of speakers is a who’s who of the internet retail community, and since I have some time to kill en route, I thought I’d see how their sites stack up performance-wise.
Here’s my unofficial Shop.org performance index:*
|Website||First view||Repeat view|
|Crate & Barrel||3.214||1.311|
|Jones New York||3.344||1.788|
|Marks & Spencer||3.749||3.762|
|Vintage Tub & Bath||3.828||2.620|
|Groupon (San Francisco)||4.224||5.257|
|Shop.org performance index||5.914||2.324|
|The Vitamin Shoppe||6.399||2.276|
|House of Fraser**||10.409||1.425|
|The Wet Seal||10.866||5.090|
Benchmarks: Helpful or not?
If you read last week’s post about setting audacious performance goals, right now you may be thinking, “Hey, Josh, what’s up with the flipflopping? Didn’t you just say that indices and benchmarks aren’t all that helpful?”
To which I will respond, “Not exactly.”
How benchmarks and performance indices mislead us
This little exercise serves to illustrate how averages and benchmarks can be misleading. In this case, a small number of slow-performing sites inflated the average page load time to almost 6 seconds. If someone were to focus on just this so-called benchmark and consider their site a performance success if it loaded in 5.914 seconds, they’d be wrong. Almost two-thirds of the sites tested performed faster than theirs.
But benchmarks are useful to a point
As I said last week, benchmarks give us a sense of our place in the world, and how we perform relative to others. They’re a good starting point for self-analysis. But to get the most out of a benchmark or an index, you need to look at the entire thing, not just a single number.
In this case, ignore the underwhelming average load time of 5.914. Instead, take a moment to appreciate how many of these sites loaded in 2 or 3 seconds. That’s where you want to be — among the pack leaders. Not average.
Now note the leader, eBay, which had a very respectable page load time of 1.511 seconds. This is to be expected — I mean, it’s eBay, after all. But look who ranks second: Guitar Center. Have you heard of Guitar Center? I hadn’t heard of Guitar Center till today. But despite not being an eBay or a Google or a Microsoft or an Amazon, Guitar Center’s home page loads in a very respectable 2.341 seconds. And I bet it got there because someone at Guitar Center had an audacious goal.
And that’s why benchmarks can be good, but audacious goals are better.
*All tests but one conducted using Webpagetest. Testing on DSL and IE8 via the server in Dulles, VA.
**House of Fraser is hosted in the UK, so was tested on DSL and IE7 via the server in Gloucester, UK.