Shop.org Performance Index: Should you care how you rank?

I’m on my way to the Shop.org Annual Summit and looking forward to sitting in on some promising-sounding sessions. The directory of speakers is a who’s who of the internet retail community, and since I have some time to kill en route, I thought I’d see how their sites stack up performance-wise.

Here’s my unofficial Shop.org performance index:*

Website First view Repeat view
eBay 1.511 0.654
Guitar Center 2.341 1.883
Best Buy 2.923 0.568
Patagonia 2.979 1.331
Kenneth Cole 3.181 1.243
Crate & Barrel 3.214 1.311
Jones New York 3.344 1.788
Marks & Spencer 3.749 3.762
Vintage Tub & Bath 3.828 2.620
Groupon (San Francisco) 4.224 5.257
Shoe Buy 4.255 2.793
Like.com 4.513 0.914
The Gap 4.812 1.103
Whole Foods 4.851 0.944
Wine.com 5.014 2.747
REI 5.292 1.708
Office Depot 5.459 2.281
Shop.org performance index 5.914 2.324
The Vitamin Shoppe 6.399 2.276
Sears 6.449 2.546
Barney’s 6.758 1.328
Southwest Airlines 7.790 1.492
Staples 8.098 2.815
Crocs 9.773 3.510
Lacoste 10.400 6.520
House of Fraser** 10.409 1.425
The Wet Seal 10.866 5.090
Godiva Chocolatiers 17.236 2.849

Benchmarks: Helpful or not?

If you read last week’s post about setting audacious performance goals, right now you may be thinking, “Hey, Josh, what’s up with the flipflopping? Didn’t you just say that indices and benchmarks aren’t all that helpful?”

To which I will respond, “Not exactly.”

How benchmarks and performance indices mislead us

This little exercise serves to illustrate how averages and benchmarks can be misleading. In this case, a small number of slow-performing sites inflated the average page load time to almost 6 seconds. If someone were to focus on just this so-called benchmark and consider their site a performance success if it loaded in 5.914 seconds, they’d be wrong. Almost two-thirds of the sites tested performed faster than theirs.

But benchmarks are useful to a point

As I said last week, benchmarks give us a sense of our place in the world, and how we perform relative to others. They’re a good starting point for self-analysis. But to get the most out of a benchmark or an index, you need to look at the entire thing, not just a single number.

In this case, ignore the underwhelming average load time of 5.914. Instead, take a moment to appreciate how many of these sites loaded in 2 or 3 seconds. That’s where you want to be — among the pack leaders. Not average.

Now note the leader, eBay, which had a very respectable page load time of 1.511 seconds. This is to be expected — I mean, it’s eBay, after all. But look who ranks second: Guitar Center. Have you heard of Guitar Center? I hadn’t heard of Guitar Center till today. But despite not being an eBay or a Google or a Microsoft or an Amazon, Guitar Center’s home page loads in a very respectable 2.341 seconds. And I bet it got there because someone at Guitar Center had an audacious goal.

And that’s why benchmarks can be good, but audacious goals are better.

*All tests but one conducted using Webpagetest. Testing on DSL and IE8 via the server in Dulles, VA.
**House of Fraser is hosted in the UK, so was tested on DSL and IE7 via the server in Gloucester, UK.

Related posts: