Three kinds of web page test visuals, and the best ways to use them

• Tweet

With Black Friday around the corner (or happening right now, if you want to jump the gun), what kind of performance blog would this be if it didn’t subject a bunch of leading ecommerce sites to some uncomfortable scrutiny? Continuing in the vein of last week’s TechCrunch post, I wanted to take this opportunity to share more ways to present performance data in a way that is clear, meaningful and compelling.

But first, a bit about this set of tests…

When choosing subjects, I turned to this Bloomberg article on the top 20 ecommerce sites of the 2009 holiday shopping season. Their list takes into account traffic numbers, online sales data, and a consumer survey that ranked sites based on criteria such as ease of use, value, variety, and customer service. I ran a bunch of tests using Webpagetest, focusing on time to interact (aka the amount of time it takes for the document to be loaded and fully interactive) and time to start drawing (aka the amount of time it takes for meaningful content to appear).

Let’s look at three different approaches to presenting the resulting data.

1. Rows and rows of numbers

Four observations about these numbers:

1. The disparity between time to start drawing and time to interact was considerable in many cases. For instance, if you focused just on Newegg’s 9.258s time to interact, you’d judge it to be one of the slowest sites. But in actuality, Newegg started to display meaningful content at 2.5 seconds, making it one of the fastest, according to that metric.
2. Having said that, about one-third of these sites didn’t display any meaningful content until the page had almost fully rendered.
3. And two of these pages — Toys’R'Us and Nordstrom — didn’t deliver meaningful content until AFTER the document was complete.
4. Tables are a really underwhelming way to present compelling information.

Would you have made observations #1-3 based solely on looking at this table? Probably. Would everyone in your organization take the same amount of care in scrutinizing these numbers? Probably not.

When to use tables: Tables are a way of showing that you’ve done your homework and tabulated your data. While necessary in the appendix of a performance report, they’re not the most effective way to light fires under butts.

Now let’s look at another way of seeing the same data.

2.  Bird’s eye view: One image to rule them all

You’ll want to enlarge this image to take it in. It’s a set of timelapse slideshows showing how the page elements loaded in 0.5s increments for each of the same 20 sites.

What’s great about this image is that it doesn’t need a lot of verbiage. You can see what you need to see in one quick glance:

• Lots of screens = bad
• Lots of empty white screens = really bad
• Screens that populate early = good

This kind of visual sells the value of performance tuning more than any table of numbers ever could. If you’re in the process of benchmarking your site and doing a competitive analysis, I strongly recommend you generate an image like this one. (It’s easy to do using Webpagetest’s visual comparison feature. First, run your side-by-side test, then click the “Export filmstrip as an image” text link on the bottom of the results page.)

When to use a bird’s eye view graphic: In the competitive analysis section of your performance audit.

And now let’s look at a really dramatic way to compare site speed.

3. Side-by-side video

In this case, I chose the fastest and the slowest pages from the list of sites above, in terms of how quick they were to start drawing.

When I show clients how their sites perform compared to their competitors, this kind of visual is always a showstopper. I mean, which site would you rather own?

When to use a video: To kick-start a presentation or lead a performance report.

Five major takeaways from this exercise:

1. Always analyze your site’s performance from the perspective of real users.
2. Benchmark yourself outside your industry. If you’re a shoe retailer, you don’t just want to know how your site compares to other shoe retailers. You also need to know how your site compares to other ecommerce sites your visitors are likely to shop at. Collectively, these sites are helping create the performance standard.
3. Load time is not the best metric for judging your site’s perceived speed.
4. If you’re forced to serve object-laden pages, serve your most meaningful content first.
5. When you’re presenting performance data to stakeholders in your organization, remember that different people respond to different kinds of visuals. Graphs work for some, videos for others. Cover all your bases in order to make sure your audience gets the visuals that work best for them.

*All tests conducted on Webpagetest — IE7 on DSL via the server in Dulles, VA.

Related posts:

• TechCrunch: The slowest tech blog, or one of the fastest? Turns out, it’s both.
• Shop.org Performance Index: Should you care how you rank?
• Case Study: How Symantec is showing its visitors the wrong content first