HTTPWatch

4 awesome slides showing how page speed correlates to business metrics at Walmart.com

28 Feb 2012

42 Comments

When one of the first slides in a presentation offers this level of candor, you know you’re in for a treat:

Walmart web performance optimization case study

File this presentation under “I wish I’d been there in person.” Cliff Crocker, Aaron Kulick, and Balaji Ram joined forces at February’s SF Web Performance Meetup to tell a RUM (real user monitoring) story through the lens of three job functions at Walmart.com: the performance data analyst, the developer, and the business analyst.

You can check out the full slide deck here (you need to be a Minus member to view it online, but anyone can download it), but I want to highlight my favourite slides.

First, some back story.

The problem: Folks at Walmart knew their pages were slow. As a for instance, initial measurement showed that an item page took about 24 seconds to load for the slowest 5% of users. Why? The usual culprits: too many page elements, slow third-party scripts, multiple hosts (25% of page content is served by partners, affiliates, and Marketplace), and various best practice no-nos.

The goal: Create and meet a performance SLA that would see Walmart’s 95th percentile traffic hit 20 seconds.

The approach: To get to that goal, they dedicated a scrum team to one sprint of performance optimization. At the start of the process, the team performed some baseline measurements in which they used their RUM data to look at the load times of key pages and look for patterns. Then the team would create targets for page performance and at the end of the sprint measure the impact of optimization on key metrics.

The following slides are a great use of historical RUM data to make a powerful case (embodied in compelling graphs) for investing in performance. Throughout this set, there’s a clear, consistent correlation  between load time and conversion.

Awesome slide #1: Overall, converted shoppers received pages that loaded 2X faster than non-converted shoppers.

This should be of interest to anyone who’s asked themselves how to set targets for their own performance.

Walmart web performance optimization case study

Awesome slide #2: The above trend persists, even on individual pages that experience greater load times.

An example is show here, in which a specific page loaded almost 2 seconds faster for buyers than it did for non-buyers. (There’s nothing I love more than seeing parallel lines on a graph like this.)

Walmart web performance optimization case study

Awesome slide #3: Non-buyers were served category pages that were 2-3 seconds slower than buyers.

I really appreciate the comprehensiveness of this set of slides. It’s a good proxy for measuring page flows, which is a much trickier task, due to the massive number of permutations involved. (Learn how you can do something similar using WebPagetest.)

Walmart web performance optimization case study

Awesome slide #4: Bounce rate strongly correlates to page speed.

No huge surprise here, but great to see this validated.

Walmart web performance optimization case study

Conclusions: Optimization resulted in improved conversions, revenue, and SEO

You may be happy to know that the Walmart team not only hit their SLA at the end of their optimization sprint, they actually overreached it by almost 3 seconds. They note the following benefits:

  • For every 1 second of improvement they experienced up to a 2% increase in conversions
  • For every 100 ms of improvement, they grew incremental revenue by up to 1%
  • SEO benefits for entry pages and reduce bounces

Do it yourself

Check out the original slide deck to learn how Walmart used RUM tools, first to baseline performance and its benefits, and then to measure the results of optimization.

If RUM isn’t an option for you, then read this post to learn how to identify and measure the performance of sample page flows using WebPagetest or HTTPwatch. And then watch this short video to learn how to use Google Analytics to perform a page speed/revenue analysis of the pages in your flows.

Before I sign off, I want to applaud the team at Walmart for providing this level of disclosure. Case studies like this are a massive boon for our industry. It’s always inspiring to see e-commerce giants joining the likes of Amazon and Shopzilla in their willingness to share data.

Related posts:

By: Tammy Everts Category: Tags: , , , , , ,

Are you focusing on first and repeat views? Then you may be ignoring 96% of your page traffic.

10 May 2011

11 Comments

Question: What do you think these four waterfalls have in common? (Hint: Same browser. Same connection speed. Same location.)

Performance measurement - first, repeat, and flow views

Answer: They’re all the same page (the Lonely Planet home page, to be specific), but from four distinct perspectives: first view, repeat view, and two different flow views.

Why measuring first view and repeat view is not enough

When most of us run simple performance tests, we usually focus on two perspectives: the first view and the repeat view. Unfortunately, life is not this simple.

On non-landing pages (for most of you, that would be all pages but your home page), first view and repeat view only represent about 4% of the total views. There’s another view — the flow view — which represents approximately 96% of the traffic for most of your web pages.

And yet despite its incredible relevance, flow view gets almost zero attention.

For instance: Let’s look at a Lonely Planet product page

Imagine you go to the Lonely Planet website and want to buy the West Africa travel guide. Well, that page has a number of different waterfalls, depending on how you visit it:

1. First view

What is it?

This is the view of the page for someone who has never been to the Lonely Planet site before (or has cleared their cache since their last visit).

Performance measurement - web page test results

Likelihood your users will see this waterfall: Small (but still important)

Product pages are most likely only a first view page when they are linked directly from ads, searches, or other websites. It is very rare for product page to be typed directly into the browsers address bar. Looking at our analytics warehouse data, we see that product pages are viewed as first view pages only 3% of the time, on average. But still, these are important views because they are often targeted leads coming from search results and referral engines.

How to see first view in a waterfall

This is an easy one. Just go to WebPagetest, enter the URL, and click ‘Start test’. Look at the first view.

2. Repeat view

What is it?

This is the view of a page if a user goes only to the page, closes their browser, and then reopens the browser and goes only to that page again as the first page they hit on the site.

Peformance measurement - waterfall

Likelihood your users will see this waterfall: Almost never

Repeat views in the context of most test tools are a good proxy to see how well you are using cache headers, but users very rarely just go to one page on a site, close the browser, open it again and go back to that exact page.

How to see repeat view in a waterfall

This is another simple view to get. As with first view, go to WebPagetest, enter the URL, and click ‘Start test’. Look at the repeat view.

3. Flow view

What is it?

This is the view of the page when a customer has previously visited at least one other Lonely Planet page.

Likelihood your users will see this waterfall: Frequently

Product pages are often viewed as part of a flow by your users. Looking at our own customer analytics here at Strangeloop, I’d estimate that upwards of 96% of product page views are flow views.

How to see flow view in a waterfall: Two methods

Seeing a page in a flow is not as straightforward as first or repeat views. Obviously your users don’t all take the same flow, so you need to ensure that you test multiple permutations of pages based on the key flows through your site, which you see in your analytics tool.

Method #1: Using HTTPWatch
The easiest way to see the flow view of a page is to install HTTPWatch, start recording your browsing session, and click from the home page to the product page.

Method #2: Using WebPagetest
Another way to see the flow view is to use the script feature of WebPagetest. Click on ‘Advanced settings’ and then click on ‘Script’.

Customizing web page speed tests using scripts

Create a script by copying and pasting the example below and changing the URLs. (If your typical path has more or less steps, feel free to cut/copy to accommodate your needs. Remember that the URL below logData 1 will be captured as the waterfall for this page.)

logData 0
navigate http://www.lonelyplanet.com/us
logData 0
navigate http://www.lonelyplanet.com/destinations
logData 0
navigate http://www.lonelyplanet.com/africa
logData 0
navigate http://www.lonelyplanet.com/benin
logData 1
navigate http://shop.lonelyplanet.com/africa/west-africa-travel-guide-7

With hundreds of flows to choose from, how do you narrow down the most relevant ones?

Obviously you could have hundreds of permutations of flows. I was interested in tracking down just how many different waterfalls can be generated from one page.

Using our performance analytics database, I tracked down a specific product page and looked at all of the different flows. The specific page I identified had been viewed almost 25,000 times and had been viewed in 975 different combinations — far too many combinations to run individuals waterfalls on.

So instead of running hundreds of tests, I simply decided to group the pages into different templates, then look at the different flows.

For example:

Home page -> Destinations -> Africa -> Benin -> West Africa Travel Guide

Was translated to:

Home page -> Destinations -> Continent -> Country -> Book

I found that by using this method, I got down to a much more manageable set of 10 unique, relevant flows.

I then performed WebPagetests, using the script feature in WebPagetest that I described above. I found that looking at the first view, repeat view, and important flow view waterfalls really helped me see how the site was doing and how real users would view a page.

What this all serves to illustrate

Measuring performance is an art as much as it is a science. While there are a growing number of tools on the market, we need to be aware of what exactly it is we need to test and why. And we still have to be ready to perform some hacks to get the data we need.

Related posts:

 

By: Tammy Everts Category: Tags: , , ,

The three biggest performance problems with third-party content (and how to fix them)

17 Aug 2010

11 Comments

[This post served as the jumping off point for a 30-minute webinar: Three Common Performance Problems Caused by Third-Party Content (And How to Fix Them).]

“Necessary evil” is too strong a phrase when it comes to describing third-party ads, analytics tools, and various other site widgets. But given how much negative talk has been centered around third-party content and web performance lately, it’s not that far off.

Traditionally we have divided the site performance problem into a front-end and back-end problem. It is becoming obvious to me as I speak to more and more customers that we need to refine this high-level taxonomy to include:

  • Front end (time that you control)
  • Back end
  • Third-party content

If your problems are at the front or back end, you have control over them. But if your problem is with third-party content, pretty much your only actionable strategy is to be smart about how you pick your vendors. (If you consider prayer an action, you can also hope and pray that your existing vendors make performance a priority. Meebo and Digg deserve special mention here for doing this, with great results.)

Most of the third-party problems I see fall into three categories:

Problem #1: Tools that promise cool new functionality, but kill your conversion gain

I’ll give you an example. At Strangeloop, we have an ecommerce client who wanted to implement a third-party recommendation engine on his site. The engine claimed to be “only one line of javascript” that would increase cart size by recommending related products. Sounds good. But that one line of code and its resulting roundtrips slowed down the site by 15%, and that slowdown leeched the conversion gains generated by the tool.

Solution: Conduct a cost-benefit analysis for all prospective tools.
This is something you should do any time you evaluate a new third-party app:

  1. Perform an A/B test of your site, with and without the tool, in a real-world environment. Generate waterfall charts for both tests, and identify how long the third-party objects take to load. Note these benchmarks.
  2. From the tool vendor, get the number for the average conversion rate bump experienced by other sites that use the tool.
  3. Using Aberdeen’s widely accepted performance stat that a 1-second page delay equals a 7% loss in conversions, calculate the potential net conversion gain or loss. For example, if a tool slows down page load by 2 seconds, that means a 14% conversion loss. But if that same tool promises a 20% conversion increase, then that’s a net gain of 6% (not including the cost of purchasing the tool).

Problem #2: Badly optimized ads

Here’s another client example. One company we helped did a performance audit and found that third-party ads comprised a whopping 50% of the calls to their pages. They were averaging 80 to 100 roundtrips per page because of the company that was serving the ads.

Solution: Think about speed when choosing your ad vendor.
When selecting a vendor, ask them how they optimize their ads to maximize performance. If they don’t have an answer for you, consider giving them a pass. (I know this isn’t always an option, and sometimes there are very good reasons to go with a company despite the performance of its content, but it’s worth keeping in mind.)

Problem #3: Badly optimized pages

I’ve talked before about how all browsers perform differently in terms of how they prioritize page elements, and about how loading elements in the wrong order can mean that your visitors’ eyeballs don’t land on the content you want them to notice. It’s worth repeating.

Solution: Prioritize page elements so that third-party content loads last, not first.
You need to look at how your key pages render across major browser types, identify which page elements are rendering first, and make sure those are the elements you most want your users to see. This might not always be possible – or desirable, if you rely on ad clicks – but it’s a strategy to bear in mind.

Further reading:

Whether you’re a developer designing third-party apps or a site owner who is choosing them, here are a few good links for further reading:

Related links:

By: Tammy Everts Category: Tags: , , , , , , , ,

Page Test Tools: Which results should you trust?

21 Jul 2010

16 Comments

When I work with a client to help them understand what kind of performance gains they can expect with Strangeloop, one of the first things we do is benchmark their site’s current performance. Right out of the gate, this can prove to be a major stumbling block, because no two tests return the same results. Part of my job is walking people through the various results and explaining which data is meaningful for their purposes, and which isn’t.

To illustrate, I put a high-traffic site, Target.com, through its paces on a handful of commonly used tests — Gomez, Keynote, HTTPWatch and Webpagetest* — to see how it performs on first views and, where possible, repeat views (click through the links below to see the test results):

Test Page load time
(first view)
 Page load time
(repeat view)
Keynote (backbone) 1.383 –
Gomez (backbone) 1.932 –
HTTPWatch:
- Firefox 3.6.6 10.180 3.886
- IE8 8.920 4.605
Webpagetest:
- IE7 – DSL 8.696 4.985
- IE7 – FIOS 4.422 4.172
- IE8 – DSL 5.733 3.595
- IE8 – FIOS 3.407 2.491

The first column – page load time (first view) – is the time it took for the Target.com home page to fully load for first-time visitors. The second column – page load time (repeat view) – shows how long it took for the home page to load for a previous visitor whose cache had not been cleared.

Both these columns are meaningful. You definitely want new visitors to your site to have a good experience, given the evidence that 80% of first-time visitors who have a poor experience will not return. But you also need to give your repeat visitors an excellent experience, because studies indicate that they have higher expectations of your site than newcomers do.

Today’s focus: first-time visitors

For the purposes of this post, however, I’m going to focus on first-time visitors, as this is the metric that most people are initially interested in, and we know this metric has a dramatic effect on conversion. (See the third graph in this post for evidence.)

As I’ve written about in the past, the emerging benchmark is a sub-2-second page load time. In this set of tests, the numbers ranged from 1.383 seconds for Gomez to 10.180 seconds for HTTPWatch (on Firefox from my office).

But how is this discrepancy possible? And which results are most applicable? To answer these questions, you have to understand one thing:

Different page test tools are designed to test in different environments

If you’re a Gomez or a Keynote client, then you are most likely using their backbone tests, named for the fact that they take place over the backbone of the internet. Backbone tests tell you how fast your site loads at major internet hubs, but because you’re skipping the “last mile” between the server and the user’s browser, you’re not seeing how your site actually performs in the real world.

If you’re testing with HTTPWatch, then you’re finding out how a website performs on your own desktop. If you’re testing while at work, using your company’s souped-up connection, then you’re going to get zippy results. Testing the same site at home on your shared DSL line, while your neighbor is downloading seasons one through four of Lost… well, that’s going to be a much less zippy experience. HTTPWatch is a good way to test how a site works for you, but not necessarily how it works for your users.

If you’re running Webpagetest, you’re testing in an environment that attempts to simulate real-world user environments and browser behavior. For instance, if your users live in mid-sized urban locations, you can test from Webpagetest’s servers in Dulles, Virginia (which is where I ran these tests). If your users are overseas, you can test from international servers spanning the UK, China and New Zealand.

In order to understand which test is best for you, you need to understand your real users’ environment.

Which brings me to my next point…

How to identify your users’ environment

You need to find out three things:

  1. Where your users live. Are they in major urban centers? Are they overseas? Do they live in small towns?
  2. What kind of browser are they using?
  3. What kind of internet connection are they using. Cable? DSL? T1? You might be surprised.

All this information is readily available through your analytics tools. After you’ve studied it, come up with a technical profile (or set of profiles) for your users. Then use this profile to customize your page test parameters to get the most accurate results for your site. That’s your benchmark.

So what is the best test for Target.com?

Without knowing the exact user breakdown for Target.com, I would assume that, like many of our key ecommerce customers, they fit the following profile:

  • Live in the US in the suburbs
  • 60-65% use IE7/8
  • Most connect from home over DSL/ADSL

Based on these assumptions, I would recommend to the Target executives they look very closely at the IE7 and IE8 Webpagetest results (which showed first-time load times in the 6-9 seconds range) and I would suggest that investing in performance would really help.

*Test parameters:
Gomez: Tested last mile and backbone, running one test per hour for six hours across five points in North America: LA, Miami, Atlanta, New York and Seattle.
Keynote: Same as Gomez.
HTTPWatch: Tested the site on Firefox 3.6.6 and  IE8 from my office in Vancouver.
Webpagetest: Tested the site on IE7 and IE8 on both DSL and FIOS networks. Averaged three runs via server hosted in Dulles, VA.

Related links:

By: Tammy Everts Category: Tags: , , , , , , ,

Waterfalls 401: Advanced analysis for non-beginners

14 Jul 2010

4 Comments

Last week, I gave a layperson-friendly breakdown of how waterfall charts work. If you want to get into waterfalls in more depth, Strangeloop’s VP Product, Hooman Beheshti, ran an excellent webinar a while back, in which he analyzes Google’s home page performance using Web Page Test and HTTPWatch. Not only does Hooman demonstrate how to interpret waterfalls, he also talks about ways to diagnose and isolate common web application performance problems.

Click the image below to launch the webinar.

Webinar: Diagnosing web application performance pain

Webinar: Waterfalls 401

Related links:

By: Tammy Everts Category: Tags: , , , , , , ,