How to perform a 5-minute page speed/revenue analysis of your ecommerce site

2 Feb 2011

15 Comments

[This was such a popular post that we turned it into a brief (11-minute) webinar. Check it out.]

I proselytize the value of performance for a living. I am a member of the tribe that believes, beyond a shadow of a doubt, that improving the performance of your website will make you more money. I have dozens of stats at my disposal to convince you, and I feel an evangelical need to save non-believers from their errant ways.

Sometimes I’m confronted by non-believers who have the audacity to question my army of statistics and feel that the results at Amazon, Bing, Microsoft, Ebay, Shopzilla, AutoAnything, etc. do not apply to them.

I was confronted with one of these non-believers last week. In this case, he was a non-believer who was open enough to spend time brainstorming how the data could be presented in a way that would be more convincing. I am proud to say that he came out the other side as a serious performance convert. I want to share his conversion story.

“These stats don’t apply to me.”

I hear this a lot and I get it. It’s difficult for mortal companies to see themselves in relation to ecommerce mega-giants like Amazon and Ebay. This is why we put together stats on mortal companies.

But even those stats didn’t feel comparable for my skeptical comrade. He wanted to get an idea for how his site speed affects his users on his site with his content. Fair enough. My challenge was to figure out how to do this.

The obvious answer was to just implement Site Optimizer, speed things up, then check out conversion rate changes using a segmentation test, but this wasn’t an option. He needed to get buy-in before we could run a test implementation.

It was a chicken-or-the-egg conundrum. After much discussion, we decided to find proxies in his current analytics program that might convince him that performance would make him more money.

Using Google Analytics to find proxies for performance

After reading my post on Internet Explorer performance, which showed that IE8 is about 25% faster than IE7 across almost 200 websites, we decided to explore the different conversion behaviour by browser version.

The first thing we did was perform a Webpagetest in IE7 and IE8. We found that his site was 30% faster in IE8.

After he was convinced that his site performed faster in IE8, we went to Google Analytics and explored the numbers for his site. We found a remarkable trend. The value per visitor on his site was 29% higher for IE8 than it was for IE7 — almost exactly matching the speed difference we noted in the Webpagetest numbers. (Click image to enlarge.)

Ecommerce revenue: Order value per visitor is 29% higher for Internet Explorer 8 users than Internet Explorer 7

Change connection speed = change order value

We then explored different connection speeds within IE8. First, we performed Webpagetests on the different connection speeds, and then we compared them to the results in Google Analytics.

Again we found a remarkable relationship between connection speed and order value. (Click image to enlarge.)

Online shoppers with T1 connections spent roughly 32% more than those using dialup

On average, online shoppers using T1 connections spent about 11% more than shoppers with DSL connections. And shoppers with T1 connections spent roughly 32% more than those using dialup.

Calculating the benefits and caveats

We then explored the performance benefits his site would gain on IE7 and IE8 with Site Optimizer, and then compared them to the current performance of his site. After correlating the performance gains with the graphs from his analytics tool,  my non-believer converted. He was convinced that, at minimum, the proposed performance enhancements would increase his per visitor value across the board to at least the same levels as IE8, if not higher.

Obviously, other mitigating factors are at play here. (I’ll say it before someone else does: Correlation does not imply causation.) The choice of browser and connection speed might imply something about the buyer that is totally unrelated to the effects of speed on shopping choices. To pass any hardcore statistical muster, a much more in-depth regression test would need to occur.

What amazed me, however, was that these two screens of his own data was enough to convince my non-believer, on the spot, that performance mattered and that he should invest in it.

This trend held across other sites I sampled

After observing these results, I had to see if this trend held. I explored the analytics data for three other Strangeloop clients and was very excited to see the results and the overall trend. These validate the fact that, across the board, page speed improvements seem to correlate to greater order value.

Value per visit by browser:

Value per visit by connection speed in IE8:

Value order per visit by connection speed using Internet Explorer 8

This trend held in the real world, too

I needed to apply this methodology to one last test to determine if this back-of-the-napkin calculation had any validity in the real world.

I took a Strangeloop customer who had been through a rigorous month-long 50/50 test. In this particular case, with all other variables accounted for, the optimized site outperformed the unoptimized site. On average, order value for the optimized site was 20% greater than for the unoptimized site:

Improved page speed correlates to increased order value per visit

Takeaway: How to perform a 5-minute analysis of your ecommerce site

Using two simple tools you probably already have at hand — Webpagetest and Google Analytics — you can quickly calculate how a faster user experience correlates to greater order value from your customers.

  1. Using Webpagetest, test your site’s speed on IE7 and IE8. Calculate the difference as a percentage.
  2. Test your site to see what performance gains you could gain from front-end performance enhancements.
  3. Look into Google Analytics (or whatever analytics package you use) to find out your current order value per visit for IE7 and IE8.
  4. Use your analytics to find out your order value per visit by connection speed.*
  5. Correlate the results in simple graphs, as I’ve done above.
  6. Look at the performance gains from front end performance enhancements and determine what lift you anticipate in value per visit.
  7. Share with your team.

If you’ve been having a tough time convincing your company to invest in performance, these results could provide your tipping point.

*If you’re using Google Analytics, I think the easiest way to get this data is with a custom report that looks something like this:

Using Google Analytics to calculate correlation between connection speed and order value

Related posts:

By: Tammy Everts Category: Case studies, Ecommerce, Tools

Is web performance optimization a “green” issue?

28 Jan 2011

5 Comments

I live in a city with a bold ambition: to be the greenest city in the world by 2020. I try to do my part. I take the bus or walk to work, my family has only have one car that we use rarely, and we even use the ugly energy-efficient Christmas lights.

When it comes to my work, I also believe that I have a positive impact, and it makes me feel good when green is highlighted in our industry.

Last May, Steve Souders came up with a list of predictions about the future of web performance optimization. Prediction #3 was this:

“Finally we’ll see studies conducted that quantify how improving web performance reduces power consumption and ultimately shrinks the web’s carbon footprint.”

When I first read this, my immediate reaction was, “Wow. That’s a really cool, bold expectation.” Ever since, I’ve kept my eyes open for anything I can find about the impact of performance on energy use. Maybe I’m hanging out in the wrong part of the internet, but I haven’t had much luck.

Analyzing the trade-off between performance optimization and energy use is a huge challenge. It’s not enough to say, “We’re delivering smaller pages and fewer/smaller objects, therefore using less energy. Problem solved!” We also have to take into consideration:

  • the energy consumed by new machines added to the network to automatically transform web pages,
  • the impact on servers as more is offloaded in the network,
  • the change in use of a content delivery network,
  • the change in energy consumption at the client level based on increased or decreased CPU use,
  • the change in energy consumption of the user as they browse more pages and buy more,
  • etc., etc.

So while it would be great to see a simple “If _____, then _____” equation for calculating performance/energy savings, perhaps it’s not a huge surprise that not a lot of people have tackled this big hairy question.

A (very) few people have tried to tackle this question. These are the best articles and blog posts I’ve come across:

Steve Souders: How green is your web page?

This blog post is almost three years old, but still worth reading. Huge kudos to Steve for this exercise in quantifying how specific performance improvements (he uses Wikipedia as an example) could lead to energy savings. When I imagine helpful equations for calculating performance/energy benefits, I imagine them looking a lot like this.

Boston.com: Taking a different measure

Really interesting article that came out last fall about how Akamai is auditing the carbon footprint of its 70,000-server network. It’s a hugely ambitious project, which the company undertook after realizing that 87% of its carbon footprint came from its network operations. This is the kind of data we need. It’s a key piece of the puzzle in figuring out how to quantify performance and energy use.

Fast Company: Is the Internet Sustainable When Everyone On Earth Uses Over 3 Gigabytes of Data Per Day?

Scary quote alert…

“That’ll come to 2,570 exabytes per year for the global population, by 2030. (An exabyte is a billion gigabytes.) The average power needed to sustain such activity would be 1,175 gigawatts. It takes an entire large coal-fired power plant to produce just one gigawatt of energy, so imagine 1,175 of those churning out power just to fuel the world’s data hunger.”

That piece came out right before Christmas, and the fact that it appeared in a relatively mainstream publication like Fast Company is an indicator of the fact that these questions are not going to go away.

According to this report, video is a major bandwidth hog. Streaming/downloaded content from Netflix, YouTube, BitTorrent, and iTunes accounts for 40% of peak U.S. web traffic. (It may be a sad statement that, when I learned that YouTube users are uploading 35 hours of video per minute, my reaction was, “That’s all?”) Video also dominates mobile in pretty much the same proportion.

And that’s just the activity in the United States. Internet users in China log a total of one billion hours online every day, twice as much as Americans. Adoption rates are expected to more than double in the next three years, and not just in China. India, Brazil, Russia, and Indonesia are also poised to see a huge growth in the amount of time their citizens spend online.

So that’s it, the sum total of information I’ve found.

I’m an optimist. I believe that most problems have solutions. My hunch tells me that Steve is correct in postulating “Make your pages faster. It’s good for your users, good for you, and good for Mother Earth.” However, I don’t think we have enough data to confirm or deny what seems obvious. We need more data, and I for one am trying to work with customers to put together case studies that demonstrate positive environmental impact, as much for myself, so I can sleep at night, as for our industry as a whole.

Related posts:

By: Tammy Everts Category: Research, Standards and benchmarks Tags: , ,

Is it time to give up Internet Explorer 7 as our default test scenario?

25 Jan 2011

5 Comments

Here at Strangeloop, there’s been a lot of debate lately about what our default browser should be when we run performance tests through Webpagetest. As discussed here previously, we had a very strong argument for using Internet Explorer 7. We’re not ready to abandon it yet, but the idea is under the microscope.

Equally important, we’ve also been debating if the default bandwidth — these days, we test using the DSL setting — is accurate.

Akamai came out with its state of the internet report yesterday, which has made me think about this issue some more.

From Akamai’s report:

“The overall average connection speed for the U.S. as a whole in the third quarter of 2010 was 5.0 Mbps. Delaware continued to maintain its standing as the state with the fastest average connection speed. The overall average peak connection speed in the U.S. during the third quarter was 20 Mbps.

“In looking at high broadband adoption in the U.S. during the third quarter, trending was mostly positive. Quarterly increases in high broadband adoption of 10% or more were seen in 23 states and the District of Columbia, with New Mexico topping the list at 60% growth. In reviewing year-over-year changes in U.S. broadband adoption, four states (Alaska, Minnesota, Montana, and Alabama) grew more than 100% year-over-year, with Alaska’s massive 191% growth leading the way.”

After reading this, I wondered about a few things:

  • First and foremost, I wondered how many people use the default settings. Maybe people are using their own custom settings, so the defaults are not actually a huge issue.
  • What would an experiment look like if we if loaded the Akamai site using different bandwidths and latencies?
  • What would a graph look like in Webpagetest for the same site if I worked my way up from 1.5Mbs (the standard WPT setting for DSL) up to 5.5 Mbs?
  • What would happen if I tested variance in latency as well — say, comparing 15ms to 50ms?
  • While I was at it, what about testing all these parameters in IE8?

How many people use Webpagetest’s default settings?

I needed to address this question first, to validate looking into the rest of my questions. To look at the use of default settings, I simply looked at the last 8,847 public tests (the sum of one day’s testing).

As these graphs show, most users are indeed using the default settings (IE7 and DSL):

Default browsers used for page speed tests

Default connections used for page speed tests

With this established, I was ready to move on to the rest of my questions. Here’s what I found, this time displayed in handy line graph form.

Four observations

First, a couple of caveats. We only ran three tests per site, then used the median results. And it goes without saying that this is only one website; all sites will perform differently, so your mileage may vary.

  1. More bandwidth does not have a significant impact on load time when we test using the current IE7 default, and has almost no impact on start render. Mike Belshe wrote about this last May, and this exercise seems to confirm his findings.
  2. When it comes to latency, we see the same pattern in IE7 that Mike did: when we decrease round trip time, we improve performance.
  3. But unlike Mike’s experiment, we see no pattern in IE8 that links latency and performance when it comes to load time; however, we do see a strong correlation in start render.
  4. Most important, as the side-by-side videos below demonstrate, when we look at perceived load time by watching the site through a real user’s eyes, we see very little difference in the user experience as we increase bandwidth or change latency in IE7 or IE8.

Test results with IE7, 15ms latency:

Test results with IE7, 50ms latency:

Test results with IE8, 15ms latency:

Test results with IE8, 50ms latency:

Conclusions

Webpagetest’s default testing scenario of IE7 with DSL (50ms latency and 1.5mbs) probably needs to be updated. We should probably move to IE8, 2Mbs and 30ms of latency (this is just my hunch) to ensure that our tests reflect the average experience of real visitors.

What do you think? Are there any patterns I’ve missed? What are your thoughts about the ideal testing scenario?

Final note

Circling back to the point that I mentioned at the top of this post: I’m not advocating for overnight change. As one of the companies that hosts Webpagetest, I know that a move to IE8 is not necessarily easy. I raise these questions as fuel for debate.

Edited to add:

Pulling this out of the comments: Patrick at Webpagetest wants to know what you think the new default browser should be. Chime in on the discussion boards.

Related posts:

By: Tammy Everts Category: Browsers, Research, Tools Tags: , ,

Web performance in the cloud: How do we retain control and visibility?

21 Jan 2011

8 Comments

My head has been in the cloud lately. For the past six months, we’ve been offering Site Optimizer as a cloud-based service, and we recently announced partnerships with Akamai and Level 3 that will extend our cloud relationships.

I’ve been thinking about the general rush to public clouds and wondering if, in the rush, are people giving up too much control? And if so, how do we get some of this control back?

A recent survey by CA Technologies suggests that we feel ambivalent about the cloud. They talked to 434 executives and found that, while interest in public and private cloud computing is widespread, IT managers and business leaders have concerns about issues of security and control.

No big surprises there. On any given day, you can see this excitement and ambivalence echoing through your Twitter feed.

When people deploy apps to cloud platforms, they take a bit of a leap of faith because they lose a bunch of visibility and control that they may not be used to living without. Their infrastructure becomes a kind of blurry black box, and it may take some time to get comfortable with that. It’s the vendor’s responsibility, then, to ease the transition and provide those tools to make site owners more comfortable with putting their stuff on the cloud.

Control as it relates to cloud performance

Last summer, Alistair Croll released an excellent study in which his company, Bitcurrent, tested the performance of five cloud providers. He highlighted some of the main areas of concern, and the tradeoff we make in choosing the cloud:

Almost all the concerns around public cloud computing have to do with a shared resource. You may be worried about neighboring cloud tenants with bad security practices. You might fear that other applications will consume all available resources. Or perhaps you think that the cloud provider has chosen an underlying architecture that helps them, but won’t make the most of your application.

All of these concerns are legitimate. The pact that you’re making with a public cloud, for better or worse, is that the advantages of elasticity and pay-as-you-go economics outweigh any problems you’ll face

Performance pain points can happen in many places, in and outside the cloud

Alistair also illustrates the many performance pain points that can crop up:

The Performance of Clouds: Pain Points

Sample scenario: Isolating your performance problems

Has this happened to you? You start getting complaints from users that your cloud-hosted site is slow. You know that there are dozens of potential performance pain points. You ask your IT manager or lead developers what the problem is. They suggest that it might be your cloud provider. So you ask your cloud provider. They say that it’s probably a dev problem. Where do you go from there?

What you need from your cloud vendor

Knowing that performance has a direct impact on revenue, site owners are putting their entire business in the hands of their cloud providers. It’s understandable that we’re tempted to demand control over the performance of our applications, because this is what we had (or at least the illusion of it) before.

This demand is, ultimately, unrealistic, but it is realistic to expect that, in exchange for losing some of the infrastructural control you used to have before you deployed your apps to the cloud, you need to ask your provider for more visibility and assurances in how their infrastructure will perform for you, such as:

  • Analytics tools – You should continue to use the performance monitoring tools you already use to monitor your app performance with (Webpagetest, Keynote, Gomez, etc.), but you also need a new visibility toolset (that your cloud provider should help you with) so you can do root cause analysis over parts of the infrastructure that you don’t have control over or visibility into.
  • Detailed performance clauses in your service level agreements – Some cloud SLAs contain a performance clause that addresses availability, but speed is every bit as important as — and possibly more important than — uptime. This clause needs to spell out how your concerns about site speed will be addressed, should they occur.

(I’m very interested to hear other people’s thoughts here. What could these tools and clauses look like? Is there any other way we can retain control and visibility? I’d love to hear people’s first-hand experiences.)

What your cloud provider can’t do for you

In exchange for asking for more from your cloud provider, we also need to know when to ask for less.  Site owners need to realize that hosting in the cloud will not, de facto, make their sites faster. I don’t believe I’ve ever heard of a cloud vendor that makes the claim that they guarantee faster load times, but I’ve encountered this myth via a few site owners. Good hosting does not replace front-end optimization.

I’m the first to admit that the cloud is not my area of expertise. Fortunately, it doesn’t have to be, because we have Hooman Beheshti, VP of Product here at Strangeloop, who is our guru of all things cloud-related. In March, Hooman’s on his way to Cloud Connect, where he’s chairing the Performance and Monitoring track. He and a lot of other very smart people are going to spend a week discussing the cloud from many angles, including the current state of cloud SLAs. If you want to get a non-sensationalistic, non-hyped take on the cloud, I strongly recommend you check it out.

Related posts:

By: Tammy Everts Category: Cloud, Tools

Front-end web performance optimization: It isn’t over till it’s over. And it’s never over.

18 Jan 2011

11 Comments

A few days ago, I was talking to a prospective client who is interested in transformation-based performance solutions like Strangeloop’s right now, but who questioned the lifespan of our industry.

“In five years, your industry won’t even exist,” he said. His rationale was that in the future everyone will have applied Steve Souders’s performance rules to their sites, and there won’t be a role for automated performance tools.

To illustrate, he sent me this page, which had been optimized by a team of in-house developers:

Waterfall chart showing website before front-end optimization

Perfect Page Speed score, short waterfall, fast load time — this is pretty much a picture-perfect chart. (If you’re not familiar with waterfalls, here’s a primer on how to interpret them.)

But we ran it through Site Optimizer anyway, to see what we could do. Here’s what we did:

Waterfall chart showing website after front-end optimization

We cut load time by more than half, which is great. But look at our Page Speed score: only 74 out of 100. And we broke ‘Compress Images’.

What does this mean?

There are three takeaways from this exercise:

  1. The current public-facing performance rules are not complete. We sped up this site by using techniques that are not recognized in the known lists of performance rules. For example, when we consolidate images, in one of the browsers we turn them into a format that Webpagetest doesn’t recognize, so it gives us an F for Compress Images. But we make the page faster. Which matters more: the failing score or the faster page?
  2. Front-end website optimization will never be static. When done well, it evolves with changes to browsers and to user behavior, so it should always keep getting better.
  3. The front-end optimization market is evolving faster than the performance measurement tools. I say this not to criticize these tools. I’ve repeatedly gone on the record as a big Webpagetest and HTTPwatch supporter, and I think these tools are essential to our industry. But they are developed as a response to the performance rules. When the rules change or new rules are added, there’s inevitable lag time as the tools evolve to catch up.

The conclusion is one that I keep coming back to: When it comes to site speed, there really is no such thing as fast enough.

Related posts:

By: Tammy Everts Category: Case studies, Tools Tags: , , ,
« 30 31 32 33 34 35 36 37 »