best practices

Case study: How effective are CDNs for mobile visitors?

Substitute “measuring mobile performance” for “herding cats” in this video, and you’ve pretty much nailed the challenge we’re up against every day.

Fortunately, we like cats. :)

Experiment: Measuring the impact of CDN deployment on 3G performance

As we continue to evolve our mobile treatments, we also monitor their effectiveness alongside other optimization solutions. Today I want to call out some interesting results we noted when, as a fun little in-house exercise, we took the O’Reilly website, de-optimized it, and then iterated through a handful of core performance best practices using our FEO service. The goal was to demonstrate the acceleration benefit (in terms of bytes in, start render time, document complete time, connections, and resources) of each practice for a typical 3G mobile user.

While we saw predictable results for step 1 — enabling keep-alives and compression — we were somewhat surprised by what we saw when we added a content delivery network.

Step 1: Apply fundamental best practices

First we added keep-alive connections:

  • What it does: Lessens the impact of TCP connection setup
  • Benefit: Addresses the problem of having too many TCP connections

Then we added HTTP compression:

  • What it does: Compresses text-based content (HTML, CSS, JavaScript, etc.)
  • Benefit: Easy way to reduce bytes/payload

We got the results we expected: faster start render, and about a 25% reduction in doc complete time. This is fantastic, even more so because both these treatments are really easy to do — usually it’s just a matter of a single configuration option on your server, proxy, or load balancer. However, these two treatments aren’t enough to give you the acceleration you need.

Step 2: Use a content delivery network (CDN)

Here’s the elevator-ride explanation of how a CDN works (for an excellent detailed explainer, go here): Static page assets (images, CSS, etc.) are served from locations near the requesting client (mobile or otherwise). The shorter distance between client and content means smaller time to first byte (TTFB) and, ostensibly, faster start render. This means users start to see content in their browser faster.

How a content delivery network works

In our experiment, we expected that adding a CDN would result in faster average time to first byte, start render, and doc complete time. Here’s what we saw:

Before and after: Document complete (aka load time)

Before and after content delivery network deployment

Before and after: Time to first byte (TTFB)

Before-and-after CDN: Time to first byte (TTFB)What we helped

CDN before and after: What we helpedFindings

With a CDN, the page got faster, but not by much. For the unoptimized page, we forced requests to travel from west coast to east coast. After, we let the CDN naturally select the closest edge. As a result, we saw that:

  • Doc complete time decreased by 10%, compared to a 20% improvement we noted in a similar experiment in desktop optimization.
  • We shaved less than a second off start render time, taking it from 7.059 seconds down to 6.245 seconds.
  • True, we cut time to first byte by 39%, but from an end-user perspective, TTFB doesn’t really mean anything because the user still isn’t seeing anything in the browser.

Five questions about CDNs and mobile acceleration

I’m not making any nutty claims like “CDNs aren’t effective for mobile devices.” (Heresy!) But these results do raise a few questions:

  1. Is edge selection for mobile devices not as effective as for desktop?
  2. Some, if not all, CDNs probably deploy servers near cell network exit points. But what if most of the latency occurs inside the cell network? (I’ll admit I’m not an expert on what happens inside a 3G network. I’m ready to be enlightened.)
  3. Does the meaning of “closeness” change for mobile?
  4. Acknowledging the existence of mobile-specific CDNs, how much more effective are they? How do they compare when it comes to 3G versus WiFi? I’ve been trying to dig up case studies, with no luck.
  5. High CDN costs may be justifiable when you see significant benefit for your desktop traffic, but do they deliver sufficient benefit/ROI for mobile users?

Takeaway

LTE is grabbing a lot of attention, but it’s a mistake to sweep 3G under the rug. There are 256 million 3G subscribers in the US, representing 81% penetration, so 3G performance is still a big deal. We need more research. If you have findings to share, please do.

Related posts:

The psychology of waiting (and 5 things you can do to make online checkout feel faster)

Americans spend an estimated 37 billion hours per year waiting in lines. That’s 118 hours per person (including babies, though I have no idea what they’re standing in lines for), which is pretty mindblowing. So it’s not surprising that there’s a large – and growing – field of research dedicated to studying the psychology of waiting. A recent trip down this research rabbit hole yielded some interesting insights about in-store versus online waiting.

In-store waiting vs. online waiting

As I wrote in a recent post on the Radware blog, 70% of online shopping carts are abandoned before checkout. A recent survey of US shoppers found that slow load times was the number one cause for half of those abandoned carts. Unfortunately, there aren’t comparable numbers for in-store shopping cart abandonment (the only measurement tool a bricks-and-mortar store has is a security camera that counts foot traffic and compares it with number of purchases), but it’s safe to hazard a guess that it’s not as high as 70%.

When it comes to standing in checkout lines, there are a few other points of dissimilarity between in-store shoppers and online shoppers:

In-store Online
A variety of potential service systems (first-come-first-served, single server vs. multiple servers, reservation-based, express line options, etc.) Perception of instantaneous service
Can see lineup(s) and estimate wait time (however erroneously) Cannot estimate transaction time in advance
Can exercise several choices when faced with perceived slow lineups: refuse to enter, enter but leave before checkout, or jockey among different lineups Only one option when faced with long wait times: abandon cart
Can be influenced by friendliness of checkout staff, which mitigates negative impact of standing in line Cannot be influenced by a friendly “Thanks for your order” confirmation page

 

However, both types of shopper do have one thing in common:

In-store Online
Associates long wait times with poor customer service, which negatively affects likelihood of returning Associates long wait times with poor customer service, which negatively affects likelihood of returning

In short, the online checkout process is characterized by uncertainty.

This is caused by relative lack of feedback about your transaction status, coupled with a lack of choice in terms of how you can respond to long wait times. In a physical store, you know the line is going to move eventually, and that if you get desperate you can hop on another line. If a page hangs during an online transaction, it introduces uncertainty that you’ll ever be able to complete your purchase. (In one survey, 44% of respondents said that page slowdowns during checkout made them anxious about the success of the transaction.) And line jockeying isn’t an option on the internet.

Shopping cart anxiety

Common-sense things we know about waiting…

  • As waiting time increases, satisfaction decreases.
  • As perceived or recalled wait duration increases, the wait becomes less acceptable.

Obvious-sounding stuff, right? These findings more or less make sense because they appeal to what we believe to be our common sense.

However, common sense is pretty thin on the ground…

…as you see when you look at the larger body of wait time research. We’re riddled with irrational feelings about waiting:

Infographic: How internet users perceive time

And best of all:

  • Even if we go into a transaction knowing our tendency to be prey to the illusions described above, most of us will still fall prey to them.

Takeaways

If you’re visiting this blog, you obviously care about delivering a better experience to whoever your users are. You’re probably already working to make your pages faster – through applying optimization best practices, deploying a CDN, etc. That’s a crucial beginning. But there’s more.

1. See what your users see: Mentally increase your measurement numbers by 35%.

Understand that the start render time (or load time, or whatever performance metric you focus on) numbers that you see in your performance measurement data may be the real picture, but your real picture doesn’t match your users’ perceived picture. If your pages load in 5 seconds, the average user remembers it as feeling like closer to 7 seconds.

2. Ensure that every page in the transaction is fast.

A lot of site owners focus on optimizing their landing pages and product pages, but as this case study shows, slowing down pages later in a transaction can cause the abandonment rate to jump from 67% to 80%. Every page matters.

3. Better yet, simplify the transaction process down to a single page.

Implementing one-click checkout, like Amazon, is one way to to this. Another is to build your checkout as a single-page application using Ajax, so that resource requests and responses happen in the background, beyond the user’s notice.

4. Know when to use spinners and progress bars.

And know how to design them. (Also know when not to use them. A progress bar on a page that loads in less than 5 seconds will actually make that page feel slower.) There are some solid best practices here.

5. Make the perceived value match (or better, surpass) the wait.

If long wait times are necessary, ensure that you’re delivering something that has value that’s commensurate with the wait. A good example of this is travel websites. When you’re searching for the best hotel rates, most of us don’t mind waiting several seconds. We rationalize the wait because we assume that the engine is searching a massive repository of awesome travel deals in order to give us the very best results.

I’m still deep in this rabbit hole. If you have any more good research to throw down, I’d love to check it out.

Related posts:

More new findings: Top ecommerce sites are 22% slower than they were last year

Let me say that again, because this is a staggering fact: The world’s top ecommerce sites are 22% slower than they were last year.

In December 2011, the median load time for a site in the Alexa Retail 2000 was 5.94 seconds. Just twelve months later, the median was 7.25 seconds. At this rate of growth, this number could hit almost 9 seconds by the end of this year.

Web page load time changes: December 2011 to December 2012

This was the key finding of our brand-new quarterly report (yes, a new report, not to be confused with last week’s report about web performance in the EU) on ecommerce web performance. If you’re new to these reports, since 2010 we’ve been measuring the load time, page composition, and best practice implementation of the same set of 2,000 leading online retailers, as ranked by Alexa. The goal is to learn how pages are changing over time and what impact, if any, these changes have on per-page performance. The results have been eye-opening.

I have to confess that I frequently feel like that tiresome guy at the party who keeps saying the same things over and over again. Pages are getting slower… pages are getting bigger… the gap between load times and user expectations is getting wider almost by the week. If you’re reading this out there and saying to yourself, “I can’t believe that Bixby guy is going on about this AGAIN,” then forward the next part of this post to five people you think could really benefit from it. If enough people internalize this message, maybe I’ll shut up.*

Three performance myths I would give anything to permanently bust:

Myth #1. Pages are, de facto, getting faster.

What with our better systems, networks, and browsers, pages must be getting faster, right? Everyone believes this instinctively, because most of us seem to be hardwired to believe that technology solves problems rather than creating new ones. But as I said at the top of this post, the quickly emerging fact is that pages seem to not just be getting slower, they’re getting slower at an alarming rate. (Optional: You may choose to take this finding as proof that we shouldn’t always trust our instincts. :) )

Myth #2. Users are more or less satisfied with the status quo.

“People are used to pages that take 5-8 seconds to load. They don’t mind that much.” I still hear this on a regular basis. Site owners rationalize that, because they’re not hearing a lot of complaints, their visitors are happy… or at least happy enough. But as numerous case studies have shown, people talk with their wallets. Faster sites earn more. And user surveys over the years are telling us that people’s expectations for a speedy online experience are continually growing:

Web Page Load Time: User expectations 200-2012

Myth #3. Browser development is more than capable of mitigating the factors, such as page size and complexity, that are causing pages to slow down.

This belief is widely held, even among technical folks. Again, looking to our findings, we saw that for all three browsers, median load times slowed down by anywhere from 3% to 12% in just six months. This downward trend isn’t a browser development issue. Instead, it’s an indicator that despite browser vendors’ huge commitment to speed, development can’t keep pace with the demands of bigger and increasingly complex web pages.
Browser performance: 2011 to 2012

Takeaway: Pass it on.

I am extremely happy that Radware is committed to continuing the tradition of releasing these quarterly  “state of the union” reports. As time passes, we’re gaining some invaluable insights. Based on these latest results, I’m very curious to see what our Summer 2013 report will hold.

I urge you to download this report (and the infographics, too). And I was only slightly kidding when I suggested that you forward this post to people who need to have a few performance myths dispelled. Our community does a lot of preaching to the choir. What seems basic to us is not necessarily basic to the rest of the world. We need to get out there and make sure these simple messages are being heard.

*Maybe.

Related posts:

New findings: Typical leading European commerce site takes 7.04 seconds to load

Last fall, at Velocity London, I had a really great talk with Stephen Thair, who is a UK-based web performance consultant, Velocity committee member, WebPerfDays organizer, and all-around knowledgeable guy. Among other things, we talked about how frustrating it can be for performance pros based in Europe to preach outside their community.

As Stephen said:

“I guess it’s a bit frustrating in the UK at the moment. One of the things that I found is that we haven’t yet got that killer web performance case study in one of the big major retailers. So we are still, I think, a bit in the evangelical stage. We are still trying to get the message out there. There are still a lot of websites in the UK that don’t even have gzip turned on.”

So we set out to help fill that gap. In December of 2012, working with Radware (our soon-to-be parent company) in conjunction with our partners at Level 3 in Europe, we studied the page speed and composition of 400 top European retailers, as ranked by Internet Retailer magazine, to see how these sites would load for visitors using Chrome 23 (the most popular browser in the EU at the time of testing) via the test server in Amsterdam. (We chose the Amsterdam location because it allowed us to test across all major browsers.) The report was released today.

While the results may not be shocking if you’ve been paying close attention to this space, they may come as an eye-opener to online retailers in the EU. Our chief finding was this:

The median page took more than 7 seconds to load.

Depending on whom you ask, the average internet user expects web pages to load in less than 3 seconds, 2 seconds, or even 400 milliseconds. The last time the average person reported being cool with 7-second load times was around 2001.

The survey also found that:

  • 1 out of 4 pages took more than 10 seconds to load.
  • 1 out of 3 pages contained more than 100 resources.
  • 79% of sites don’t use a recognized CDN. (A “recognized CDN” refers to any CDN listed in the extensive directory of CDNs maintained by WebPagetest.)
  • Speaking to Stephen’s point about gzip at the top of this post, 1 out of 5 sites failed to implement text compression, a relatively simple technique that delivers easy, significant performance gains.

Why you should care about these findings

I may be pointing out the obvious, but it may need to be pointed out: these expectations are universal. Internet users in the EU do not have lower performance expectations than users in North America. These findings should be a wake-up call for European site owners. (Not that North American site owners should be resting easy. Last fall, we found that the median leading US commerce home page took almost 7 seconds to load.)

Download the report: State of the Union: European Ecommerce Page Speed and Web Performance

Related posts:

Joshua Marantz (Google): Why erring on the conservative side is a good thing [PODCAST]

One of the greatest perks of my job is that I get to talk to geniuses on a daily basis. I was extremely privileged to be able to chat recently with yet another genius: Joshua Marantz, leader of mod_pagespeed at Google.

We covered a lot of ground, from philosophical to pragmatic topics. You’re going to enjoy this podcast if you’ve ever wondered:

  • What the heck is the difference between PageSpeed and mod_pagespeed?
  • Where does front-end optimization belong in order to deliver the best possible results: near the edge, near the server, or some point in between?
  • Will FEO someday be completely taken care of at the browser and server level?

We also talked quite a bit about images, which has been a recurring theme here lately. While Josh and I agree that what’s unambiguously good for the speed of a page is aggressive lossless image compression, he explains why that feature is not turned on by default in mod_pagespeed (though he says that he always tells people that it’s the first feature they should try to turn on).

Enjoy. And as always, if you have any questions, feedback, or suggestions for future guests or topics, drop me a line at joshua[[at]]webperformancetoday[[dot]]com.

Listen to the podcast: Joshua Marantz

Related posts: