Case study: How effective are CDNs for mobile visitors?

Substitute “measuring mobile performance” for “herding cats” in this video, and you’ve pretty much nailed the challenge we’re up against every day.

Fortunately, we like cats. :)

Experiment: Measuring the impact of CDN deployment on 3G performance

As we continue to evolve our mobile treatments, we also monitor their effectiveness alongside other optimization solutions. Today I want to call out some interesting results we noted when, as a fun little in-house exercise, we took the O’Reilly website, de-optimized it, and then iterated through a handful of core performance best practices using our FEO service. The goal was to demonstrate the acceleration benefit (in terms of bytes in, start render time, document complete time, connections, and resources) of each practice for a typical 3G mobile user.

While we saw predictable results for step 1 — enabling keep-alives and compression — we were somewhat surprised by what we saw when we added a content delivery network.

Step 1: Apply fundamental best practices

First we added keep-alive connections:

  • What it does: Lessens the impact of TCP connection setup
  • Benefit: Addresses the problem of having too many TCP connections

Then we added HTTP compression:

  • What it does: Compresses text-based content (HTML, CSS, JavaScript, etc.)
  • Benefit: Easy way to reduce bytes/payload

We got the results we expected: faster start render, and about a 25% reduction in doc complete time. This is fantastic, even more so because both these treatments are really easy to do — usually it’s just a matter of a single configuration option on your server, proxy, or load balancer. However, these two treatments aren’t enough to give you the acceleration you need.

Step 2: Use a content delivery network (CDN)

Here’s the elevator-ride explanation of how a CDN works (for an excellent detailed explainer, go here): Static page assets (images, CSS, etc.) are served from locations near the requesting client (mobile or otherwise). The shorter distance between client and content means smaller time to first byte (TTFB) and, ostensibly, faster start render. This means users start to see content in their browser faster.

How a content delivery network works

In our experiment, we expected that adding a CDN would result in faster average time to first byte, start render, and doc complete time. Here’s what we saw:

Before and after: Document complete (aka load time)

Before and after content delivery network deployment

Before and after: Time to first byte (TTFB)

Before-and-after CDN: Time to first byte (TTFB)What we helped

CDN before and after: What we helpedFindings

With a CDN, the page got faster, but not by much. For the unoptimized page, we forced requests to travel from west coast to east coast. After, we let the CDN naturally select the closest edge. As a result, we saw that:

  • Doc complete time decreased by 10%, compared to a 20% improvement we noted in a similar experiment in desktop optimization.
  • We shaved less than a second off start render time, taking it from 7.059 seconds down to 6.245 seconds.
  • True, we cut time to first byte by 39%, but from an end-user perspective, TTFB doesn’t really mean anything because the user still isn’t seeing anything in the browser.

Five questions about CDNs and mobile acceleration

I’m not making any nutty claims like “CDNs aren’t effective for mobile devices.” (Heresy!) But these results do raise a few questions:

  1. Is edge selection for mobile devices not as effective as for desktop?
  2. Some, if not all, CDNs probably deploy servers near cell network exit points. But what if most of the latency occurs inside the cell network? (I’ll admit I’m not an expert on what happens inside a 3G network. I’m ready to be enlightened.)
  3. Does the meaning of “closeness” change for mobile?
  4. Acknowledging the existence of mobile-specific CDNs, how much more effective are they? How do they compare when it comes to 3G versus WiFi? I’ve been trying to dig up case studies, with no luck.
  5. High CDN costs may be justifiable when you see significant benefit for your desktop traffic, but do they deliver sufficient benefit/ROI for mobile users?

Takeaway

LTE is grabbing a lot of attention, but it’s a mistake to sweep 3G under the rug. There are 256 million 3G subscribers in the US, representing 81% penetration, so 3G performance is still a big deal. We need more research. If you have findings to share, please do.

Related posts:

The psychology of waiting (and 5 things you can do to make online checkout feel faster)

Americans spend an estimated 37 billion hours per year waiting in lines. That’s 118 hours per person (including babies, though I have no idea what they’re standing in lines for), which is pretty mindblowing. So it’s not surprising that there’s a large – and growing – field of research dedicated to studying the psychology of waiting. A recent trip down this research rabbit hole yielded some interesting insights about in-store versus online waiting.

In-store waiting vs. online waiting

As I wrote in a recent post on the Radware blog, 70% of online shopping carts are abandoned before checkout. A recent survey of US shoppers found that slow load times was the number one cause for half of those abandoned carts. Unfortunately, there aren’t comparable numbers for in-store shopping cart abandonment (the only measurement tool a bricks-and-mortar store has is a security camera that counts foot traffic and compares it with number of purchases), but it’s safe to hazard a guess that it’s not as high as 70%.

When it comes to standing in checkout lines, there are a few other points of dissimilarity between in-store shoppers and online shoppers:

In-store Online
A variety of potential service systems (first-come-first-served, single server vs. multiple servers, reservation-based, express line options, etc.) Perception of instantaneous service
Can see lineup(s) and estimate wait time (however erroneously) Cannot estimate transaction time in advance
Can exercise several choices when faced with perceived slow lineups: refuse to enter, enter but leave before checkout, or jockey among different lineups Only one option when faced with long wait times: abandon cart
Can be influenced by friendliness of checkout staff, which mitigates negative impact of standing in line Cannot be influenced by a friendly “Thanks for your order” confirmation page

 

However, both types of shopper do have one thing in common:

In-store Online
Associates long wait times with poor customer service, which negatively affects likelihood of returning Associates long wait times with poor customer service, which negatively affects likelihood of returning

In short, the online checkout process is characterized by uncertainty.

This is caused by relative lack of feedback about your transaction status, coupled with a lack of choice in terms of how you can respond to long wait times. In a physical store, you know the line is going to move eventually, and that if you get desperate you can hop on another line. If a page hangs during an online transaction, it introduces uncertainty that you’ll ever be able to complete your purchase. (In one survey, 44% of respondents said that page slowdowns during checkout made them anxious about the success of the transaction.) And line jockeying isn’t an option on the internet.

Shopping cart anxiety

Common-sense things we know about waiting…

  • As waiting time increases, satisfaction decreases.
  • As perceived or recalled wait duration increases, the wait becomes less acceptable.

Obvious-sounding stuff, right? These findings more or less make sense because they appeal to what we believe to be our common sense.

However, common sense is pretty thin on the ground…

…as you see when you look at the larger body of wait time research. We’re riddled with irrational feelings about waiting:

Infographic: How internet users perceive time

And best of all:

  • Even if we go into a transaction knowing our tendency to be prey to the illusions described above, most of us will still fall prey to them.

Takeaways

If you’re visiting this blog, you obviously care about delivering a better experience to whoever your users are. You’re probably already working to make your pages faster – through applying optimization best practices, deploying a CDN, etc. That’s a crucial beginning. But there’s more.

1. See what your users see: Mentally increase your measurement numbers by 35%.

Understand that the start render time (or load time, or whatever performance metric you focus on) numbers that you see in your performance measurement data may be the real picture, but your real picture doesn’t match your users’ perceived picture. If your pages load in 5 seconds, the average user remembers it as feeling like closer to 7 seconds.

2. Ensure that every page in the transaction is fast.

A lot of site owners focus on optimizing their landing pages and product pages, but as this case study shows, slowing down pages later in a transaction can cause the abandonment rate to jump from 67% to 80%. Every page matters.

3. Better yet, simplify the transaction process down to a single page.

Implementing one-click checkout, like Amazon, is one way to to this. Another is to build your checkout as a single-page application using Ajax, so that resource requests and responses happen in the background, beyond the user’s notice.

4. Know when to use spinners and progress bars.

And know how to design them. (Also know when not to use them. A progress bar on a page that loads in less than 5 seconds will actually make that page feel slower.) There are some solid best practices here.

5. Make the perceived value match (or better, surpass) the wait.

If long wait times are necessary, ensure that you’re delivering something that has value that’s commensurate with the wait. A good example of this is travel websites. When you’re searching for the best hotel rates, most of us don’t mind waiting several seconds. We rationalize the wait because we assume that the engine is searching a massive repository of awesome travel deals in order to give us the very best results.

I’m still deep in this rabbit hole. If you have any more good research to throw down, I’d love to check it out.

Related posts:

Aaron Kulick (Walmart Labs): “RUM is trivial to implement but complicated to make actionable.” [PODCAST]

Hi all. Many thanks to everyone who sent me friendly greetings on Twitter and LinkedIn yesterday. Hopefully I’ll get to meet you in person some day.

Josh left some big shoes to fill. He also left some excellent unaired podcasts, which I’ll be rolling out over the next few weeks, starting with this great interview with Aaron Kulick. Like most of us, Aaron wears a lot of hats. If you live and work in Silicon Valley, then you probably know him as the founder and co-organizer of the SF Web Performance Meetup Group — not surprisingly, the biggest performance meetup in the world. Or you may know him as the organizer of the first-ever WebPerfDays that followed Velocity Santa Clara last year. Or you might know him in his former role as senior software engineer at Walmart Labs — and particularly as one of the guys behind this awesome slide deck, which went viral last year.

Like a few of our previous podcast guests, Aaron has been in the enviable position of working with massive amounts of RUM data. There are rumblings of some folks saying “RUM is so 2012″, but in many regards it’s still in its infancy — or at least its toddlerhood. The tools are there, the data is there, but turning this data into actionable insights is the next great challenge: it takes a lot of multidisciplinary know-how to make RUM data truly relevant at an operational level. As Aaron said, “It’s a bit of an uphill charge, but many of the teams [at Walmart] have started to incorporate analysis around [performance data] or at least keep it as a, ‘Hmm, we should keep this in mind.’” Baby steps, right? :)

I hope you enjoy the podcast. If you have feedback or suggestions for future podcast subjects, email me at tammye [[at]] radware [[dot]] com.

Listen to the podcast: Aaron Kulick

Related posts:

Thank you.

I’ll come right to it: As of this week, I’m retiring.

Between the birth of my third child and preparing the move to Radware, the past sixteen months have been exhilarating. And, like most exhilarating experiences, it’s been exhausting. I’m tapped out. It’s time to unplug and recharge. I know — those metaphors contradict each other. That’s what happens when you’re tired. :) I’ll be taking an extended break. For how long, I can’t say — possibly till my wife gets sick of having me around the house. Maybe a bit longer.

Radware has a sustainable vision for our technology and the chops to execute that vision. I’m looking forward to watching and cheering from the sidelines.

Kent Alstad – one of Strangeloop’s cofounders, our former CTO, author of all our patents, and the most knowledgeable person I’ve ever met when it comes to hardcore FEO – will continue to drive innovation on the product side of things at Radware. Kent is also an awesome speaker, so you can look forward to seeing more of him on the conference circuit and on this blog.

On the performance evangelism side, Tammy Everts will be taking over here on the blog, as well as with speaking engagements and everything that goes with that. Under her leadership, you’ll continue to enjoy the high quality of research, insights, and industry news that I hope you’ve come to associate with this site. Some of you already know Tammy, but if not, you should know that she’s been working closely with me behind the scenes for the past three years. This blog has been very much a collaborative effort, and I want to publicly thank Tammy for both her ideas and her much-needed editing skills. Before coming to Strangeloop, she had a long background in user experience. She brings a really cool perspective on the human side of performance, and I know she’s already planning some exciting new research in that area, as well as continuing to cover the business and technical aspects. I’ve also pestered her into reviving her semi-dormant Twitter account, so I encourage you to follow her there and force her to post. :)

Radware has also taken the forward-looking step of training a global team of technical experts who can preach our technology in every major market. With dozens of performance-related Meetup groups popping up around the world, I’m very excited about the prospect of connecting with as many new people as possible — something Strangeloop, as a smaller company, was always limited in doing.

People keep asking me what’s next for me, and the answer isn’t that glamorous. I’m sure I’ll return to the tech world some day. Startup fever is like malaria: it keeps coming back. But for now, my plans include enjoying more time with my beautiful wife and our three boys, taking long walks with my dog, and spending some quality time connecting with my garden. (I have some ambitious pond-building plans.)

If I tried to thank everyone who’s supported me over the past several years, I’d be here all day, so let me just say thank you to everyone, including you.

Lori MacVittie (F5): “Fast is an incredibly, incredibly stressful thing to try and do.” [PODCAST]

I am so incredibly tempted to call Lori MacVittie the First Lady of Web Performance, if only because I know how hard it would make her laugh. But seriously, in the intro to my latest podcast, I said that Lori’s forgotten more about the history of our industry than most of us ever knew, and I meant it.

Performance has become a hot topic in the past three or four years, but it’s been around since the inception of the web, and it’s meant different things at different times. Up till six or seven years ago, the focus was on just keeping sites up under load stress, no matter whether or not this meant losing a few seconds. Fast forward to today, and things have done a complete turnabout. As most of us know, optimal performance is not only about supporting millions of users, but also ensuring that each of these users is blazing fast. As Lori points out, “it’s an incredibly, incredibly stressful thing to try and do, because you’ve got even more moving parts now than you did then.”

Lori and I covered a lot of ground in our chat, from the early acceleration pioneers who helped birth modern-day FEO, to ADCs and SDNs and a bunch of other cool acronyms. Lori also made a heroic attempt to convince me that the cloud is interesting. You’ll have to listen to the podcast to see if she succeeded. :)

(Note: I want to apologize in advance for the audio quality in this podcast. We’re working on making it better.)

Listen to the podcast: Lori MacVittie

Related posts: