Case study: How effective are CDNs for mobile visitors?

Substitute “measuring mobile performance” for “herding cats” in this video, and you’ve pretty much nailed the challenge we’re up against every day.

Fortunately, we like cats. :)

Experiment: Measuring the impact of CDN deployment on 3G performance

As we continue to evolve our mobile treatments, we also monitor their effectiveness alongside other optimization solutions. Today I want to call out some interesting results we noted when, as a fun little in-house exercise, we took the O’Reilly website, de-optimized it, and then iterated through a handful of core performance best practices using our FEO service. The goal was to demonstrate the acceleration benefit (in terms of bytes in, start render time, document complete time, connections, and resources) of each practice for a typical 3G mobile user.

While we saw predictable results for step 1 — enabling keep-alives and compression — we were somewhat surprised by what we saw when we added a content delivery network.

Step 1: Apply fundamental best practices

First we added keep-alive connections:

  • What it does: Lessens the impact of TCP connection setup
  • Benefit: Addresses the problem of having too many TCP connections

Then we added HTTP compression:

  • What it does: Compresses text-based content (HTML, CSS, JavaScript, etc.)
  • Benefit: Easy way to reduce bytes/payload

We got the results we expected: faster start render, and about a 25% reduction in doc complete time. This is fantastic, even more so because both these treatments are really easy to do — usually it’s just a matter of a single configuration option on your server, proxy, or load balancer. However, these two treatments aren’t enough to give you the acceleration you need.

Step 2: Use a content delivery network (CDN)

Here’s the elevator-ride explanation of how a CDN works (for an excellent detailed explainer, go here): Static page assets (images, CSS, etc.) are served from locations near the requesting client (mobile or otherwise). The shorter distance between client and content means smaller time to first byte (TTFB) and, ostensibly, faster start render. This means users start to see content in their browser faster.

How a content delivery network works

In our experiment, we expected that adding a CDN would result in faster average time to first byte, start render, and doc complete time. Here’s what we saw:

Before and after: Document complete (aka load time)

Before and after content delivery network deployment

Before and after: Time to first byte (TTFB)

Before-and-after CDN: Time to first byte (TTFB)What we helped

CDN before and after: What we helpedFindings

With a CDN, the page got faster, but not by much. For the unoptimized page, we forced requests to travel from west coast to east coast. After, we let the CDN naturally select the closest edge. As a result, we saw that:

  • Doc complete time decreased by 10%, compared to a 20% improvement we noted in a similar experiment in desktop optimization.
  • We shaved less than a second off start render time, taking it from 7.059 seconds down to 6.245 seconds.
  • True, we cut time to first byte by 39%, but from an end-user perspective, TTFB doesn’t really mean anything because the user still isn’t seeing anything in the browser.

Five questions about CDNs and mobile acceleration

I’m not making any nutty claims like “CDNs aren’t effective for mobile devices.” (Heresy!) But these results do raise a few questions:

  1. Is edge selection for mobile devices not as effective as for desktop?
  2. Some, if not all, CDNs probably deploy servers near cell network exit points. But what if most of the latency occurs inside the cell network? (I’ll admit I’m not an expert on what happens inside a 3G network. I’m ready to be enlightened.)
  3. Does the meaning of “closeness” change for mobile?
  4. Acknowledging the existence of mobile-specific CDNs, how much more effective are they? How do they compare when it comes to 3G versus WiFi? I’ve been trying to dig up case studies, with no luck.
  5. High CDN costs may be justifiable when you see significant benefit for your desktop traffic, but do they deliver sufficient benefit/ROI for mobile users?

Takeaway

LTE is grabbing a lot of attention, but it’s a mistake to sweep 3G under the rug. There are 256 million 3G subscribers in the US, representing 81% penetration, so 3G performance is still a big deal. We need more research. If you have findings to share, please do.

Related posts:

10 thoughts on “Case study: How effective are CDNs for mobile visitors?

  1. It shouldn’t be so hard to peek into the 3g network and see how much latency it adds to the retrieval of the http request – there is a traceroute app for windows phone, and I think android and probably iphone that will give you all the hops and which edge server the request finally ends up at – it would be interesting to see the results, whether different cell towers point to the right edge servers.

  2. Pingback: Pingdom's must-read articles #47

  3. Pingback: Pingdom’s must-read articles #47 | AsterHost

  4. Pingback: Case study: How effective are CDNs for mobile visitors?

  5. Pingback: Rounded Corners 427 — Ode to a shipping label | Labnotes

  6. Pingback: The WordPress Weekend Roundup - WP Daily

  7. I think you will see a lot of improvement if you reduce the number of requests you make.

    With keep-alive and no pipelining each TCP socket is limited to one http request at a time. Suppose you have the following: 95 requests, 1 socket and a RTT of 50ms. This will cost you atleast 95*50 = 4750ms of overhead. This is excluding the time needed to transfer the files.
    Luckily browsers use multiple sockets, not just 1, so the this figure will be much lower.

    You could try experimenting with data: URI’s. See what happends if you replace all your images with a data uri (which is not always a good idea if you think about caching, but it could be a good experiment.). Even though a data uri is 33% bigger (but only like 3% after compression) I saw a lot of improvement on my own pages by using it on images that are part of the design (logo, buttons, arrows, etc)

    Also, serving content from a lot of different servers will hurt page load times if you overdo it.

  8. Pingback: Increase your website’s speed?! « Life…, and technology

  9. Pingback: WORDPRESS ROUNDUP | Website Design, Websites For Sale and SEO

  10. The CDNs are not near cell network exit points (whatever they may be). A 3G network has a base station (also called NodeB), RNC (Radio Network Controller), SGSN and GGSN which has an open IP interface to the Internet. Below the GGSN (also called mobile core router), all the protocols are wireless-specific and CDNs cannot get in there since these elements are completely controlled by the mobile operator. At most, you can place an edge node at the Gi interface (IP interface above the GGSN) but as you can see, this aggregation point is just too far away from the customer. For comparison purposes a cable router typically supports anywhere from 100K to 200K users, while a GGSN supports as many as 1M to 3M users. That’s the reason a CDN is kinda worthless in a mobile 3G environment.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>