Research

Case study: How effective are CDNs for mobile visitors?

9 May 2013

9 Comments

Substitute “measuring mobile performance” for “herding cats” in this video, and you’ve pretty much nailed the challenge we’re up against every day.

Fortunately, we like cats. :)

Experiment: Measuring the impact of CDN deployment on 3G performance

As we continue to evolve our mobile treatments, we also monitor their effectiveness alongside other optimization solutions. Today I want to call out some interesting results we noted when, as a fun little in-house exercise, we took the O’Reilly website, de-optimized it, and then iterated through a handful of core performance best practices using our FEO service. The goal was to demonstrate the acceleration benefit (in terms of bytes in, start render time, document complete time, connections, and resources) of each practice for a typical 3G mobile user.

While we saw predictable results for step 1 — enabling keep-alives and compression — we were somewhat surprised by what we saw when we added a content delivery network.

Step 1: Apply fundamental best practices

First we added keep-alive connections:

  • What it does: Lessens the impact of TCP connection setup
  • Benefit: Addresses the problem of having too many TCP connections

Then we added HTTP compression:

  • What it does: Compresses text-based content (HTML, CSS, JavaScript, etc.)
  • Benefit: Easy way to reduce bytes/payload

We got the results we expected: faster start render, and about a 25% reduction in doc complete time. This is fantastic, even more so because both these treatments are really easy to do — usually it’s just a matter of a single configuration option on your server, proxy, or load balancer. However, these two treatments aren’t enough to give you the acceleration you need.

Step 2: Use a content delivery network (CDN)

Here’s the elevator-ride explanation of how a CDN works (for an excellent detailed explainer, go here): Static page assets (images, CSS, etc.) are served from locations near the requesting client (mobile or otherwise). The shorter distance between client and content means smaller time to first byte (TTFB) and, ostensibly, faster start render. This means users start to see content in their browser faster.

How a content delivery network works

In our experiment, we expected that adding a CDN would result in faster average time to first byte, start render, and doc complete time. Here’s what we saw:

Before and after: Document complete (aka load time)

Before and after content delivery network deployment

Before and after: Time to first byte (TTFB)

Before-and-after CDN: Time to first byte (TTFB)What we helped

CDN before and after: What we helpedFindings

With a CDN, the page got faster, but not by much. For the unoptimized page, we forced requests to travel from west coast to east coast. After, we let the CDN naturally select the closest edge. As a result, we saw that:

  • Doc complete time decreased by 10%, compared to a 20% improvement we noted in a similar experiment in desktop optimization.
  • We shaved less than a second off start render time, taking it from 7.059 seconds down to 6.245 seconds.
  • True, we cut time to first byte by 39%, but from an end-user perspective, TTFB doesn’t really mean anything because the user still isn’t seeing anything in the browser.

Five questions about CDNs and mobile acceleration

I’m not making any nutty claims like “CDNs aren’t effective for mobile devices.” (Heresy!) But these results do raise a few questions:

  1. Is edge selection for mobile devices not as effective as for desktop?
  2. Some, if not all, CDNs probably deploy servers near cell network exit points. But what if most of the latency occurs inside the cell network? (I’ll admit I’m not an expert on what happens inside a 3G network. I’m ready to be enlightened.)
  3. Does the meaning of “closeness” change for mobile?
  4. Acknowledging the existence of mobile-specific CDNs, how much more effective are they? How do they compare when it comes to 3G versus WiFi? I’ve been trying to dig up case studies, with no luck.
  5. High CDN costs may be justifiable when you see significant benefit for your desktop traffic, but do they deliver sufficient benefit/ROI for mobile users?

Takeaway

LTE is grabbing a lot of attention, but it’s a mistake to sweep 3G under the rug. There are 256 million 3G subscribers in the US, representing 81% penetration, so 3G performance is still a big deal. We need more research. If you have findings to share, please do.

Related posts:

By: Tammy Everts Category: Tags: , , , ,

More new findings: Top ecommerce sites are 22% slower than they were last year

27 Mar 2013

8 Comments

Let me say that again, because this is a staggering fact: The world’s top ecommerce sites are 22% slower than they were last year.

In December 2011, the median load time for a site in the Alexa Retail 2000 was 5.94 seconds. Just twelve months later, the median was 7.25 seconds. At this rate of growth, this number could hit almost 9 seconds by the end of this year.

Web page load time changes: December 2011 to December 2012

This was the key finding of our brand-new quarterly report (yes, a new report, not to be confused with last week’s report about web performance in the EU) on ecommerce web performance. If you’re new to these reports, since 2010 we’ve been measuring the load time, page composition, and best practice implementation of the same set of 2,000 leading online retailers, as ranked by Alexa. The goal is to learn how pages are changing over time and what impact, if any, these changes have on per-page performance. The results have been eye-opening.

I have to confess that I frequently feel like that tiresome guy at the party who keeps saying the same things over and over again. Pages are getting slower… pages are getting bigger… the gap between load times and user expectations is getting wider almost by the week. If you’re reading this out there and saying to yourself, “I can’t believe that Bixby guy is going on about this AGAIN,” then forward the next part of this post to five people you think could really benefit from it. If enough people internalize this message, maybe I’ll shut up.*

Three performance myths I would give anything to permanently bust:

Myth #1. Pages are, de facto, getting faster.

What with our better systems, networks, and browsers, pages must be getting faster, right? Everyone believes this instinctively, because most of us seem to be hardwired to believe that technology solves problems rather than creating new ones. But as I said at the top of this post, the quickly emerging fact is that pages seem to not just be getting slower, they’re getting slower at an alarming rate. (Optional: You may choose to take this finding as proof that we shouldn’t always trust our instincts. :) )

Myth #2. Users are more or less satisfied with the status quo.

“People are used to pages that take 5-8 seconds to load. They don’t mind that much.” I still hear this on a regular basis. Site owners rationalize that, because they’re not hearing a lot of complaints, their visitors are happy… or at least happy enough. But as numerous case studies have shown, people talk with their wallets. Faster sites earn more. And user surveys over the years are telling us that people’s expectations for a speedy online experience are continually growing:

Web Page Load Time: User expectations 200-2012

Myth #3. Browser development is more than capable of mitigating the factors, such as page size and complexity, that are causing pages to slow down.

This belief is widely held, even among technical folks. Again, looking to our findings, we saw that for all three browsers, median load times slowed down by anywhere from 3% to 12% in just six months. This downward trend isn’t a browser development issue. Instead, it’s an indicator that despite browser vendors’ huge commitment to speed, development can’t keep pace with the demands of bigger and increasingly complex web pages.
Browser performance: 2011 to 2012

Takeaway: Pass it on.

I am extremely happy that Radware is committed to continuing the tradition of releasing these quarterly  “state of the union” reports. As time passes, we’re gaining some invaluable insights. Based on these latest results, I’m very curious to see what our Summer 2013 report will hold.

I urge you to download this report (and the infographics, too). And I was only slightly kidding when I suggested that you forward this post to people who need to have a few performance myths dispelled. Our community does a lot of preaching to the choir. What seems basic to us is not necessarily basic to the rest of the world. We need to get out there and make sure these simple messages are being heard.

*Maybe.

Related posts:

By: Tammy Everts Category: Tags: , , , , , , ,

New findings: Typical leading European commerce site takes 7.04 seconds to load

19 Mar 2013

3 Comments

Last fall, at Velocity London, I had a really great talk with Stephen Thair, who is a UK-based web performance consultant, Velocity committee member, WebPerfDays organizer, and all-around knowledgeable guy. Among other things, we talked about how frustrating it can be for performance pros based in Europe to preach outside their community.

As Stephen said:

“I guess it’s a bit frustrating in the UK at the moment. One of the things that I found is that we haven’t yet got that killer web performance case study in one of the big major retailers. So we are still, I think, a bit in the evangelical stage. We are still trying to get the message out there. There are still a lot of websites in the UK that don’t even have gzip turned on.”

So we set out to help fill that gap. In December of 2012, working with Radware (our soon-to-be parent company) in conjunction with our partners at Level 3 in Europe, we studied the page speed and composition of 400 top European retailers, as ranked by Internet Retailer magazine, to see how these sites would load for visitors using Chrome 23 (the most popular browser in the EU at the time of testing) via the test server in Amsterdam. (We chose the Amsterdam location because it allowed us to test across all major browsers.) The report was released today.

While the results may not be shocking if you’ve been paying close attention to this space, they may come as an eye-opener to online retailers in the EU. Our chief finding was this:

The median page took more than 7 seconds to load.

Depending on whom you ask, the average internet user expects web pages to load in less than 3 seconds, 2 seconds, or even 400 milliseconds. The last time the average person reported being cool with 7-second load times was around 2001.

The survey also found that:

  • 1 out of 4 pages took more than 10 seconds to load.
  • 1 out of 3 pages contained more than 100 resources.
  • 79% of sites don’t use a recognized CDN. (A “recognized CDN” refers to any CDN listed in the extensive directory of CDNs maintained by WebPagetest.)
  • Speaking to Stephen’s point about gzip at the top of this post, 1 out of 5 sites failed to implement text compression, a relatively simple technique that delivers easy, significant performance gains.

Why you should care about these findings

I may be pointing out the obvious, but it may need to be pointed out: these expectations are universal. Internet users in the EU do not have lower performance expectations than users in North America. These findings should be a wake-up call for European site owners. (Not that North American site owners should be resting easy. Last fall, we found that the median leading US commerce home page took almost 7 seconds to load.)

Download the report: State of the Union: European Ecommerce Page Speed and Web Performance

Related posts:

By: Tammy Everts Category: Tags: , , , , , , ,

Discuss: If you’re in the web performance business, you’re in the happiness business.

23 Jan 2013

5 Comments

It’s ironic that I’m writing this post at a time when my own flow has been seriously compromised by the arrival of my third child, who arrived a couple of weeks ago. But I’m going to do my best. Stick with me. :)

I’ve written about flow in the past. It’s a fascinating topic — one that I keep coming back to as I discover new (or new to me) research into it. Last year–

…

Sorry. Crying baby. What was I talking about? Oh yeah. Flow. What was that about irony?

Last week, I was up late (again: baby) and decided to watch a documentary called Happy, which, as you might guess, is an exploration of the factors that truly make us truly happy. I won’t give away all the secrets to happiness here, but I want to share one line from the movie that jumped out:

“People who experience flow on a regular basis
are happier than people who don’t.”

This was really interesting to me. In the past I’ve talked about research that links flow, in web terms, to conversions, pageviews, and revenue, but I’ve never explored the blunt statement that flow = happiness. So I decided to do some digging with this angle in mind.

What is flow?

First, let’s back up for a minute and define our terms. In web performance, when I talk about flow, I’m talking about one of two things:

A. Flow as a user’s path through a site or application.
B. Flow as a descriptor for the seamlessness of a sequence of actions.

Ideally, you want to experience as much as possible of flow B while you’re experiencing flow A.

Mihaly Csíkszentmihályi (considered by many to be pioneer of the concept of flow) lists the ideal components of flow in his 1998 book Finding Flow: The Psychology of Engagement with Everyday Life:

  • Clear goals that, while challenging, are still attainable.
  • Strong concentration and focused attention.
  • The activity is intrinsically rewarding.
  • Feelings of serenity; a loss of feelings of self-consciousness.
  • Timelessness; a distorted sense of time; feeling so focused on the present that you lose track of time passing.
  • Immediate feedback.
  • Knowing that the task is doable; a balance between skill level and the challenge presented.
  • Feelings of personal control over the situation and the outcome.
  • Lack of awareness of physical needs.
  • Complete focus on the activity itself.

What gets in the way of flow?

If, like most people, you live a busy urban life — particularly one that involves new babies — achieving flow has countless challenges from the time you get up until you go to bed:

  • Sitting in traffic
  • Workplace interruptions (phone calls, people stopping at your desk, last-minute tasks, impromptu meetings)
  • Standing in lines (Side note: As an exercise, count how many lines you stand in on a given weekday.)
  • Waiting for elevators
  • Texts, email, and other mobile beeps
  • More sitting in traffic
  • Bizarre inexplicable demands of small children
  • Sudden random diaper failures

This list was disturbingly easy to write, and I’m sure I could go on. To summarize, achieving a state of flow is hard work.

Now let’s hone in on our area of focus: web performance.

As I’ve written in the past, we humans are hard-wired to perform tasks seamlessly. That’s because for hundreds of thousands of years, our brains have evolved to help us carry out day-to-day tasks — from building a fire to planting a field — that are comprised of a series of minute actions that flow virtually without interruption from one to the next.

It’s only in the past forty years, with the advent of computers, that we’ve imposed a new set of demands on our brains. As most of us are painfully aware, instead of offering a series of smoothly sequential actions, human-computer interaction is characterized by lag, downtime, and restarts.

In my travels, I encounter people who are skeptical about the impact of lag, downtime, and restarts on productivity and other key performance indicators. The argument I hear is that most people do, in fact, adjust to poor performance.

As it turns out, these people may be somewhat correct, but they may also be focusing on the wrong part of the picture.

Questioning our assumptions: Do delays really hurt productivity?

I recently came across a really interesting study into workplace interruptions: Temporal factors in mental work: Effects of interrupted activities (Fred R. H. Zijlstra and Robert A. Roe, 1999). In it, groups of workers were subjected to various disruptions in the course of their day-to-day responsibilities, and then were measured in terms of both their productivity and their self-reported state of mind. While this study focused on general workplace interruptions, with only some attention given to human-computer interaction, there were some fascinating findings that are arguably relevant to web performance.

Finding 1: Participants developed strategies that let them deal effectively with interruptions and maintain their productivity.

This research suggests that, at least for some workers in some environments, not only do they learn how to cope with interruptions, they may even strive to overcompensate for their potential performance decline.

Finding 2: However, this coping mechanism is achieved at the expense of higher psychological costs.

Cumulatively, interruptions had a negative impact on emotion and well-being. Participants ultimately needed to increase the amount of effort required to perform the same tasks.

Finding 3: Over time, interruptions affected participants’ ability and willingness to resume work and take on new tasks.

Interruptions seemed to have a cumulative effect. When the number of interruptions grew, the resumption time (i.e. the time needed to re-start the task) became disproportionally longer. The participants seemed to lose motivation and develop mental fatigue.

What does this mean in web performance terms?

It’s possible that people can develop coping strategies for dealing with application delays, and that these coping strategies can allow them to maintain productivity in the short term. But the missing ingredient here is flow. And without flow, eventually our sense of motivation and well-being suffers.

It’s also eye-opening to think about our small world of application performance as just one part of a bigger world. As I mentioned at the top of this post, our days are filled with challenges to flow. Poor web performance is just one factor, but it is a significant factor. Consider the cumulative effects of lack of flow in the fast-paced world most of us live in.

As countless studies have proven, human beings are really good at convincing ourselves that we understand what makes us happy, and we’re really bad at actually realizing what makes us happy. Because of this, it’s easy to kid ourselves that, because our productivity is more or less the same — and because we place a great deal of value on productivity — somehow this equates to happiness.

In other words, we can convince ourselves that we’re fine — or if you’re an employer, you can convince yourself that your workers are fine — when perhaps we’re not.

Here’s where I’m tempted to make a grandiose claim.

I’d love to close with something like “if you’re in the web performance business, you’re in the happiness business”, but I can sense the imminent mockery of my colleagues here at Strangeloop. And in all truth, I don’t believe that human beings have that much power to make each other happy.

But I do believe that, if you’re in the performance business, you’re in the flow business. By improving flow, we’re helping to remove an obstacle, and in doing this, in our own small way we’re allowing people to find their own happiness. I do believe this. Let the mockery commence. :)

Related posts:

By: Tammy Everts Category: Tags:

Browser innovation and the 14 rules for faster loading websites: Revisiting Steve’s work (part 3)

28 Nov 2012

2 Comments

This is the final post in a series in which I’ve been addressing a question that was put to me by a customer a few months ago:

“Aren’t many of the web performance rules described by Steve Souders in 2007 already outdated or made obsolete by browser innovation?”

You can check out parts 1 and 2, though it’s not necessary to read those first in order to understand today’s post. Today, I’m looking at the last four rules:

  • Avoid redirects
  • Remove duplicate scripts
  • Configure ETags
  • Make AJAX cacheable

Methodology

1. As with my previous post, I used the test cases of the performance rules that Steve created five years ago, which are still available. In my earlier posts, I  timestamped them to October 2007 using the Wayback Machine.

2. Next, I ran each test case on Internet Explorer 6 on WebPagetest to get a sense of what Steve would have seen for performance rules 11 through 14. I did three runs of each test and used the median result.

3. Then I ran each test case on Chrome 23 to see what impact, if any, the rules have on a modern browser. Again, I performed three runs of each test and recorded the median result.

4. Using the above approach, I was able to see and compare the before-and-after results for each rule for both browsers, and then calculate the benefit.

Reminder: As I stated in my earlier posts, the goal here is not to look at how fast Chrome 19 is versus Internet Explorer 6. We have looked at this in the past. This time I want to look at the relative benefit of each performance rule.

Findings

As it turned out, there were no test cases for three out of four of these rules, for reasons I’ll explain below. Here’s a high-level look at the results:

Rule 11: Avoid redirects

What this rule means: In broad terms, a redirect is a permanent or temporary redirect from one URL to the other. A permanent redirect is response code 301. There are multiple temporary ones, but the response code most commonly used to describe a temporary redirect is 302. There are several reasons why sites use redirects, such as fixing missing trailing slashes, connecting websites, and internal/outbound tracking, to name just a few.

Fixing missing trailing slashes

This is a really common dev mistake. As Steve says, “One of the most wasteful redirects happens frequently and web developers are generally not aware of it. It occurs when a trailing slash (/) is missing from a URL that should otherwise have one.”

Connecting websites

For example, if the URL for your old website was www.goodsite.com and you wanted to change it to www.bettersite.com, you’d implement a 301 redirect from the old URL to the new one. Now whoever typed in your old URL (or clicked on a leftover link to your old URL) would automatically get taken to your new URL.

Internal/outbound tracking

Redirects are a way to figure out where visitors are going next. When you’re about to leave a site (say, from a search result page), instead of just hyperlinking to the new site, the link takes you to a URL on the current site that then redirects you to the new site. That in-between URL tracks the fact that you’re leaving the site and where you’re going.

Browsers can’t fix redirect problems. This fix lies with site owners.

As Steve points out, “The main thing to remember is that redirects slow down the user experience. Inserting a redirect between the user and the HTML document delays everything in the page since nothing in the page can be rendered and no components can start being downloaded until the HTML document has arrived.”

As the rule states, avoid redirects. For example, don’t use redirects to keep track of clicks that leave your site. Instead, Steve offers two alternative techniques that send a non-blocking beacon as a visitor clicks away from the page, so the visitor doesn’t have to wait for the redirect before going to a new site.

Testing the rule: N/A

Rule 12: Remove duplicate scripts

What this rule means: It’s no surprise that duplicate scripts — which generate unnecessary HTTP requests and waste time evaluating the same script more than once — hurt performance. What is surprising is that this mistake happens so often. This issue is more likely to pop up when development teams are large and when pages contain a huge number of scripts.

As Steve said back in 2007, “Unnecessary HTTP requests happen in Internet Explorer, but not in Firefox. In Internet Explorer, if an external script is included twice and is not cacheable, it generates two HTTP requests during page loading. Even if the script is cacheable, extra HTTP requests occur when the user reloads the page.”

Testing the rule: Steve experimented with implementing caching on a page to test its ability to deal with duplicate scripts. As you can see in the table below, this had a significant impact on IE6. The impact on Chrome was negligible, probably because Chrome does better parallelism and is more aggressive with caching duplicate scripts, even with resources that were not coded to be cached.

I also tested different versions of Internet Explorer, in order to pinpoint the version that evolved to address the duplicate scripts problem. As you can see in the table below, this happened with IE8.

Test case
 Benefit in IE6 Benefit in IE7 Benefit in IE8 Benefit in
Chrome 23
Duplicate script – cached 20% 16% 1% 1%
Duplicate script – 10 cached 19% 15% 1% 1%

Rule 13: Configure ETags

What this rule means: There’s no test case for this rule — because this is a server issue, not a front-end issue — but I still want to review it here in order to talk about why this rule is important.

An ETag (also known as entity tag) is a string that uniquely identifies a specific version of a page object. Web servers and browsers use the ETag to determine whether an object in the browser’s cache matches the object on the origin server. When implemented correctly, ETags can help performance for repeat visits because they provide a kind of shortcut that allows the browser to validate objects more quickly.

Sounds good, right? But there’s a catch. As Steve points out, “The problem with ETags is that they typically are constructed using attributes that make them unique to a specific server hosting a site. ETags won’t match when a browser gets the original component from one server and later tries to validate that component on a different server—a situation that is all too common on web sites that use a cluster of servers to handle requests. By default, both Apache and IIS embed data in the ETag that dramatically reduces the odds of the validity test succeeding on web sites with multiple servers.”

As Steve counsels, “If you’re not taking advantage of the flexible validation model that ETags provide, it’s better to just remove the ETag altogether. The Last-Modified header validates based on the component’s timestamp. And removing the ETag reduces the size of the HTTP headers in both the response and subsequent requests.”

To clarify: ETags are not bad things in and of themselves. They’re actually a more accurate way of performing invalidation. But, because of their preciseness, if you want to harness the power of ETags, you need to make sure the same ETag is deterministically generated across your servers for a specific resource. If you’re not going to do that, it’s best not to use them, since misconfigured ETags will hurt you.

Testing the rule: N/A

Rule 14: Make AJAX cacheable

What this rule means: No test case for this rule, this time because Steve counsels that you apply the same performance rules already discussed to your Ajax requests — particularly having a far future Expires header.

Testing the rule: N/A

Let’s recap.

Here’s a summary of everything I learned over the course of revisiting all 14 of Steve’s rules:

Most of the performance rules have stood the test of time.

As I stated in the earlier posts, it’s striking how similar the results are for many of the test cases. I had expected that the rules would still be relevant, but I had also expected that browser evolution would have resulted in a greater gap between the results for IE6 and Chrome. If you build web applications and you care about performance, Steve’s book should still be your bible.

Reducing roundtrips still matters.

It doesn’t matter how much better modern browsers are at rendering page objects; fewer calls to the server still make a huge difference. Seeing that adding an Expires header leads to a 59% improvement in Chrome 19 tells me this technique is still incredibly relevant.

A CDN helps in some situations, but not all.

A CDN is a must for many sites, but it’s not a standalone performance solution. Benefits will vary depending on which CDN you choose, as well as things like how your CDN stores content and how far their PoPs are from users.

Compression still helps. A lot.

Not only did Gzipping offer benefits in Chrome, it offered even greater benefits than in IE6. This is a really compelling finding. It shows that, despite the advances made in modern browsers, modern pages have also changed a lot — meaning that more than ever they can benefit from compression.

The location of stylesheets is important, but it depends on the page composition.

Start render really matters and the location of the stylesheets is critical.

“Avoid CSS expressions” is obsolete… for all the right reasons.

It was interesting to see how a rule — avoid CSS expressions — has become so entrenched over time that it’s no longer an issue.

Unless you’re developing for older versions of IE, you probably don’t need to worry about avoiding duplicate scripts.

Newer browsers take care of the duplicate script problem — possibly due to the mitigating impact of doing better parallelism and/or more aggressive caching. But be sure to know whether or not a significant portion of your traffic uses IE6 and 7. If they do, then you still need to apply this rule. If you don’t apply it, you could be missing out on an opportunity to make a 15-20% performance gain.

Conclusion

I had a great time with this set of posts and want to thank Steve, not just for pioneering the original set of rules, but for his support and advice as I worked my way through all my tests. Thanks, Steve!

Related posts:

By: Tammy Everts Category: Tags: , , , , ,
1 2 3 4 5 6 7 8 »