Case studies

The 27 best web performance links of Q1 2012

12 Apr 2012

3 Comments

This week, I had a chance to go through the latest links added to Strangeloop’s WPO Hub, and as always, I’m impressed by the breadth and depth of the topics covered. There are a lot of smart people in our industry.

A few things I noted in the first quarter of 2012:

  • We saw a lot of jockeying for performance dominance and audience dominance among browsers, indicating that the war isn’t over yet. This is great news for all of us — from users to site owners.
  • Our understanding of issues that affect mobile performance and third-party performance — topics that most of us were ignorant of just a couple of years ago — is increasingly nuanced.
  • More and more, we’re seeing mainstream coverage of web performance, signalling that this topic has permanently entered the public spotlight. It definitely makes my job easier. :)

Mainstream coverage

For Impatient Web Users, an Eye Blink Is Just Too Long to Wait
Really great piece in the New York Times about Google’s research into the neuroscience behind perceived page speed. Excerpt: ”Remember when you were willing to wait a few seconds for a computer to respond to a click on a Web site or a tap on a keyboard? These days, even 400 milliseconds — literally the blink of an eye — is too long, as Google engineers have discovered. That barely perceptible delay causes people to search less.”

How One Second Could Cost Amazon $1.6 Billion in Sales
Fast Company published this piece about our (lack of) tolerance for wait times in general, and about page speed in particular. Among other things, they cite the stat that one in four people abandons a page if it takes longer than four seconds to load.

Webpages showing sharp growth in girth
This was a fun example of the trickle-up process in action. On December 20th, I predicted that the average web page would hit 1MB in 2012. On December 21st, GigaOM picked it up. On December 22nd, it showed up on BBC News. I love the internet.

Think Quarterly: The Speed Issue
Google’s “Think Quarterly” isn’t exactly a mainstream publication, though it’s pretty widely read in the tech community, but I’m including this link here because I love the broad approach — ranging from technical to philosophical — of this particular issue, which is dedicated to the idea of speed. It includes articles by Urs Hoelzle, Kristen Gil, Jeff Jarvis, and Tjaco Walvis. Definitely a must-read.

Browsers

Which Browsers are the Fastest?
Interesting study from New Relic. Key findings were that, while IE 9 outperforms other browsers on Windows, Chrome 13 on Mac was overall the fastest experience (2.4 seconds). In mobile speed tests, the fastest experience was on Blackberry Opera Mini (2.6 seconds), twice as fast as Safari 5.1 on iPad.

Internet Explorer market share surges, as IE 9 wins hearts and minds
Summary: ”The browser wars are back on in earnest. For the second time in three months, Internet Explorer made large gains, picking up almost 1 point of market share. Chrome, Firefox, and Safari all lost out, as Internet Explorer 9 won over new users.”

Browser Speed Tests: Chrome 17, Firefox 10, Internet Explorer 9, and Opera 11.61
As always, the latest round of Lifehacker browser speed tests generated some interesting results. Overall, Chrome was the winner, but the tests crossed a lot of parameters, and each browser had its strengths. Definitely worth checking out, especially if you’re new to understanding the nuances of browser performance.

Internet Explorer Performance Lab: reliably measuring browser performance
The MSDN blog offers a detailed look behind the scenes at the IE Performance Lab. Be sure to check out the comments, too.

Chrome Fast for Android
A must-read if you care about mobile web performance: Google engineer Tony Gentilcore’s top 10 Chrome for Android speed features.

Chrome 16 – World’s Most Popular Browser
Interesting observation from Google engineer Mike Belshe: In January 2012, Chrome 16 quietly became the world’s most popular browser.

Case studies and other research

Just One Second Delay In Page-Load Can Cause 7% Loss In Customer Conversions
The folks at TagMan partnered with the UK’s leading glasses e-tailor, Glasses Direct, to study page speed and conversion behavior. Their findings substantiated Aberdeen’s 2009 findings, in which slower page load times correlated with a 7% drop in conversion rate.

Real User Monitoring at Walmart.com
Cliff Crocker, Aaron Kulick, and Balaji Ram joined forces at a meeting of the SF Web Performance Meetup to tell a RUM (real user monitoring) story through the lens of three job functions at Walmart.com: the performance data analyst, the developer, and the business analyst. Some excellent slides demonstrating the business value of performance. (I featured my four favourite slides in this post.)

The Performance Golden Rule
Excellent post from Steve Souders. He revisits his six-year-old stat about where end-user response time happens and confirms that 76-92% still happens at the front end. I really love this kind of post. In our industry, it’s crucial to keep revisiting these kinds of numbers and testing our assumptions.

The Need for Speed
Great case study for web performance geeks: How Chemeo delivers “pure bounce with pure value”.

No framework needed
Nice case study from 37signals about how they shaved 500ms from a page and got a 5% conversion bump.

January 2012 Site Performance Report
It’s great to trumpet success stories, but we learn as much by talking about what’s not working as by talking about what is.  This report card — in which the team at Wayfair shares how they suffered page slowdowns of up to 370% —  is a good reminder that optimization is a neverending task.

When good back-ends go bad
In this post for the annual Performance Advent Calendar, Pat Meenan talks about the fact that, while the back end is only responsible for 10-20% of response time, when it goes wrong, it can really go wrong. Among other things, Pat shares his findings that ”It is not uncommon to see 8-20s back-end times from virtually all of the different CMS systems.” Ouch.

Mobile

M-commerce site performance hampered by Verizon
Not exactly news, but still a good reminder from Internet Retailer that you’re at the mercy of networks when it comes to mobile performance: “The load times for virtually all of the 30 retailers on the Keynote Mobile Commerce Performance Index for the week ending Feb. 12 were negatively affected by a dip in performance on the Verizon wireless network.”

Mobile UI performance considerations
More goodness from the Performance Advent Calendar. If you’re just starting out with mobile web performance and don’t know where to begin, this post by Estelle Weyl is a good place to start. You should also check out her session slides from Velocity EU.

Tips and how-tos

Web Font Performance: Weighing @font-face Options and Alternatives
Nice collection of tips for making sure your web fonts aren’t slowing down your pages.

Timing the Web
Still more Advent Calendar goodness. dynaTrace technology strategist Alois Reitbauer explains how to use navigation timing to measure page load time.

Third parties

3rd Party Providers and DNS Poisoning Risks
Here’s a topic that doesn’t get talked about enough, via the Catchpoint blog: ”DNS cache poisoning refers to data breach, whereby new DNS records are introduced in the DNS Cache of a resolver or computer and divert traffic to a different IP address. The cache poisoning can be caused by hackers trying to divert traffic to a phishing/malware sites or it could be a configuration mistake. In either case the end result is not good for end users – they will end up accessing the wrong site or not access your site at all.”

Are External Services Slowing You Down? New Relic Infographic Reveals the Fastest and Most Popular External APIs
More research (with infographics) from New Relic, this time showing the four fastest 3rd-party APIs among the 200,000+ applications the company monitors.

Tools

Welcome YSlow Open Source
Yahoo has released the source code for YSlow under the BSD Open Source license. They’re encouraging devs to “use the source code, learn how it works, fork it to make your own projects and enhance it with new rules, features, and whatever will improve this tool we all love.”

Google: SPDY Gaining Adoption
Article in Web Pro News about the fact that SPDY is gaining support among browsers, developers, and vendors: “Recently, Firefox has added SPDY support, which means that soon half of the browsers in use will support SPDY. On the server front, nginx has announced plans to implement SPDY, and we’re actively working on a full featuredmod-spdy for Apache. In addition, Strangeloop, Amazon, and Cotendo have all announced that they’ve been using SPDY.”

Introducing mod_spdy, a SPDY module for the Apache HTTP server
Google engineers Matthew Steele and Bryan McQuade break down how Google’s mod_spdy for Apache works.

The Biggest Misconception about Google Page Speed
Really interesting findings from Catchpoint: “There is no correlation between Page Speed Score and how fast a page loads.” This corroborates our own findings here at Strangeloop. A while back, I was talking about this with Pat Meenan, and a probable reason for the drop is because the newest version of Page Speed checks for more optimizations, which has resulted in lower scores across the board. It’s also important to know that Page Speed doesn’t take into account advanced front-end content optimization techniques — nor could it, because there are way too many of them to track and measure. As a for-instance, we’ve had experiences here at Strangeloop where we’ve taken a page with a perfect Page Speed score, accelerated it with Site Optimizer, cut load time in half, and ended up with a new Page Speed score of 74%.

And finally, a shameless plug for Strangeloop’s annual performance state of the union, which came out in January. We tested the top 2,000 Alexa-ranked retail sites and analyzed the results against our findings from the previous year. Among other things, we found that the average e-commerce website takes 10 seconds to load, web pages are getting bigger, and Internet Explorer 9 outperforms other browsers.

If you have any other links to share, let me know in the comments. And if you’re looking for more great links, we have hundreds — sorted by topic, industry, and type — over in the Strangeloop WPO Hub.

Related links:

By: Tammy Everts Category: Tags: , , , , , , , , , , ,

4 awesome slides showing how page speed correlates to business metrics at Walmart.com

28 Feb 2012

42 Comments

When one of the first slides in a presentation offers this level of candor, you know you’re in for a treat:

Walmart web performance optimization case study

File this presentation under “I wish I’d been there in person.” Cliff Crocker, Aaron Kulick, and Balaji Ram joined forces at February’s SF Web Performance Meetup to tell a RUM (real user monitoring) story through the lens of three job functions at Walmart.com: the performance data analyst, the developer, and the business analyst.

You can check out the full slide deck here (you need to be a Minus member to view it online, but anyone can download it), but I want to highlight my favourite slides.

First, some back story.

The problem: Folks at Walmart knew their pages were slow. As a for instance, initial measurement showed that an item page took about 24 seconds to load for the slowest 5% of users. Why? The usual culprits: too many page elements, slow third-party scripts, multiple hosts (25% of page content is served by partners, affiliates, and Marketplace), and various best practice no-nos.

The goal: Create and meet a performance SLA that would see Walmart’s 95th percentile traffic hit 20 seconds.

The approach: To get to that goal, they dedicated a scrum team to one sprint of performance optimization. At the start of the process, the team performed some baseline measurements in which they used their RUM data to look at the load times of key pages and look for patterns. Then the team would create targets for page performance and at the end of the sprint measure the impact of optimization on key metrics.

The following slides are a great use of historical RUM data to make a powerful case (embodied in compelling graphs) for investing in performance. Throughout this set, there’s a clear, consistent correlation  between load time and conversion.

Awesome slide #1: Overall, converted shoppers received pages that loaded 2X faster than non-converted shoppers.

This should be of interest to anyone who’s asked themselves how to set targets for their own performance.

Walmart web performance optimization case study

Awesome slide #2: The above trend persists, even on individual pages that experience greater load times.

An example is show here, in which a specific page loaded almost 2 seconds faster for buyers than it did for non-buyers. (There’s nothing I love more than seeing parallel lines on a graph like this.)

Walmart web performance optimization case study

Awesome slide #3: Non-buyers were served category pages that were 2-3 seconds slower than buyers.

I really appreciate the comprehensiveness of this set of slides. It’s a good proxy for measuring page flows, which is a much trickier task, due to the massive number of permutations involved. (Learn how you can do something similar using WebPagetest.)

Walmart web performance optimization case study

Awesome slide #4: Bounce rate strongly correlates to page speed.

No huge surprise here, but great to see this validated.

Walmart web performance optimization case study

Conclusions: Optimization resulted in improved conversions, revenue, and SEO

You may be happy to know that the Walmart team not only hit their SLA at the end of their optimization sprint, they actually overreached it by almost 3 seconds. They note the following benefits:

  • For every 1 second of improvement they experienced up to a 2% increase in conversions
  • For every 100 ms of improvement, they grew incremental revenue by up to 1%
  • SEO benefits for entry pages and reduce bounces

Do it yourself

Check out the original slide deck to learn how Walmart used RUM tools, first to baseline performance and its benefits, and then to measure the results of optimization.

If RUM isn’t an option for you, then read this post to learn how to identify and measure the performance of sample page flows using WebPagetest or HTTPwatch. And then watch this short video to learn how to use Google Analytics to perform a page speed/revenue analysis of the pages in your flows.

Before I sign off, I want to applaud the team at Walmart for providing this level of disclosure. Case studies like this are a massive boon for our industry. It’s always inspiring to see e-commerce giants joining the likes of Amazon and Shopzilla in their willingness to share data.

Related posts:

By: Tammy Everts Category: Tags: , , , , , ,

The 20 best web performance links of Q4

15 Dec 2011

1 Comment

Every time I write one of these posts, I’m impressed by the volume and quality of writing that happens in our industry on an ongoing basis. It’s truly an exciting time for web performance. I feel endlessly engaged by the dialogue that happens every day, and honoured to be part of it it.

This roundup (which includes links pulled from the Strangeloop WPO Hub), includes some increasingly refined thinking about mobile optimization, a handful of excellent tutorials and case studies (including some great new presentations from Velocity EU), and some revolutionary browser developments.

But my favourite link is this first one…

The best link of Q4

Retailers need for tech speed
Does it tell us anything new? No. But I’ve lost count of how many people I’ve forwarded this two-minute segment on CNBC’s “Power Lunch” — which discusses the importance of speed for e-commerce sites, particularly for mobile users, during the holiday shopping season. For me, this shows that site speed has finally jumped into the mainstream. I’m excited to see how this attention snowballs in 2012.

Mobile

Mobile UI Performance
This slide deck from Estelle Weyl’s excellent presentation at Velocity EU gives an overview of mobile performance challenges, why we need to address them differently than we deal with desktop sites, and detailed tips on how to do just that.

Performance Automation 101
This slide deck from Jeroen Tjepkema’s Velocity EU presentation explains what performance automation is, how it works, and why it’s the only viable solution for dealing with the challenges of mobile device/browser fragmentation.

HTML5 Techniques for Optimizing Mobile Performance
Great post on HTML5 Rocks: ”In this article, we will discuss the bare minimum of what it takes to create a mobile HTML5 web app. The main point is to unmask the hidden complexities which today’s mobile frameworks try to hide. You will see a minimalistic approach (using core HTML5 APIs) and basic fundamentals that will empower you to write your own framework or contribute to the one you currently use.”

Mobile Performance Manifesto
Love this post from David Calhoun itemizing and describing mobile performance best practices.

Tools

How WebPagetest works
If you’ve ever wondered how exactly WebPagetest gathers performance data from the various browsers it simulates, this is great post from Pat Meenan in which he cracks the hood of WebPagetest and explains all that.

Mobile Perf Bookmarklet
Steve Souders offers one mobile bookmarklet to rule them all: a new “master bookmarklet” that lets you install a handful of common debugger and profiler bookmarklets in your mobile broswer in one step.

Is Synthetic Monitoring Really Going to Die?
Alois Reitbauer asks: “Will User Experience Management using JavaScript agents eventually replace synthetic monitoring or will there be a coexistence of both approaches in the end?” As you might guess, the answer is not cut and dried.

Case studies, how-tos, and other research

Diagnosing Slow Web Servers with Time to First Byte
Much as it pains me to admit it, from time to time performance pains aren’t caused at the front end. Performance expert Andy King gives some good tips on how to use the time to first byte metric, as displayed on a waterfall chart, to help diagnose a slow server.

The art and craft of the async snippet
Stoyan Stefanov examines the topic of asynchronous code “from the perspective of a third party – when you’re the third party, providing a snippet for other developers to include on their pages, be it an ad, a plugin, widget, visits counter, analytics, or anything else.”

Why loading third party scripts async is not good enough
We talk about asynchronously loading third-party snippets as if that’s the sole cure for performance pains, but in this case study, Aaron Peters reminds us that sometimes it’s okay to defer their loading until after onload.

Fast Loading JavaScript
Slide deck from performance consultant Aaron Peters’ great Velocity EU presentation: “A walk-through of several JavaScript loading techniques with a characteristics table for each and at the end a decision tree to help you decide which technique to use.”

How Downtime Financially Impacts Top Ecommerce Websites
Compelling infographic showing how downtime affected the Internet Retailer 500 in 2010. Includes the estimate that downtime resulted in more than $300 million in lost revenue for the IR 500.

Testing for Frontend SPOF
Excellent post from Pat Meenan in which he simulates third-party outage with a blackhole server in order to demonstrate — via WebPagetest-generated video — how that outage slows down or disrupts page load.

Browsers and connectivity

SPDY of the Future Might Blow Your Mind Today
Great post (“definitely for protocol geeks”) by Google software engineer Mike Belshe on SPDY’s evolution and how Kindle Silk is taking it beyond other browsers.

Chrome Fast
Slides from Google software engineer Tony Gentilcore’s excellent presentation at Velocity EU, in which he gives an overview of the Chrome platform and explains what makes Chrome fast.

Report reveals drop between peak and off-peak surfing
No big surprise, but a good reminder (especially if you rely on synthetic testing) that real-world performance is a nebulous thing: UK study finds that web speed is up to 69% slower during evening peak time.

The end of an era: Internet Explorer drops below 50% of Web usage
Mark the month and year. November 2011 was the first time in more than a decade that Internet Explorer’s share of global browser usage dropped below 50%.

Opinion pieces

Your CDN is not a silver bullet for web performance
In the e-commerce and SaaS world, the two most common causes of poor web performance are third-party content and server-side processing. Neither of these bottlenecks are addressed by loading static content from a closer location via a content delivery network.

Why you have less than a second to deliver exceptional performance
dynaTrace’s Alois Reitbauer writes: “Being exceptionally fast is becoming the dogma for developing web applications. But what is exceptionally fast and how hard is it to build a top performing web site?” I like posts like this because they remind us what the fundamental questions are that our industry is trying to address.

If you have any other great links to share, let me know in the comments. And if you’re looking for more great links, we have hundreds — sorted by topic, industry, and type — over in the Strangeloop WPO Hub.

Related posts:

By: Tammy Everts Category: Tags: , , , , , , , , , , , , ,

Case study: How to use network quality as a proxy for measuring mobile performance

6 Dec 2011

3 Comments

As Steve Souders recently wrote, measuring mobile performance is hard. There are a number of reasons for this. Just a few:

  • The same event can report completely different timings on different browsers.
  • Depending on which measurement tool you use, you can get completely different results than the results generated by another tool.
  • And more central to the problem, no one can even agree on what metrics we should be optimizing for.

In my recent presentations at Velocity EU and Velocity China, I presented some real-world case studies that show the relationship between mobile performance and business key performance indicators such as revenue, conversions, and bounce rate. I was able to get a lot of my data thanks to a couple of Strangeloop customers who were as curious as I was. These customers were willing to segment a portion of their mobile traffic and serve them unoptimized, slower pages, then compare results against the optimized pages.

But this kind of experimentation isn’t always possible. How do you measure the impact of mobile performance changes when slowing down pages isn’t an option? Or if you want to see the impact of page slowdowns that exceed the 1s or so that my customers were willing to allow?

I’m a big fan of using proxies to identify performance trends, as you may recall if you read my post on how to use browser type and network connection as proxies, via Google Analytics, to see the relationship between site speed and revenue. (We’ve since turned this post into an 11-minute video.) So we here at Strangeloop decided to investigate possible proxies for mobile performance.

Methodology

We set our sights on network quality as a viable proxy. We gathered data on bounce rate and performance for iPad, Android, and iPhone users of a single e-commerce site, which had already been optimized using RUM. Using beacons, we also gathered data about the users’ latency and bandwidth.

We took this vast data set and divided it into cohorts based on network quality: 250 Kbps (300ms+ latency), 500 Kbps (200-300ms latency), 1 Mbps 150-250ms latency), and so on in 0.5 Mbps increments all the way up to 5 Mbps. In other words, we tracked how bounce rate changed between users with a really crappy modem to a really fast connection.

Findings

What happens to bounce rate and the average performance across these groups as network speed improves? As this animated graph shows, the dots for each group — iPad, Android, and iPhone — all start moving down and to the left:

In other words, as network connection speed improves, performance improves, and as performance improves, bounce rate improves. It doesn’t matter which line you follow, the trend is clear and consistent. This tells us that performance matters across the entire spectrum of users.

It’s also interesting to note that iPad users are much less patient at high speeds than they are at low speeds. When things are really slow, iPad users bounce at about the same rate as Android and iPhone users. But as network speed improves, iPad users tend to stay and their bounce rate gets dramatically lower — around 5% compared to 8% for iPhone users and 11% for Android users.

This is expecially interesting given what we now know about conversion rates for iPad owners versus other mobile owners. (Over the Black Friday weekend, shoppers using iPads converted at a much higher rate than other mobile consumers, 4.6% vs. 2.8% for users of all other mobile devices.) Clearly, keeping your iPad traffic happy should be a priority.

Takeaway

As I mentioned to the crowd at Velocity China yesterday, this case study might be boring to hardcore performance geeks because it doesn’t have code snippets, but it’s important because it will justify investing in the code snippets they want to write. This how-to is a really helpful way to present stats to a site owner and tell them, hey, measuring performance optimization isn’t about 500 ms increments, or even 1s increments. When we look across the entire scope of performance, we can see bounce rates going from 24% all the way down to 5% as network quality gets better.

Related links:

By: Tammy Everts Category: Tags: , , , ,

Case study: The impact of HTML delay on mobile business metrics

23 Nov 2011

19 Comments

When I gave my talk about mobile performance and business KPIs at Velocity Berlin a couple weeks back, one of the areas I got the most questions about later was the experiments we were able to do in which we delayed HTML on a customer’s site and tracked the results over a 12-week period. I thought it might be useful to break some of this out into its own post.

As I mentioned at Velocity, I was insanely jealous of Google and Bing a couple years ago, when they revealed their own in-house experiments with HTML delay. Most of us in the performance community would kill for that kind of experimentation platform. So I was extremely happy and grateful when one of our customers at Strangeloop expressed an interest in figuring out the value of time for their business.

Methodology

We conducted a split test over the course of 12 weeks, in which we segmented mobile traffic into four groups: fully optimized, 200 ms delay, 500 ms delay, and 1000 ms delay. We monitored four metrics: bounce rate, conversion rate, cart size, and page views. We also monitored and analyzed user behavior for 6 weeks after the test ended, to gauge the long-term impact, if any, of slow performance even after users begin to receive an accelerated site.

Results

The results of the 200 ms delay weren’t significant, but the customer and I were both taken by the dramatic impact of the 500 ms and 1000 ms slowdown. Our customer was blown away that they were losing 3.5% percent of their conversions when their site was delayed by just one second on a mobile device. This was a major epiphany for them, and it’s already helped them change their business and how they view mobile.

I’m including this next graph to reinforce the connection between load time and user behavior.

Over the 12 weeks, you can see that, while the HTML delays were constant — 500 ms and 1000 ms — users’ reaction to these delays fluctuated. The drop in bounce rate ranged from around -2% and -12% for users who experienced a 1000 ms delay, and 0% and -6% for those who experienced a 500 ms delay. While the bounce rate may have varied, the one thing that was constant was the fact that the behavior trends are strongly linear for both groups, and the bounce rate for the 1000 ms group was consistently worse.

Finally, and most interestingly to me, we wanted to look beyond just the effect of delay in the timeframe of the experiment. We know that slower pages have an immediate impact on user behavior and customer satisfaction. We wanted to find out if there was any long-term impact on customer satisfaction.

So we looked at our traffic data for the 6 weeks immediately after the experiment — specifically at the behavior of returning visitors. As any site owner will tell you, repeat customers are the bread and butter of an e-commerce vendor. These are the people you need to keep happy. If you look at the graph below, you can see that, even after the experiment was over and the shoppers in the 500 ms and 1000 ms group started to be served the same accelerated site as the baseline group, they were significantly less likely to return to the site. By the end of the 6-week period, you can see that return traffic is slowly improving as visitors seem to finally be recovering from their poor experience.

Conclusions

As I’ve already mentioned, these findings have been a huge revelation to the company that owns this site. It’s had a major impact on how they’re tackling performance on their mobile platform. The most important over-arching takeaways from this experiment were:

  • Mobile shoppers are now fully engaging with e-commerce sites in significant enough numbers that we can analyze their behavior as a group.
  • Even a 500 ms delay has a major impact on metrics. For mobile sites, which can suffer excruciating load times (the latest Keynote index is about 12 seconds), this is a wake-up call that they need to take a hard-line approach to optimizing their pages.
  • The damage of poor performance is lasting.

See the full presentation: Case Studies from the Mobile Frontier: The Relationship Between Faster Mobile Sites and Business KPIs

Related posts:

By: Tammy Everts Category: Tags: , , , , , , , ,
1 2 3 4 5 6 7 8