Velocity

Case study: How to use network quality as a proxy for measuring mobile performance

6 Dec 2011

3 Comments

As Steve Souders recently wrote, measuring mobile performance is hard. There are a number of reasons for this. Just a few:

  • The same event can report completely different timings on different browsers.
  • Depending on which measurement tool you use, you can get completely different results than the results generated by another tool.
  • And more central to the problem, no one can even agree on what metrics we should be optimizing for.

In my recent presentations at Velocity EU and Velocity China, I presented some real-world case studies that show the relationship between mobile performance and business key performance indicators such as revenue, conversions, and bounce rate. I was able to get a lot of my data thanks to a couple of Strangeloop customers who were as curious as I was. These customers were willing to segment a portion of their mobile traffic and serve them unoptimized, slower pages, then compare results against the optimized pages.

But this kind of experimentation isn’t always possible. How do you measure the impact of mobile performance changes when slowing down pages isn’t an option? Or if you want to see the impact of page slowdowns that exceed the 1s or so that my customers were willing to allow?

I’m a big fan of using proxies to identify performance trends, as you may recall if you read my post on how to use browser type and network connection as proxies, via Google Analytics, to see the relationship between site speed and revenue. (We’ve since turned this post into an 11-minute video.) So we here at Strangeloop decided to investigate possible proxies for mobile performance.

Methodology

We set our sights on network quality as a viable proxy. We gathered data on bounce rate and performance for iPad, Android, and iPhone users of a single e-commerce site, which had already been optimized using RUM. Using beacons, we also gathered data about the users’ latency and bandwidth.

We took this vast data set and divided it into cohorts based on network quality: 250 Kbps (300ms+ latency), 500 Kbps (200-300ms latency), 1 Mbps 150-250ms latency), and so on in 0.5 Mbps increments all the way up to 5 Mbps. In other words, we tracked how bounce rate changed between users with a really crappy modem to a really fast connection.

Findings

What happens to bounce rate and the average performance across these groups as network speed improves? As this animated graph shows, the dots for each group — iPad, Android, and iPhone — all start moving down and to the left:

In other words, as network connection speed improves, performance improves, and as performance improves, bounce rate improves. It doesn’t matter which line you follow, the trend is clear and consistent. This tells us that performance matters across the entire spectrum of users.

It’s also interesting to note that iPad users are much less patient at high speeds than they are at low speeds. When things are really slow, iPad users bounce at about the same rate as Android and iPhone users. But as network speed improves, iPad users tend to stay and their bounce rate gets dramatically lower — around 5% compared to 8% for iPhone users and 11% for Android users.

This is expecially interesting given what we now know about conversion rates for iPad owners versus other mobile owners. (Over the Black Friday weekend, shoppers using iPads converted at a much higher rate than other mobile consumers, 4.6% vs. 2.8% for users of all other mobile devices.) Clearly, keeping your iPad traffic happy should be a priority.

Takeaway

As I mentioned to the crowd at Velocity China yesterday, this case study might be boring to hardcore performance geeks because it doesn’t have code snippets, but it’s important because it will justify investing in the code snippets they want to write. This how-to is a really helpful way to present stats to a site owner and tell them, hey, measuring performance optimization isn’t about 500 ms increments, or even 1s increments. When we look across the entire scope of performance, we can see bounce rates going from 24% all the way down to 5% as network quality gets better.

Related links:

By: Tammy Everts Category: Tags: , , , ,

Case study: The impact of HTML delay on mobile business metrics

23 Nov 2011

20 Comments

When I gave my talk about mobile performance and business KPIs at Velocity Berlin a couple weeks back, one of the areas I got the most questions about later was the experiments we were able to do in which we delayed HTML on a customer’s site and tracked the results over a 12-week period. I thought it might be useful to break some of this out into its own post.

As I mentioned at Velocity, I was insanely jealous of Google and Bing a couple years ago, when they revealed their own in-house experiments with HTML delay. Most of us in the performance community would kill for that kind of experimentation platform. So I was extremely happy and grateful when one of our customers at Strangeloop expressed an interest in figuring out the value of time for their business.

Methodology

We conducted a split test over the course of 12 weeks, in which we segmented mobile traffic into four groups: fully optimized, 200 ms delay, 500 ms delay, and 1000 ms delay. We monitored four metrics: bounce rate, conversion rate, cart size, and page views. We also monitored and analyzed user behavior for 6 weeks after the test ended, to gauge the long-term impact, if any, of slow performance even after users begin to receive an accelerated site.

Results

The results of the 200 ms delay weren’t significant, but the customer and I were both taken by the dramatic impact of the 500 ms and 1000 ms slowdown. Our customer was blown away that they were losing 3.5% percent of their conversions when their site was delayed by just one second on a mobile device. This was a major epiphany for them, and it’s already helped them change their business and how they view mobile.

I’m including this next graph to reinforce the connection between load time and user behavior.

Over the 12 weeks, you can see that, while the HTML delays were constant — 500 ms and 1000 ms — users’ reaction to these delays fluctuated. The drop in bounce rate ranged from around -2% and -12% for users who experienced a 1000 ms delay, and 0% and -6% for those who experienced a 500 ms delay. While the bounce rate may have varied, the one thing that was constant was the fact that the behavior trends are strongly linear for both groups, and the bounce rate for the 1000 ms group was consistently worse.

Finally, and most interestingly to me, we wanted to look beyond just the effect of delay in the timeframe of the experiment. We know that slower pages have an immediate impact on user behavior and customer satisfaction. We wanted to find out if there was any long-term impact on customer satisfaction.

So we looked at our traffic data for the 6 weeks immediately after the experiment — specifically at the behavior of returning visitors. As any site owner will tell you, repeat customers are the bread and butter of an e-commerce vendor. These are the people you need to keep happy. If you look at the graph below, you can see that, even after the experiment was over and the shoppers in the 500 ms and 1000 ms group started to be served the same accelerated site as the baseline group, they were significantly less likely to return to the site. By the end of the 6-week period, you can see that return traffic is slowly improving as visitors seem to finally be recovering from their poor experience.

Conclusions

As I’ve already mentioned, these findings have been a huge revelation to the company that owns this site. It’s had a major impact on how they’re tackling performance on their mobile platform. The most important over-arching takeaways from this experiment were:

  • Mobile shoppers are now fully engaging with e-commerce sites in significant enough numbers that we can analyze their behavior as a group.
  • Even a 500 ms delay has a major impact on metrics. For mobile sites, which can suffer excruciating load times (the latest Keynote index is about 12 seconds), this is a wake-up call that they need to take a hard-line approach to optimizing their pages.
  • The damage of poor performance is lasting.

See the full presentation: Case Studies from the Mobile Frontier: The Relationship Between Faster Mobile Sites and Business KPIs

Related posts:

By: Tammy Everts Category: Tags: , , , , , , , ,

The relationship between faster mobile sites and business KPIs: Case studies from the mobile frontier [Velocity EU]

8 Nov 2011

5 Comments

As promised, here’s the slide deck from my Velocity keynote. A little back story for those who didn’t attend: this data was gathered over many months of the beta of Strangeloop’s Mobile Site Optimizer. It wouldn’t have been possible without the generous participation of two of our customers. They would prefer not to be named, but I’d still like to thank them here.

Velocity EU: The Relationship Between Faster Mobile Sites and Business KPIs: Case Studies from the Mobile Frontier

If you have any questions, please let me know in the comments or ping me on Twitter.

Related posts:

By: Tammy Everts Category: Tags: , , , , ,

Preview of my mobile keynote slides for Velocity Europe

7 Nov 2011

0 Comment

Tomorrow (Tuesday) morning at 10:35, I’ll be sharing some never-before-released case study data about mobile performance to the Velocity Europe crowd. To give a taste of what I’ll be talking about, here’s a sneak peek at a few of my slides:

Velocity Europe: Mobile Case Studies: M.site traffic & final sales

Velocity Europe: Mobile Case Studies: Impact of site delays on returning visitors

Velocity Europe: Mobile Case Studies: Impact of network quality and performance on bounce rate

I hope to see you there. I’ll be sharing the full slide deck shortly after my session.

There’s a great speaker lineup at this inaugural event. My schedule is filling up fast, but here are some of the sessions I’m hoping to attend:

If you’re at Velocity, too, I’d love to chat. Send me a note or ping me on Twitter.

Related posts:

By: Tammy Everts Category: Tags: , , , , ,

WPO? WCO? FEO? Unmuddying the web performance waters

10 Jun 2011

0 Comment

With Velocity coming up fast, now seems like a good time to write a post I’ve been meaning to write for a while, clarifying some of the language we use in our industry.

Bur first, a little context:

A couple of weeks ago, I was in London, talking with the Web Performance Meetup crew there about the history and taxonomy of the performance space. Stephen Thair has done an awesome job of summarizing that aspect of the session (which you can read about on his blog, so I won’t go into it in detail here), but suffice to say for my purposes today that performance solutions have, historically, fallen into two major groups:

  • Delivery solutions, such as content delivery networks, dynamic site accelerators, and load balancers, which focus on delivering the exact content the server produces from the server to the browser as quickly as possible.
  • Transformation solutions, which change the content itself, optimizing it so that it renders faster in the user’s browser.

For a long time, from 1993 till about 2006, delivery solutions ruled the performance marketplace. However, when Steve Souders announced four years ago, in his book High Performance Websites, that up to 85% of performance problems happen at the front end — at the browser level — it opened the door for transformation solutions. And this is when the waters started to get a bit muddy.

Hopefully, these definitions will unmuddy them:

Web performance optimization (WPO)

First public appearance: May 7, 2010

This is a catch-all term that applies to all performance solutions, whether delivery or transformation based. Steve Souders coined this term just over a year ago, and because Steve works in the front-end space, those of us who also work in this space quickly picked it up and ran with it. But as my colleague Fred Beringer pointed out recently:

website performance optimization

Fair enough. There have been efforts to introduce more specific terminology into the transformation industry, such as…

Front-end optimization (FEO)

First public appearance: January 10, 2011

Dan Rayburn wrote a really excellent post in which he defines front-end optimization:

FEO might sound similar to another subject I have written about lately, dynamic site acceleration (DSA), but it’s very different. DSA’s focus is to bring network resources closer to the user by prefetching or caching files. FEO makes the content itself faster. DSA makes page resources download faster. FEO reduces the number of page resources required to download a given page and makes the browser process the page faster. For example, analysis shows that popular sites like CNN, who already use a CDN, can double current performance by implementing FEO.

In other words, FEO is a transformation solution, as I described near the top of this post. However, different people have different interpretations of what constitutes the “front end”, so even this term might not be clear enough.

Web content optimization (WCO)

First public appearance: January 18, 2011

This is the term we here at Strangeloop have settled on (for now, anyway) as the clearest, most specific definition of what we do. We coined it with Akamai when we announced our partnership at the beginning of the year, and it still feels like a pretty good fit:

Web Content Optimization technology takes HTML that has been optimized for readability, supportability and maintainability and, while retaining these benefits, transforms it to HTML that is optimized for fast page rendering. This involves implementing numerous best practices such as rewriting object names, re-ordering when and how objects are rendered, re-ordering when scripts are executed, and optimizing content based on the requesting browser.

It may not be a perfect term — we don’t accelerate content such as video or Flash, for example — but it’s the best we’ve found so far. (Alternately, we’ve also bandied about the term “web code optimization” to see how it feels, but it strikes us as perhaps too limiting.)

Just looking back at the time frame over which these terms have emerged makes it clear that our industry is moving fast, and our terminology is still in flux. (Though some of us have probably used our pet terms so much in our writing that we’re pretty attached to it for SEO purposes. :) ) But at the end of the day, we’ll serve ourselves and our customers better if we can give the clearest definition possible for what we do.

Related posts:

By: Tammy Everts Category: Tags: , , , , , ,
1 2 3 4 5 6 7