Case study: The impact of HTML delay on mobile business metrics

When I gave my talk about mobile performance and business KPIs at Velocity Berlin a couple weeks back, one of the areas I got the most questions about later was the experiments we were able to do in which we delayed HTML on a customer’s site and tracked the results over a 12-week period. I thought it might be useful to break some of this out into its own post.

As I mentioned at Velocity, I was insanely jealous of Google and Bing a couple years ago, when they revealed their own in-house experiments with HTML delay. Most of us in the performance community would kill for that kind of experimentation platform. So I was extremely happy and grateful when one of our customers at Strangeloop expressed an interest in figuring out the value of time for their business.

Methodology

We conducted a split test over the course of 12 weeks, in which we segmented mobile traffic into four groups: fully optimized, 200 ms delay, 500 ms delay, and 1000 ms delay. We monitored four metrics: bounce rate, conversion rate, cart size, and page views. We also monitored and analyzed user behavior for 6 weeks after the test ended, to gauge the long-term impact, if any, of slow performance even after users begin to receive an accelerated site.

Results

The results of the 200 ms delay weren’t significant, but the customer and I were both taken by the dramatic impact of the 500 ms and 1000 ms slowdown. Our customer was blown away that they were losing 3.5% percent of their conversions when their site was delayed by just one second on a mobile device. This was a major epiphany for them, and it’s already helped them change their business and how they view mobile.

I’m including this next graph to reinforce the connection between load time and user behavior.

Over the 12 weeks, you can see that, while the HTML delays were constant — 500 ms and 1000 ms — users’ reaction to these delays fluctuated. The drop in bounce rate ranged from around -2% and -12% for users who experienced a 1000 ms delay, and 0% and -6% for those who experienced a 500 ms delay. While the bounce rate may have varied, the one thing that was constant was the fact that the behavior trends are strongly linear for both groups, and the bounce rate for the 1000 ms group was consistently worse.

Finally, and most interestingly to me, we wanted to look beyond just the effect of delay in the timeframe of the experiment. We know that slower pages have an immediate impact on user behavior and customer satisfaction. We wanted to find out if there was any long-term impact on customer satisfaction.

So we looked at our traffic data for the 6 weeks immediately after the experiment — specifically at the behavior of returning visitors. As any site owner will tell you, repeat customers are the bread and butter of an e-commerce vendor. These are the people you need to keep happy. If you look at the graph below, you can see that, even after the experiment was over and the shoppers in the 500 ms and 1000 ms group started to be served the same accelerated site as the baseline group, they were significantly less likely to return to the site. By the end of the 6-week period, you can see that return traffic is slowly improving as visitors seem to finally be recovering from their poor experience.

Conclusions

As I’ve already mentioned, these findings have been a huge revelation to the company that owns this site. It’s had a major impact on how they’re tackling performance on their mobile platform. The most important over-arching takeaways from this experiment were:

  • Mobile shoppers are now fully engaging with e-commerce sites in significant enough numbers that we can analyze their behavior as a group.
  • Even a 500 ms delay has a major impact on metrics. For mobile sites, which can suffer excruciating load times (the latest Keynote index is about 12 seconds), this is a wake-up call that they need to take a hard-line approach to optimizing their pages.
  • The damage of poor performance is lasting.

See the full presentation: Case Studies from the Mobile Frontier: The Relationship Between Faster Mobile Sites and Business KPIs

Related posts:

21 thoughts on “Case study: The impact of HTML delay on mobile business metrics

  1. Pingback: Case study: How to use network quality as a proxy for measuring mobile performance

  2. Pingback: Colonoscopies, cold water and pain: How our memory works and how this relates to web performance

  3. Pingback: Interesting new findings about page views, time on site, and bounce rate across browsers and platforms

  4. Pingback: Speed does matter!

  5. Pingback: Fixing long-distance latency comes with a $4.5 billion price tag

  6. Pingback: Savings, Checking, Pay Bills – Wait | Application Performance Engineering Blog - Shunra Software

  7. Pingback: Four important web performance lessons for SMBs

  8. Pingback: Performance Calendar » A/B testing and front-end optimization: Not as easy as A-B-C

  9. Pingback: Make the mobile web better by NOT making these 4 responsive design mistakes | Latestwire

  10. Pingback: The Capitals™ – Capitalists' Magazine | 資本家札記 | Make the mobile web better: Don’t make these 4 responsive-design mistakes

  11. Pingback: RWD Summit 2013 Presentations - Adapdevelopment - Adapdevelopment

  12. Pingback: The Two Flavors of a ‘One Web’ Approach: Responsive vs. Adaptive | JVZ News

  13. Pingback: The Two Flavors of a ‘One Web’ Approach: Responsive vs. Adaptive - BMAgads.com

  14. Pingback: Blue Note Tech Blog » The Two Flavors of a ‘One Web’ Approach: Responsive vs. Adaptive

  15. Pingback: The Two Flavors of a ‘One Web’ Approach: Responsive vs. Adaptive | Web Design Florida Organization

  16. Pingback: The Two Flavors of a ‘One Web’ Approach: Responsive vs. Adaptive - 7u8.net

  17. Pingback: The Two Flavors of a ‘One Web’ Approach: Responsive vs. Adaptive - Breaking News, Latest News and Current News from 7u8.net. Breaking news and video. Latest Current News: U.S., World, Entertainment, Health

  18. Pingback: Web Design India, Website Designing India, Outsource Web Designing to India, Website Designing Company in India

  19. Pingback: The Two Flavors of a ‘One Web’ Approach: Responsive vs. Adaptive | Reme Le Hane - Front-end Web Developer

  20. Pingback: The Two Flavors of a ‘One Web’ Approach: Responsive vs. Adaptive - Breaking News, Latest News and Current News from 7u8.net. Breaking news and video. Latest Current News: U.S., World, Entertainment, Health

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>