Mobile Web Performance: Desperately seeking data

At some point you’ve probably asked yourself some variation of this question:

If my site takes X seconds to load now, and generates Y dollars, will speeding it up by Z seconds lead to more dollars? And if so, how many dollars?

Substitute “conversions” or “downloads” for “dollars” – whatever word you choose, that’s what it boils down to, right? It’s a fundamental question. The mainstream web performance community is rapidly amassing reams of data in this area. But when it comes to the mobile web, we’re pretty much at square one.

These days, you can’t scan the tech news without reading headline after headline trumpeting the growth of the mobile market. Anyone can tell you that there are 75 million American adults using the mobile web and that by 2013 that number is projected to hit 134 million. And you’ve probably already read the study stating that more than half of mobile device users expect sites to download as quickly on their mobile devices as they do on their home computers.

But you know what kinds of numbers are hard to come by? Real data about mobile web performance. If you look at the Analytics section of Mobile Commerce Daily, you’ll see all of three articles. None of these articles touch on mobile performance and key business metrics such as conversion rates, downloads and revenues.

What are the challenges in gathering mobile performance data?

The same challenges that faced those of us in the web performance industry a few years ago:

  • Lack of tools for measuring performance. We need tools that are the mobile equivalent of Webpagetest that can give us data about how individual mobile sites perform in the real world.
  • Need for large-scale A/B testing by major mobile sites. Interest in web performance began to soar after online monoliths like Microsoft, Amazon and Shopzilla conducted hardcore long-term studies of how performance changes affected user behavior and business metrics.
  • Lack of information sharing. Further to my point above, I’m sure that large companies are already analyzing their mobile performance, but they’re keeping these numbers close to their vest. At some point down the road – Velocity 2011, I hope? – this information will start to trickle down to the general public. This will be what drives companies to finally focus on improving the speed of their mobile sites.

Wouldn’t it be great to have charts and graphs like these?

Business impact of improved performance on order size and conversion rate

Conversion rate falloff by landing page slowdown

This is all the information I’ve ever managed to corral about how improving a mobile website’s performance may have affected conversion and cart size. Please note these words: May have. This is data gathered over the course of eight months of A/B testing on Strangeloop client AutoAnything‘s mobile site. It appeared that accelerating the site by 40% caused the average order size to increase by 3% and conversions to increase by 5%. As the second graph shows, we also saw that slowing down the mobile site caused a conversion decrease that paralleled the decrease on the regular site.

Again, I want to emphasize that this is very preliminary stuff (read: I don’t want to read these numbers being touted as universal truths on some other blog a month from now), but it goes to show what kind of numbers we should be trying to gather and share. The information is out there. We just need to get our mitts on it. It took our community almost ten years to generate meaningful data around regular web performance. We don’t have that luxury with the mobile internet.

Do you have any numbers or methodologies to share? Ideas to suggest? Get in touch with me by email (joshua /at/ webperformancetoday /dot/ com) or in the comments.

Related posts: