Interesting findings: 3G mobile performance is up to 10X slower than throttled broadband service

• Tweet

This post was inspired by two slide decks from performance consultant Stephen Thair. (Velocity plug: If you’re going to Velocity EU, be sure to drop in on Stephen’s session. He always has great insights to share.)

The first deck is Stephen’s Measuring Mobile Performance presentation, and the second is his Practical Web Performance presentation. In these decks, he talks about the difficulty of trying to measure real-world mobile performance, and shares his experiments in hacking his own test approach… which basically involves roaming around and conducting random tests in various locations, as shown on these slides:

Inspired by these slides, I set out to map the 3G performance (in my typical unscientifically scientific way) in my own haunting grounds and see what I could deduce from the findings.

Methodology

1. I recorded the data over the course of a week in the following locations:

Legend:
Red plot is the dog park where I spend a lot of my time.
Green dots are my house. (Please don’t stalk me.)
Red dots are my office.
Blue dots are measurements taken outside of my house and office.


2. Every random time I thought about it, I opened up my iPhone and clicked on the Speed Test app.



3. I took 10 measures each time.
4. I recorded the location and the data for each set of measurements. (This was painstaking.)
5. I made some graphs.

Key findings

1. The phone took two or three tests to warm up.

It took a few tests to get to full power. This wasn’t surprising. (See Steve Souders’s post about Rajiv Vijayakumar’s research into smartphone power consumption while loading a web page.)

2. Variability was huge, even in the same location at around the same time.

This amazed me. I really wasn’t expecting it. As you can see from this chart, even at home I couldn’t arrive at any kind of consistent latency. This is probably due to a number of variables: the weather, how busy the tower is from moment to moment, and what direction I was facing. It just goes to show how little control you have over your network connection.

3. The mobile web is painfully slow.

No surprise here. 3 Mbps is absolutely brutal. Even the worst throttling by a broadband internet provider takes you down to 10 Mbps, more than three times faster than the fastest 3G performance I recorded. And upload time is ten times worse than download.

4. My dog park has really slow speed.

Not relevant to you, maybe, but I spend a lot of time there.

Thoughts

No groundbreaking revelations here. The mobile performance challenge circles back to the same two things we’ve been talking about for years: payload and round trips.

Even if you use a content delivery network to accelerate your site for desktop users, your content isn’t next to your mobile user, so you can’t rely on your CDN to bail you out. If you’re on 3G, experiencing download times of 1-3 Mbps, sites with massive pages are going to be painful. And if you’re experiencing latency of up to 350ms on a page with 50 resources, that’s about 17 seconds of load time eaten up by latency alone. That’s beyond painful — it’s excruciating.

Working on this post has really highlighted the challenges of building a strong mobile measurement tool. I’m working on a new post  that rounds up the tools that are currently available, so if you know of any, let me know which ones you’ve discovered and how well they’ve worked for you.

Related posts:

• “Phone rage”: How people react to slow mobile sites
• Announcing the Strangeloop Mobile Site Optimizer
• How to DIY your own mobile graphs