2 Apr 2012
Last year, we here at Strangeloop conducted an in-house research project in which we determined that 97% of mobile end-user response time happens at the front end. We did this by digging into our customers’ beacon data and analyzing millions of unique transactions.
At the time, I mentioned that I planned to conduct a similar project to identify typical latency for mobile users versus desktop users. We recently completed our research, and the results were surprising. While we expected mobile latency to be high (last fall, some ad hoc research led to the realization that latency can hit 350ms on 3G), we also expected latency for desktop users to be relatively low — and this was not the case.
(If you’re a newcomer to the web performance scene, here’s a short primer describing what latency is, why it’s a problem, and a few common solutions.)
- We took latency measurements for three US-based Strangeloop customers for discrete periods of time between February 15, 2012 and March 15, 2012. Each site uses a different content delivery network (CDN). Latency measurements were performed once per visit. We measured a total of three million unique visits.
- Desktop and mobile results were each aggregated and graphed for each site.
Check out the histograms depicting the latency spread for each site. Medians are highlighted.
Site #1: Desktop browsers
Site #1: Mobile browsers
Site #2: Desktop browsers
Site #2: Mobile browsers
Site #3: Desktop browsers
Site #3: Mobile browsers
1. Desktop latency was worse than expected.
Median desktop latency ranged from 65ms to 145ms. My experience is that people tend to lowball their latency predictions for their own sites, estimating numbers in the 20-30ms range. If you do this, these numbers should make you think again.
I can’t stress this often enough: If you rely on your backbone tests to understand your site’s performance, you’re not getting an accurate view of your site’s true latency issues. Even 20-30ms is on the incredibly optimistic end of the real-world spectrum. Numbers like 76ms and 138ms are a lot more realistic. If you’re being told that your site experiences numbers below 20ms, you’re getting bad information.
2. Median mobile latency was between 26% to 96% poorer than desktop.
There’s been a spate of reports lately citing the fact that mobile users are using devices over their wifi connections at home. As a result, I’ve heard more than one web dev/design person say that this takes some of the heat off of optimizing for mobile — a dangerous opinion, in my mind. Median latency of 190ms is a serious performance hit.
3. The CDN you pick really matters.
Further to the above, the CDN you choose makes a big difference. Each of the sites we measured is on a different CDN, and as you can see, desktop latency was almost doubled from one site to another.
4. Outliers for mobile had double the latency of desktop outliers.
Check out the long tail of outliers for mobile. Individually, outliers aren’t hugely significant, but when you see a tail this long, it represents a large number of people who are experiencing painful latency – in this case up to 990ms. That’s almost a full second for EVERY page resource. Even if you’re maxing out the browser’s ability to make concurrent connections, those nigh-seconds add up fast.
5. Mobile traffic accounted for about 10% of total traffic.
As an interesting aside, we found this number held across all three sites. Note that these were mobile users who were visiting the full site, not an m.site.
While the sample size of three sites isn’t large enough to draw a sweeping, industry-wide conclusion, the number of visits measured — three million in total — is still significant enough to draw some meaningful conclusions.
Latency is hard to get a bead on. It’s all too easy to ignore your site’s latency issues if you assume that your site is at the fastest end of the latency spectrum, or if you assume your CDN is a catch-all solution, or if you rely on backbone tests for your performance measurement. If you’re in the habit of doing any of these things, these findings should give you pause and encourage you to perform this kind of testing on your own pages.
As a further area of study, I’m interested in looking at the wifi versus 3/4G latency numbers, as well as the latency breakdown by device. If you have any suggestions for ways to cut and slice this data, let me know.
If you have any questions about how you can replicate our latency measurement tests on your own site, ask away. I’ll be happy to help.