Latency reality check: Early findings show that desktop latency ranges from 65-145ms

Last year, we here at Strangeloop conducted an in-house research project in which we determined that 97% of mobile end-user response time happens at the front end. We did this by digging into our customers’ beacon data and analyzing millions of unique transactions.

At the time, I mentioned that I planned to conduct a similar project to identify typical latency for mobile users versus desktop users. We recently completed our research, and the results were surprising. While we expected mobile latency to be high (last fall, some ad hoc research led to the realization that latency can hit 350ms on 3G), we also expected latency for desktop users to be relatively low — and this was not the case.

(If you’re a newcomer to the web performance scene, here’s a short primer describing what latency is, why it’s a problem, and a few common solutions.)

Methodology

  1. We took latency measurements for three US-based Strangeloop customers for discrete periods of time between February 15, 2012 and March 15, 2012. Each site uses a different content delivery network (CDN). Latency measurements were performed once per visit. We measured a total of three million unique visits.
  2. While we tested across all mobile and desktop browsers, the results for Internet Explorer are not included, due to some incompatibilities with the latency measurement process. The browsers included in the results are Firefox, Chrome, Safari, and any others that were able to run the test JavaScript.
  3. Desktop and mobile results were each aggregated and graphed for each site.

Results

Check out the histograms depicting the latency spread for each site. Medians are highlighted.

Site #1: Desktop browsers

Desktop latency: site #1

Site #1: Mobile browsers

Mobile latency: site #1

Site #2: Desktop browsers

Desktop latency: site #2

Site #2: Mobile browsers

Mobile latency: site #2

Site #3: Desktop browsers

Desktop latency: site #3

Site #3: Mobile browsers

Mobile latency: site #3

Observations

1. Desktop latency was worse than expected.

Median desktop latency ranged from 65ms to 145ms. My experience is that people tend to lowball their latency predictions for their own sites, estimating numbers in the 20-30ms range. If you do this, these numbers should make you think again.

I can’t stress this often enough: If you rely on your backbone tests to understand your site’s performance, you’re not getting an accurate view of your site’s true latency issues. Even 20-30ms is on the incredibly optimistic end of the real-world spectrum. Numbers like 76ms and 138ms are a lot more realistic. If you’re being told that your site experiences numbers below 20ms, you’re getting bad information.

2. Median mobile latency was between 26% to 96% poorer than desktop.

There’s been a spate of reports lately citing the fact that mobile users are using devices over their wifi connections at home. As a result, I’ve heard more than one web dev/design person say that this takes some of the heat off of optimizing for mobile — a dangerous opinion, in my mind. Median latency of 190ms is a serious performance hit.

3. The CDN you pick really matters.

Further to the above, the CDN you choose makes a big difference. Each of the sites we measured is on a different CDN, and as you can see, desktop latency was almost doubled from one site to another.

4. Outliers for mobile had double the latency of desktop outliers.

Check out the long tail of outliers for mobile. Individually, outliers aren’t hugely significant, but when you see a tail this long, it represents a large number of people who are experiencing painful latency – in this case up to 990ms. That’s almost a full second for EVERY page resource. Even if you’re maxing out the browser’s ability to make concurrent connections, those nigh-seconds add up fast.

5. Mobile traffic accounted for about 10% of total traffic.

As an interesting aside, we found this number held across all three sites. Note that these were mobile users who were visiting the full site, not an m.site.

Conclusions

While the sample size of three sites isn’t large enough to draw a sweeping, industry-wide conclusion, the number of visits measured — three million in total — is still significant enough to draw some meaningful conclusions.

Latency is hard to get a bead on. It’s all too easy to ignore your site’s latency issues if you assume that your site is at the fastest end of the latency spectrum, or if you assume your CDN is a catch-all solution, or if you rely on backbone tests for your performance measurement. If you’re in the habit of doing any of these things, these findings should give you pause and encourage you to perform this kind of testing on your own pages.

As a further area of study, I’m interested in looking at the wifi versus 3/4G latency numbers, as well as the latency breakdown by device. If you have any suggestions for ways to cut and slice this data, let me know.

If you have any questions about how you can replicate our latency measurement tests on your own site, ask away. I’ll be happy to help.

Related links:

12 thoughts on “Latency reality check: Early findings show that desktop latency ranges from 65-145ms

  1. Pingback: Latency 101: What is latency and why is it such a big deal?

  2. Joshua can you share what JavaScript technique did you rely on to measure latency?

  3. AWESOME data! I can’t tell you how often I get told that the default 50ms latency that WebPagetest uses is way too high so getting real-world measurements is great.

    If you’re willing to share, the testing methodology would be awesome.

    Breaking down mobile across WiFi and Cellular was my top ask but if you could also slice by country or region that might be helpful. Same goes for time of day/day of week (daytime weekday traffic has a large bias towards offices with good connectivity and evening/weekend skews more residential).

  4. I second George’s comment. Does “latency” include DNS/TCP time? Server response time? Or is it just round trip time? Please do share how you measured latency so we can better interpret the data.

  5. Good questions – though I wouldn’t expect anything else. :)

    To test the latency, we pretty much used the standard technique. There’s JavaScript on the page as part of our normal analytics package. Post onload, it starts a timer, fetches a 1pixel image, and stops the timer. We got creative with the image in a way where it never comes out of the browser cache, but will exist and be cached on the CDN, if we chose to test the CDN domain.

    The image itself is on the same domain as the page, or its resources/assets. It’s not somewhere else, on some test domain that we set up. As a matter of fact, it’s sort of integrated with the normal optimization that we do, to make sure it’s written to the same domains that we write the page assets to; we specifically wanted to check the domains and CDNs of the customers. The goal was to never incur DNS lookup or TCP setup time in the latency check. By the time the browser gets to fetching the image, it would have connections opened to the domain already, and it just uses one of those. So, the technique was designed and implemented with the goal of only fetching the image, with no need for DNS lookup or connection setup.

    To answer your question, Steve, we didn’t want to publicly call out any companies, but ping me directly if you’d like this info.

    Hope that answers the rest of your questions. Let me know if you want more clarification.

  6. Pingback: Low-latency networks aren’t just for Wall Street anymore — Broadband News and Analysis

  7. Pingback: Community News: Focus on Latency and Scalability | New Relic blog

  8. Pingback: Bad news for site owners and mobile users: The average web page is now 1 MB

  9. Pingback: Quel est le débit moyen en France | BrainCracking - HTML5, Performances Web et les technos du Web, par Jean-pierre VINCENT

  10. Pingback: Case study: How effective are CDNs for mobile visitors?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>