Research

What we learned from testing the mobile load times of 200 top retail sites over 3G and LTE

3 Oct 2012

6 Comments

Earlier today, I had an exciting opportunity to present some of the findings of Strangeloop’s newest study, the 2012 State of Mobile Ecommerce Performance at Velocity EU. I’ll be posting those slides shortly, but first want to take a moment to talk about this study and why we’re so excited about it.

Why test over cellular networks?

Last spring, we were sitting in the office talking about mobile performance measurement and why it’s so hard to get reliable real-world numbers, especially over cellular networks. This seemed like a glaring knowledge gap, but it’s an understandable gap given the massive variability in network performance. Last year, I did an informal latency survey and found that latency can fluctuate wildly, even in the same location at the same time.

3G latency

Methodology: How to DIY your own RUM tests for mobile

So we set out to develop a reliable methodology for measuring mobile performance over cellular networks, which takes this variability into account. We also wanted to create a methodology that anyone could recreate fairly easily.

Here’s how we did it:

  • We focused on 200 leading retail sites, as ranked by Alexa.com.
  • We selected six testing devices: iPhone 4, iPhone 5, Samsung Galaxy S smartphone, Samsung Galaxy S3 smartphone, iPad 2, and Samsung Galaxy tablet.
  • iPhone 4, Galaxy S, iPad 2 and Galaxy tablet were tested over 3G.
  • iPhone 5 and Galaxy S3 were tested over LTE.
  • We tested each site’s home page 30 times per device, using the device’s native browser. By testing this many times and capturing the median result, we hoped to take into account latency variability.
  • For all tests, the devices were positioned in the same location, in an attempt to mitigate the latency impact caused by location changes.
  • For all tests, wifi was turned off, devices and radios were at full power, and screens were not allowed to lock during testing.
  • The tests in this study were conducted using a RUM beacon inserted into the page that captured the onload time. The cache and cookies were cleared automatically between each test.

The results

As I shared at Velocity, the results were a mix of the predictable and the surprising:

Predictable: Pages are slow over 3G.

It’s not that shocking to learn that the median pages took 11 or more seconds to load over 3G.

Surprising: A significant number of pages took 20+ seconds to load.

While we expected pages to be slow, we didn’t expect that 9% of the pages we tested took more than 20 seconds to load on the iPhone.

Predictable: LTE was faster than 3G.

No explanation needed here.

Surprising: LTE was only 27% faster than 3G.

I’ve heard claims that LTE is, on average, ten times faster than 3G. Our results suggest that LTE performance gains might be more unpredictable than this.

Predictable: Pages loaded 22% faster on the iPad than on the Galaxy tablet.

iPad fans might scoff that we even bothered to test this.

Surprising: The Galaxy S3 phone loaded pages 9% faster than the iPhone 5.

It’ll be interesting to hear what iPhone fans make of this. (Full disclosure: I have an iPhone 5 and have to admit I was kind of affronted to see these results.)

Check out our infographics of the report’s highlights, below, and I encourage you to read the findings for yourself. If you have any questions about our findings or approach, don’t hesitate to ask.

Load times of 200 leading retail sites on mobile devices over 3G and LTE networks

Related posts:

By: Tammy Everts Category: Tags: , , , , , , ,

Browser innovation and the 14 rules for faster loading websites: Revisiting Steve’s work (part 2)

13 Sep 2012

4 Comments

A few months back, I posted part 1 of a 3-part post to answer the question:

“Aren’t many of the web performance rules described by Steve Souders in 2007 already outdated or made obsolete by browser innovation?”

Despite the fact that my company was founded on the premise that Steve’s rules are our bible, I still place a great deal of value on testing core assumptions. Fortunately, testing the first five rules revealed that these performance rules have stood the test of time and are as valid today as they ever were.

Today (finally), I want to look at the next five rules:

Methodology

1. As with my previous post, I used these test cases of the performance rules that Steve created five years ago, and which are still available. Using the Wayback Machine, I timestamped them to October 2007.

Steve Souders: 14 rules for high performing web sites2. Next, I ran each test case on Internet Explorer 6 on WebPagetest to get a sense of what Steve would have seen for performance rules 6 through 10. I did three runs of each test and used the median result.

3. Then I ran each test case on Chrome 19 to see what impact, if any, the rules have on a modern browser. Again, I performed three runs of each test and recorded the median result.

4. Using the above approach, I was able to see and compare the before-and-after results for each rule for both browsers, and then calculate the benefit. You can see the detailed test results, along with links to the tests, in this spreadsheet.

Important: The goal here is not to look at how fast Chrome 19 is versus Internet Explorer 6. We have looked at this in the past. This time I want to look at the relative benefit of each performance rule.

Findings

Here’s a high-level look at the results:

Rule 6: Move scripts to the bottom of the page

What this rule means: It’s better to move scripts from the top to as low in the page as possible. One reason is to enable progressive rendering, but another is to achieve greater download parallelization.

With scripts, progressive rendering is blocked for all content below the script. Moving scripts as low in the page as possible means there’s more content above the script that is rendered sooner. The second problem caused by scripts is blocking parallel downloads. While a script is downloading, the browser won’t start any other downloads, even on different hostnames. [From High Performance Web Sites]

Testing the rule: Steve experimented with placing scripts in various places on a page in order to see their performance impact. Using the methodology above, I was able to see the performance benefit (defined by the improvement in document complete time) of applying two of these techniques to IE6 and Chrome 19:

Rule
 Benefit in IE6 Benefit in Chrome 19
Scripts at the bottom 0.13% 18%
Deferred scripts
 31% 18%

Rule 7: Avoid CSS expressions

What this rule means: CSS expressions are an older technique, supported in Internet Explorer up until the release of IE8, which let you set CSS properties dynamically. The problem with CSS expressions was that they were evaluated more often than most people realized. As Steve pointed out in 2007, moving the mouse around the page can easily generate more than 10,000 evaluations.

This technique has fallen out of practice, largely because of findings like Steve’s. Nobody with any common sense is using this technique anymore, so it’s not relevant.

There may, however, be a modern version of this same issue. I’ve heard reports abut similar issues with JavaScript performance due to easy-to-use JS libraries such as JQuery. Similar to CSS expressions, it’s very easy to hook up functions to JavaScript events like moving the mouse and scrolling, causing similar tens of thousands of calculations to happen. I’m not sure how often this is actually happening out in the real world, but it’s something to be aware of.

Testing the rule: N/A

Rule 8: Make JavaScript and CSS external

What this rule means: If users on your site have multiple page views per session and many of your pages re-use the same scripts and stylesheets, you could potentially benefit from cached external files.

Pages that have few (perhaps only one) page view per session may find that inlining JavaScript and CSS results in faster end-user response times.

For front pages that are typically the first of many page views, there are techniques that leverage the reduction of HTTP requests that inlining provides, as well as the caching benefits achieved through using external files. One such technique is to inline JavaScript and CSS in the front page, but dynamically download the external files after the page has finished loading. Subsequent pages would reference the external files that should already be in the browser’s cache. [From High Performance Web Sites]

Testing the rule: Below are the performance gains (defined as the improvement in document complete time in the repeat view) for both browsers:

Rule
 Benefit in IE6 Benefit in Chrome 19
Cacheable external JS and CSS 37% 40%
Post-onload download 42% 45%
Dynamic inlining 38% 44%

Rule 9: Reduce DNS lookups

What this rule means: There’s no test case for this rule — because it’s a given that fewer DNS connections is a good thing — but I still want to review it here in order to talk about why it’s a good thing.

DNS lookup is when the browser looks up the domain of the object being requested by the browser. Think of this as asking the “phone book” of the internet to find someone’s phone number using their first and last name. DNS lookups are cached for better performance — on caching servers maintained by your ISP or LAN, on your operating system’s DNS cache, and in your browser cache.

Steve offers this advice: Reducing the number of unique hostnames has the potential to reduce the amount of parallel downloading that takes place in the page. Avoiding DNS lookups cuts response times, but reducing parallel downloads may increase response times. My guideline is to split these components across at least two but no more than four hostnames. This results in a good compromise between reducing DNS lookups and allowing a high degree of parallel downloads.

It’s important to know that some newer browsers (at least Chrome does) do DNS prefetching as well.

Testing the rule: N/A

Rule 10: Minify JavaScript

What this rule means: Minification is the practice of removing unnecessary characters — such as comments and white space — from code. This improves JavaScript response time because the size of the downloaded file is reduced. Minification is fairly straightforward, unlike its sister practice: obfuscation.

Like minification, obfuscation removes comments and white space, but it also munges the code — converting function and variable names into smaller strings, thereby making the code more compact as well as harder to read. While munging wasn’t originally conceived of as a performance practice, it was found that it can help performance because it reduces the code size beyond what is achieved by minification. (Warning: Obfuscation can lead to bugs, unlike minification, which is relatively problem free.)

In addition to minifying external scripts, you can also minify inlined script blocks. Even if you’re already gzipping your scripts, minifying them will still reduce the size by at least 5%.

Testing the rule: Below are the performance gains for both minification and obfuscation of different-sized files:

Rule
 Benefit in IE6 Benefit in Chrome 19
Small script minified 14% 16%
Small script obfuscated 14% 15%
Large script minified 22% 21%
Large script obfuscated 23% 21%

Conclusion

Once again, it’s reassuring to see how many of the performance rules have stood the test of time. I was really struck by how similar the results were for many of the test cases. I had expected that the rules would still be relevant, but to be perfectly honest, I had also expected that browser evolution would have resulted in more disparity between the results.

It was also interesting to see how a rule — avoid CSS expressions — has become so entrenched over time that it’s no longer an issue.

This has been a really interesting exercise so far. I’m looking forward to part 3.

Related posts:

By: Tammy Everts Category: Tags: , , , , ,

The 33 best web performance links of Q2 2012

31 Jul 2012

4 Comments

For some reason, I thought that the past few months had been kind of quiet on the research front, so when I started this post, I thought it would be one of my shortest roundups yet. I was pleasantly surprised to watch it grow to become one of the longest!

There are some great case studies here, of both large and small sites, which I love to see. There’s also some truly excellent debate about responsive design and the mobile web, sparked by a post from Jakob Nielsen last spring, as well as some good stuff about the browser wars and third-party content. So enough with the intro. Let’s get into it.

Case studies

Optimizing Retr-O-Mat’s Web Performance

A casual performance optimizer details her efforts to get Retr-O-Mat’s average load times under 2 seconds. Good information for anyone interested in the nuts and bolts of front-end optimization (FEO).

Web performance can be beautiful too

After performing poorly in a 2011 web performance comparison of leading retailers, Crate and Barrel made WPO a priority moving forward. This blog post from Catchpoint shows just how “beautiful” their performance has been in 2012.

How the Post is improving site performance

Responding to a flood of user frustrations with their website, the IT team at the Washington Post rolled out a number of performance upgrades to their site over the past year. Find out what they did to improve their page speed by 32.4%.

Tips and how-tos

Building a faster web: Tools, tips, and lessons

If “faster connectivity and more bandwidth won’t save us,” then what will? Google’s Ilya Grigorik shares his insight on making the web faster in this in-depth slide deck, and he draws some very interesting conclusions.

How to Make Progress Bars Feel Faster to Users

The human perception of time is anything but linear, and with just minor visual tricks, it gets even more skewed. After reading this post, you may never trust a progress bar again. :)

The 3 white lies behind Instagram’s lightning speed

More cool perceptual tricks. The “secret sauce” behind Instagram’s stellar user experience is rooted in a combination of coding tricks aimed at giving users a feeling of constant responsiveness. Find out how their site “always pretends to work.”

Mobile

The web only works thanks to reload (and why the mobile web fails)

As Mike Belshe points out, web page resources routinely fail, but thanks to the ever-handy reload/refresh button, we can often solve these problems ourselves. With mobile browsing, however, the rules are different. Find out what this means for the future of HTML 5.

Web first for mobile

Performance evangelist Steve Souders focuses his performance research strictly on the mobile web – not on native apps. Why? He’s got more than a few good reasons.

A taste test of mobile website development

A solid webcast on the complex world of mobile development, touching on topics including Responsive Web Design (RWD), server-side device detection, and HTML5 performance on mobile.

Jakob Nielsen on mobile sites vs. full sites

Jakob Nielsen believes that mobile and full sites should be entirely different entities. Summarizing his argument, he states that “good mobile user experience requires a different design than what’s needed to satisfy desktop users. Two designs, two sites, and cross-linking to make it all work.”

Is Nielsen wrong on mobile? Arguments abound

From Net magazine: Jakob Nielsen’s assertion that “good mobile user experience requires a different design” is being challenged by a noted mobile expert, who argues that rather than stripping down for mobile, companies should be doing more.

Why we shouldn’t make separate mobile websites

More counterpoint to Nielsen’s post. Smashing Magazine’s Bruce Lawson argues that mobile redirection is unreliable, and excluding features for mobile browsers “perpetuates the digital divide.”

Responsive web design: Missing the point

Still more Nielsen backlash: Brad Frost states that, though mobile browsers are getting better at rendering full websites, creating adaptive sites for mobile users is essential to improving the user experience.

HTML5 features increase mobile usage by 28%

Interesting piece explaining how static pages needing an upgrade can vastly improve mobile user engagement through the addition of HTML5. The new release features interactive galleries, overlays, and expandable/collapsible boxes, driving up pageviews and decreasing bounce rates.

Tools

More ways to measure web performance with User Timings

Google Analytics has expanded its collection of Site Speed reports with a new feature called User Timings. The feature enables tracking of specific load times for discrete hits, images, and other user interactions.

New mod_spdy release supports Apache servers

More from Google. The latest version of mod_spdy – an Apache module that adds SPDY server support – is intended to fix bugs found in the original release.

“Speed Index” introduced as new performance metric

The Speed Index metric has been added to WebPagetest, helping measure the speed at which page contents are visually populated. The tool is especially useful for comparing page experience before and after optimization.

Browsers

Browser Speed Tests: Chrome 19, Firefox 13, Internet Explorer 9, and Opera 12

Lifehacker conducts it’s semi-regular browser speed tests, pitting the four titans of desktop browsing against each other in races for startup speed, tab loading times, and other performance indicators.

Which Browsers are the Fastest?

An interesting companion read to the Lifehacker piece, New Relic’s “Speed Wars” study shows that, while IE 9 speeds past other browsers on Windows, Chrome 13 on Mac was overall the fastest experience. In mobile speed tests, the fastest experience was delivered by Blackberry Opera Mini at 2.6 seconds, twice as fast as Safari 5.1 on iPad.

How the Chrome Predictor hides latency from users

Ilya Grigorik demonstrates how Google Chrome hides latency from users. Interesting stuff here.

Internet Explorer market share surges, as IE 9 wins hearts and minds

From Ars Tecnica: “The browser wars are back on in earnest. For the second time in three months, Internet Explorer made large gains, picking up almost 1 point of market share. Chrome, Firefox, and Safari all lost out, as Internet Explorer 9 won over new users.”

CDNs

A one-size-fits-all CDN solution isn’t always best

Server configurations come in all shapes and sizes, which means a one-size-fits-all CDN is seldom effective. Find out which Level 3 customer was the beneficiary of a custom CDN solution and how it worked out.

Third-party content

10 Golden Rules for 3rd Party Providers

Murphy’s Law reigned supreme throughout June, with a flood of large-scale outages taking down some of the world’s most popular websites. Given the inevitability of online failures, third-party providers must be prepared to deal with the worst. The folks at Catchpoint outline the 10 Golden Rules by which all third-party providers should live by.

The vendor who flunked the web performance test

Are third-party vendors ignorant to the consequences of slow web performance? According to Catchpoint they are, as they detail a story of one such vendor who was completely unaware of the performance impact of their product.

Average UK website has 14 trackers per page

Interesting findings from TRUSTe: Despite the prevalence of privacy policies, over two-thirds of trackers on UK websites originate from third-party companies, and almost half embed themselves permanently.

Google releases +1 button preview – loads 20% faster

Google announced that they’ve improved performance of the +1 button and Google+ badge. By reducing the size of the js/plusone.js loader and making the code smarter, page elements now load 20% faster.

Third-party JavaScript should be loaded asynchronously

Old news to some, but still worth mentioning: Third-party JavaScript should be loaded asynchronously, as it helps avoid slowdowns and can speed up page loads.

Third-party front-end performance, Act 1

Application provider Bazaarvoice is delving into the realm of front-end performance, and provides an interesting third-party perspective.

Opinions and analysis

Performance Nightmare: Nasdaq & the Facebook IPO

When Facebook began trading on May 18, 2012, a series of performance failures on Nasdaq.com caused a huge headache for the company. This article from Intechnica asks how much these badly timed hiccups cost investors.

More, better, faster: Steve Souders on WPO

Steve Souders kicked off O’Reilly’s Velocity video podcast series with an in-depth discussion of the state of web performance optimization. Key topics included measuring slowness, performance monitoring tools, and whether mobile disrupts performance.

Other research

How complex systems fail

As a complex and interdependent system, the web is prone to catastrophe at the highest levels. In this fascinating paper on resilience engineering, presented at Velocity 2012, Dr. Richard Cook outlines the reasons why all complex systems are intrinsically hazardous, why disaster is always just around the corner, and how failure-free operations still require experience with failure.

The growing epidemic of page bloat

I don’t usually pimp my own writing here, but this information is too important not to share. I wrote a piece for GigaOM showing that the average page size is now over 1MB, according to the HTTP Archive. At current growth rates, the average page could hit 2MB by 2015, which is a really big deal, especially for mobile users.

How fast are websites around the world?

Some fascinating findings here. Google’s Site Speed Reports provides detailed latency data for page load times by separating data according to device, location, and industry.

These links were all sourced from Strangeloop’s Web Performance Hub, which contains hundreds (and by now, possibly even thousands) of industry-wide links, organized by topic, source, research type, and industry. It’s a pretty good resource, if I do say so. If you have any new links to recommend, let me know.

http://t.co/EHbRdT6r
10 Golden Rules for 3rd Party Providers [article]
Catchpoint – June 26, 2012
Summary: Murphy’s Law reigned supreme throughout June, with a flood of large-scale outages taking down some of the world’s most popular websites. Given the inevitability of online failures, third-party providers must be prepared to deal with the worst. Find out the 10 Golden Rules by which all third-party providers should operate by.
http://t.co/LGYUpGn1
End-to-end optimization: Taking content delivery to the next level [blog post]
Web Performance Today – June 27, 2012
Summary: Strangeloop Networks is thrilled to announce the launch of our latest product, Network Accelerator. Learn all about how this product works, what it does – and most importantly – why it’s a major step forward for content delivery networks.
http://t.co/T4z69s7k
How complex systems fail [research paper]
CTALab.org – June 26, 2012
Summary: As a complex and interdependent system, the web is prone to failure and catastrophe at the highest levels. In this fascinating paper on resilience engineering, Dr. Richard Cook outlines the reasons why all complex systems are intrinsically hazardous, why catastrophe is always just around the corner, and how failure-free operations require experience with failure.
http://bit.ly/LzGPqN
Mobile optimization starts with mindset: Hooman Beheshti interviewed at Velocity 2012
O’Reilly Media – June 25, 2012
Summary: Where are we in the mobile optimization life-cycle? What mindset should site owners adopt when boosting mobile performance? Are performance measurements improving? In this video, Strangeloop Technology VP Hooman Beheshti offers his unique insight on the current state of mobile.
http://t.co/Vnced8tq
The 90-Minute Mobile Optimization Life Cycle [slides]
Strangeloop Networks – June 25, 2012
Summary: Strangeloop Technology VP Hooman Beheshti wowed attendees at this year’s Velocity Conference with a presentation on the mobile optimization life cycle. For those who missed it, be sure to check to check out these fascinating slides.
http://bit.ly/Nu1gCi
Ghosts of Velocities Past: 9 presentations that are still relevant today [blog post]
Web Performance Today – June 20, 2012
Summary: Velocity’s short (yet incredibly important) history is filled with memorable moments, and these 9 presentations from past conferences remain relevant today. Perhaps not trendsetting anymore, but certainly trend affirming, which may just be better.
http://bit.ly/MFuxMR
My recent post on SEOMoz: 13 Questions (and Answers) About Google, Site Speed, and SEO [article]
SEOmoz – June 18, 2012
Summary: In this article, Strangeloop president Joshua Bixby breaks down how site speed and performance metrics affect Google page ranks. For anyone who has ever wondered how Google manages to make performance metrics affect SEO, this article is a must-read.
http://mz.cm/M24fGc
Introducing: New Browser Tax feature for our ecommerce customers [blog post]
Web Performance Today – June 14, 2012
Summary: Ever wish you could arbitrarily tax your customer base for failing to stay current, with zero repercussions? With the new Strangeloop Browser tax, your wish is now a reality!
http://bit.ly/NC5Q65
Optimizing Retr-O-Mat’s Web Performance [blog post]
Finding Marbles – June 9, 2012
Summary: For the “WPO guy’s wife,” average load times just aren’t good enough. In this post, a blogger and casual performance optimizer details her efforts to get Retr-O-Mat’s average load times under 2 seconds. Great information for anyone interested in the nuts and bolts of WPO.
http://t.co/k0lXkniG
Browser Speed Tests: Chrome 19, Firefox 13, Internet Explorer 9, and Opera 12 [article]
Lifehacker – June 12, 2012
Summary: It’s a battle of startup times, tab loading times and other KPIs between the four titans of Windows browsing. Lifehacker’s speed tests are always entertaining for what they’re not afraid to say, and this article is no exception.
http://bit.ly/Nb2ibS
Marrying CDNs with front-end optimization for maximum acceleration [blog post]
Web Performance Today – June 12, 2012
Summary: Front-end optimization (FEO) has been weaving its way further into content delivery networks (CDNs) over the past two years, and the dynamic between these two technologies continues to evolve. In this video presentation, Strangeloop’s Joshua Bixby breaks down the benefits of combining these performance solutions.
http://bit.ly/L3aatz
How the Chrome Predictor hides latency from users [article]
Igvita.com – June 4, 2012
Summary: Google Chrome features countless tools for supercharging load times, but when those aren’t enough, the browser can hide latency from users. Find out how!
http://bit.ly/Nw6ZMh
Building a faster web: tools, tips, and lessons [slides]
Igvita.com – June 3, 2012
Summary: If “faster connectivity and more bandwidth won’t save us,” then what will? Google’s Ilya Grigorik shares his insight on making the web faster in this in-depth slide deck, and draws some very interesting conclusions.
http://bit.ly/L0eERH
The “performance poverty line”: What is it and why does it matter? [blog post]
Web Performance Today – June , 2012
Summary: The “performance poverty line” is the point at which business metrics have sunk so low, load times cease to matter. But how is this line measured? Does it differ between industries? And most importantly: is there hope?
http://bit.ly/NJAqs2
A one-size-fits-all CDN solution isn’t always best [article]
Level 3 – June , 2012
Summary: Server configurations come in all shapes and sizes, which means a one-size-fits-all CDN is seldom effective. Find out which Level 3 customer was the beneficiary of a custom CDN solution.
http://bit.ly/M9P5xt
Why the Facebook outage is (yet another) wakeup call for site owners [blog post]
Web Performance Today – June , 2012
Summary: The hazards of running third-party scripts are well documented, but the May 31st Facebook outage was another stern reminder. In this post, Strangeloop’s Joshua Bixby discusses all things third-party, including rogue content and common performance pitfalls caused by third-party content.
http://bit.ly/JQj7GX
The web only works thanks to reload (and why the mobile web fails) [article]
Belshe.com – June , 2012
Summary: Web page resources routinely fail, but thanks to the ever-handy reload/refresh button, we can often solve these problems ourselves. With mobile browsing, however, the rules are different. Find out what this means for the future of HTML 5.
http://bit.ly/NAAQ3O
How to Make Progress Bars Feel Faster to Users
UXMovement – June , 2012
Summary: The human perception of time is anything but linear, and with just minor visual tricks, it gets even more skewed. After reading this post, you’ll never trust a progress bar again!
http://bit.ly/NcsVfg
Does the average web user waste two days a year waiting for pages to load? [blog post]
Strangeloop Networks – June , 2012
Summary: It may not be true, but in web performance, perception is reality. Web users in the UK are less than pleased about their online experience, but just how cranky are they?
http://bit.ly/LSFIlv

Related posts:

By: Tammy Everts Category: Tags: , , , , , , , , , , , , , , , , ,

Summer reading

18 Jul 2012

0 Comment

I’ll be unplugging for the next ten days while I take my wife and boys on vacation. In the meantime, I looked through my analytics and pulled some of the most popular posts from the archives.

Enjoy. See you in August. :)

By: Tammy Everts Category: Tags:

Early findings: Mobile browser cache persistence and behaviour

12 Jul 2012

1 Comment

If you’ve spent any time in the mobile space, you know that the browser cache is a critical limiting factor for performance. I often talk about:

  • how unreliable the mobile browser cache is,
  • how poorly it persists data, and
  • how poorly it observes basic caching headers.

You may also know that one of the key benefits that front-end optimization (FEO) can bring to the mobile world is a more useful browser cache. Our team, led by Hooman Beheshti and Jarrod Connolly, has been working for years to perfect mobile acceleration, and they’ve spent a huge amount of time digging into the browser cache conundrum. Hooman recently presented some very interesting findings at Velocity, which I want to expand on today.

Background:

Much has been written about how poor mobile caching is. For a thorough overview, read more here, here, here, here, and — my favourite link — here. But for our purposes today, here’s the Reader’s Digest version:

1. Mobile cache sizes are far too small.

The stock browser in Android (2.x) has a cache size limit of around 5.7MB. iOS is much larger (>50MB).

2. When we think about mobile cache sizes, we need to take into account a bunch of variables.

We need to consider the total cache size (which is a shared cache for all our browsing), individual object size limits for caching, and the fact that HTML pages may be treated slightly differently.

But we had more questions about mobile caches, so Hooman set out to get some answers. Below is what he found.

Question 1: How do non-stock browsers cache on Android?

Further to this question, Hooman wanted to see if anything has changed with Android ICS (Ice Cream Sandwich, version 4.0 of Android). His methodology was pretty similar to what others have done to determine mobile cache sizes (see the links above).

Here’s what came out of it:

  • On Gingerbread (Android 2.3.x), Firefox and Opera both had cache sizes that were slightly larger than the stock browser. They both came out at around 10MB.
  • The stock ICS browser had an HTTP cache that was much bigger than its predecessor. Hooman’s tests showed a limit of 25MB.

He was also going to test Chrome on ICS (Chrome is not available for Gingerbread), but then found this great article from Tony Gentilcore from Google that outlines how the Chrome cache is large and persistent. Better yet, the cache size is dynamic and based on the amount of disk space available, which is great news.

To put things into perspective, it’s good to compare mobile cache sizes to those of desktop browsers, which are usually in the range of 100s of MB:

  • This article from IE tells us that in IE6/7/8 the IE cache size was 1/32 of the disk, with a default cap of 50MB. IE9 is 1/256 of disk size (not a bad thing since disks are much bigger these days) and has a default cap of 250MB. Cache size is user configurable in IE.
  • Spot checking our own Chrome and Firefox browsers showed a cache size of >300MB on Chrome and >700MB on Firefox.
  • Steve has a great post on cache sizes with some crowdsourced info on what’s out there now.

Conclusion: Mobile caches are lacking when it comes to how much content they can hold.

This is especially true when you consider that an HTTP object cache is a shared resource for a browser. If you do even minimal browsing on your smartphone, you’re bound to fill up the cache rather quickly. But things are getting better. ICS is a good example of that. And even though progress may not be occurring at the pace we’d like, any progress is good. This also highlights the value of using localStorage as an object cache, which mobile acceleration solutions like Strangeloop’s Mobile Optimizer automatically enable a site to do.

Question 2: How do mobile browser caches behave with various user actions (e.g., closing the browser, locking the phone, shutting down the phone)?

This is an extremely important question that’s never been satisfactorily answered. There is very little data on the persistence of the cache and how (and if) objects are fetched from cache. Here’s a breakdown of Hooman’s suite of experiments in this area.

Methodology

Considering the hundreds of permutations that something like this could end up with, we tried to keep things simple. We put an Android Nexus S (version 2.3.7) and an older iPhone 3GS (iOS version 5.1.1) through the following paces:

1. We used a single page with ~2MB of images on it to prime the cache.

2. In priming the cache, we used four different sets of cache control headers (including none at all) on the images to see their effect on caching:

  • No cache headers at all
  • A Last-Modified header only
  • A Cache-Control: max-age header, together with a Last-Modified header
  • A Cache-Control: max-age header only

3. Then, after priming the cache, we performed some user actions and revisited the page.

4. Through network traces, we determined if, when, and how the browser went back to the server to fetch any of the images.

Results

The table below shows all the permutations and how each phone behaved in various circumstances.

It’s not an easy table to read, so here’s what’s happening:

  • In all the green cells, all the images were fetched from the cache. No requests for those images went over the network to the server.
  • In all the orange cells, nothing was fetched from the cache. All the images were re-fetched from the server.
  • In the blue cells, the requests were made from the server, but with some relevant caching headers in the requests. If you want to know exactly what these headers mean, it’s best to consult RFC2616.
User action Device Primed with no cache headers Primed with Last-Modified Primed with Last-Modified and max-age Primed with max-age
New window or tab Android All from cache All from cache All from cache All from cache
iOS None from cache All from cache All from cache All from cache
Exit browser, lock phone, unlock phone, open browser Android All from cache All from cache All from cache All from cache
iOS None from cache All from cache All from cache All from cache
Kill browser, launch browser Android All from cache All from cache All from cache All from cache
iOS None from cache All from cache All from cache All from cache
Reboot phone Android All from cache All from cache All from cache All from cache
iOS None from cache All from cache All from cache All from cache
After page loads, click URL and hit “Enter” Android All from cache If-Modified-Since and Cache-Control:max-age=0 If-Modified-Since and Cache-Control:max-age=0 All from cache
iOS None from cache All from cache All from cache All from cache
Refresh after page loads Android Cache-Control:no-cache Cache-Control:no-cache Cache-Control:no-cache Cache-Control:no-cache
iOS Cache-Control:max-age=0 If-Modified-Since and Cache-Control:max-age=0 If-Modified-Since and Cache-Control:max-age=0 Cache-Control:max-age=0

Observations and conclusions

Most of the cells show behavior that is expected from a well-behaving browser. But there are a few areas Hooman highlighted as particularly interesting:

Primed with No Cache Headers

It’s interesting that Android was so aggressive and chose to cache the images even if they didn’t have caching headers on them. This is particularly aggressive.  But that’s not necessarily a bad thing, especially if this behavior is specific to images. But it makes it that much more likely that the cache will fill up quickly. iOS, by contrast, doesn’t cache any images if they don’t have any caching headers at all.

After page loads, click URL and hit “Enter”

This whole row was interesting because, to the Android, this is very much like a refresh of the page. For situations where the original images were put into the cache with the Last-Modified validator, the browser wants to validate all the way to the origin server. (Intermediate caches can’t validate on their own because of the max-age=0 directive.) This is true even if there was a max-age put on the image when it originally went into the cache.

Primed with Last-Modified

When the image went into the cache with just a Last-Modified validator, caching was rather aggressive form both Android and iOS. This is actually probably a good thing. It’s also possible that this behavior would be different if we had revisited the page much later than when the images originally went into the cache.

A few caveats

As Hooman points out, even though this is all very telling, it’s important to consider that we didn’t test the hundreds of permutations that could have included how full the cache was and combinations of different content types (images, scripts, CSS, HTML, etc). So these observations apply just to images. It’s possible that some of the behaviours could have been different if other file types were introduced into the mix.

Also, while some research out there suggests that the iOS cache isn’t persistent (it reportedly gets wiped after the device shuts down), we didn’t find this in our research.  However, the fact that we used a small amount of images could explain this discrepancy.

Final thoughts: Mobile browser vendors need to offer more visibility into their products

This is all great info and we’ll continue to test and get insight into how mobile browsers work. This is not only important for the performance community, but also vital to our own product development here at Strangeloop.

But we’d also like to re-iterate what others have said, which is to ask browser vendors to provide more visibility and documentation into what their browsers are doing (kudos to Chrome and Tony for starting to walk that path). As mobile browsing is poised to rival and overtake desktop browsing in the near future, the more we know, the better we can design our web applications and the products that help optimize them.

For a greater understanding of mobile performance challenges, I encourage you to download Strangeloop’s mobile optimization whitepaper (not a product shill, I promise).

Related posts:

By: Tammy Everts Category: Tags: , , , , ,
1 2 3 4 5 6 7 8 »