mobile web performance

What we learned from testing the mobile load times of 200 top retail sites over 3G and LTE

3 Oct 2012

6 Comments

Earlier today, I had an exciting opportunity to present some of the findings of Strangeloop’s newest study, the 2012 State of Mobile Ecommerce Performance at Velocity EU. I’ll be posting those slides shortly, but first want to take a moment to talk about this study and why we’re so excited about it.

Why test over cellular networks?

Last spring, we were sitting in the office talking about mobile performance measurement and why it’s so hard to get reliable real-world numbers, especially over cellular networks. This seemed like a glaring knowledge gap, but it’s an understandable gap given the massive variability in network performance. Last year, I did an informal latency survey and found that latency can fluctuate wildly, even in the same location at the same time.

3G latency

Methodology: How to DIY your own RUM tests for mobile

So we set out to develop a reliable methodology for measuring mobile performance over cellular networks, which takes this variability into account. We also wanted to create a methodology that anyone could recreate fairly easily.

Here’s how we did it:

  • We focused on 200 leading retail sites, as ranked by Alexa.com.
  • We selected six testing devices: iPhone 4, iPhone 5, Samsung Galaxy S smartphone, Samsung Galaxy S3 smartphone, iPad 2, and Samsung Galaxy tablet.
  • iPhone 4, Galaxy S, iPad 2 and Galaxy tablet were tested over 3G.
  • iPhone 5 and Galaxy S3 were tested over LTE.
  • We tested each site’s home page 30 times per device, using the device’s native browser. By testing this many times and capturing the median result, we hoped to take into account latency variability.
  • For all tests, the devices were positioned in the same location, in an attempt to mitigate the latency impact caused by location changes.
  • For all tests, wifi was turned off, devices and radios were at full power, and screens were not allowed to lock during testing.
  • The tests in this study were conducted using a RUM beacon inserted into the page that captured the onload time. The cache and cookies were cleared automatically between each test.

The results

As I shared at Velocity, the results were a mix of the predictable and the surprising:

Predictable: Pages are slow over 3G.

It’s not that shocking to learn that the median pages took 11 or more seconds to load over 3G.

Surprising: A significant number of pages took 20+ seconds to load.

While we expected pages to be slow, we didn’t expect that 9% of the pages we tested took more than 20 seconds to load on the iPhone.

Predictable: LTE was faster than 3G.

No explanation needed here.

Surprising: LTE was only 27% faster than 3G.

I’ve heard claims that LTE is, on average, ten times faster than 3G. Our results suggest that LTE performance gains might be more unpredictable than this.

Predictable: Pages loaded 22% faster on the iPad than on the Galaxy tablet.

iPad fans might scoff that we even bothered to test this.

Surprising: The Galaxy S3 phone loaded pages 9% faster than the iPhone 5.

It’ll be interesting to hear what iPhone fans make of this. (Full disclosure: I have an iPhone 5 and have to admit I was kind of affronted to see these results.)

Check out our infographics of the report’s highlights, below, and I encourage you to read the findings for yourself. If you have any questions about our findings or approach, don’t hesitate to ask.

Load times of 200 leading retail sites on mobile devices over 3G and LTE networks

Related posts:

By: Tammy Everts Category: Tags: , , , , , , ,

O’Reilly webcast: Mobile web performance trends and predictions [SLIDES]

27 Sep 2012

4 Comments

Earlier this week, I had the privilege of participating in an O’Reilly webcast as a preview for my session at Velocity EU next week, where I’ll be presenting the results of Strangeloop’s first annual state of the union on mobile ecommerce performance (not to be confused with our quarterly SoTU on desktop performance, which was most recently released last week).

If you missed the webcast on Tuesday, here are the slides:

We covered a lot of ground. Here’s an overview:

  • 3-5: The mobile market
  • 6-12: Case studies: Why is faster better?
  • 13-23: Measuring mobile performance: tools and tips
  • 24-29: Visualizing performance
  • 30: How does data work on a mobile network?
  • 31-91: Best practices in action
  • 92-102: Mobile caching (it’s a good thing)
  • 103-111: Evolution: Where mobile is heading
  • 112-115: Sneak peek: 2012 State of Mobile Ecommerce Performance

As you can see, I saved the preview for near the end. But I saved the best slide for last… my costume for our “Celebrity Dress-Alike Day” at Strangeloop on Tuesday:

If you have any questions about these slides, drop me a note in the comments. And if you’re going to Velocity next week, I hope to see you there!

Related posts:

By: Tammy Everts Category: Tags: , , , , , ,

New findings: Ecommerce sites are 9% slower than in 2011

18 Sep 2012

12 Comments

Two years ago, when Strangeloop started tracking the load times of 2,000 top North American ecommerce sites, we had a hunch we’d spot some interesting trends over time. We did not expect, however, to see that pages are continuing to get slower rather than faster. Yet according to the Fall 2012 release of our quarterly Ecommerce Page Speed and Web Performance State of the Union, which came out today, that’s exactly what’s happening.

Not only are pages slower, they’re dramatically slower. Since November 2011, when we last tested these sites, the median home page has taken a 9% performance hit, with load time increasing from 5.94 seconds to 6.5 seconds. This flies in the face of conventional belief that, thanks to faster browsers, networks, and devices, the average end user is enjoying a premium online experience. This is clearly not the case. As user expectations continue to grow, the gap between expectations and reality continues to widen.

Ecommerce Page Speed and Web Performance State of the Union [Fall 2012]

These graphics (higher res version here) illustrate a few of our  findings. I encourage you to download the report to read the rest. Without giving it all away, here are a few highlights:

  • Internet Explorer 10 served pages faster than other browsers, most notably 8% faster than Chrome 20. We tested each page across a number of browsers, including the latest versions of IE, Chrome, and Firefox. It’s important to bear in mind that these were simple tests that didn’t take into account the many nuances of browser performance (which is discussed further in the report), but we considered these results interesting enough to share.
  • Top sites are 10% slower than the pack as a whole. While the median site took 6.5 seconds to load, we saw even poorer results when we looked at the top 100 sites (ranked by revenue and profitability), with the median Alexa 100 home page having a load time of 7.14 seconds. Check out the report for our thoughts on this.
  • Many sites are still not following core performance best practices. We found that 30% of sites tested did not use compression, and 12% did not use keep-alives. As I’ve talked about elsewhere, these two fairly simple techniques can yield big results, including up to 52% improvement in start render time.

Why you should care about these findings

To my knowledge, Strangeloop’s state of the union reports (which we’re now releasing on a quarterly basis) are the only ongoing surveys that measure performance from the perspective of real users. By using WebPagetest, we can simulate performance across browsers and realistic latencies, and get a real-world look at how websites actually behave. It’s easy for site owners to fall into the trap of thinking that their sites are fast for everyone, because site owners are typically seeing benchmark tests run out of datacenters.

I want to emphasize that reports like this one are not a substitute for the real user monitoring you should be performing on your site on an ongoing basis. Instead, consider it a snapshot that we can collectively hold up as a mirror of big-picture ecommerce performance.

As always, I welcome your feedback and questions.

Download the report: State of the Union: Ecommerce Page Speed and Website Performance [Fall 2012]

Download a high-res version of the infographics above (and feel free to re-post): Poster: Ecommerce Page Speed and Website Performance [Fall 2012]

Related posts:

By: Tammy Everts Category: Tags: , , , , , , , , , , , ,

Four things you need to know about Millennials, mobile adoption, and performance expectations

6 Sep 2012

3 Comments

This past summer, Strangeloop was fortunate to welcome some excellent interns, who’ve just returned to school. While not all were developer types, I was really struck by how comfortable they all seemed when working with and talking about new technologies. We could throw anything at them — including some tricky research projects — and their ability to innovate on their feet was incredible.

While there’s something to be said for our ability to pick awesome interns, I think these  observations are an indicator of a larger trend. While it’s a total cliché to talk about how wired kids are these days, it’s a cliché for a reason: all the college/university-aged people I know take mobile devices and connectivity as a given. It’s no coincidence that Amazon chose the first week of school to make a bunch of big announcements about its e-reader and tablet products.

To me, this marks a critical opportunity to talk about the major shifts we’re going to see in consumer behaviour as “Millennials” (not a term I’m a fan of, by the way; feel free to suggest another) enter the mainstream media/commerce marketplace. I want to use this post to corral some interesting research about the next generation of mainstream web users, from their use of mobile devices to their performance expectations.*

Why a post that focuses on young people, and on students in particular?

We can’t fall into the Victorian trap of thinking that younger people are just nascent versions of ourselves. For a number of reasons — from neurological to cultural — people between the ages of 18-25 think differently and use the web and technology differently.

Millennials are the next generation of mainstream media consumers and online shoppers. As they emerge into their full consumer power in the next few years, site owners who are prepared to respond to their online expectations will have a serious competitive advantage.

Most students are online…

  • 98-99% of undergraduate and graduate students access the internet, compared to 75% of all adults [*]
  • 93% of students research online rather than at the library [*]
  • 70% of students use laptops or tablets instead of pen and paper for taking notes [*]
  • e-text sales have grown from $5M to $47M between 2005 and 2012 [*]

…and most are mobile.

  • 92% use smartphones, tablets, or laptops to go online, compared to 57% of all adults. [*]
  • Nearly half of smartphone users ages 18-29 (45%) do most of their online browsing on their phone, compared to 29% of adults 30-49. [*]
  • Tablet ownership among college students and high school seniors has risen dramatically in just one year — from 7% in 2011 to 25% in January 2012. [*]
  • As well, in January 2012, over one-third of college students (36%) and one-quarter of college-bound high school seniors (26%) said they planned to purchase a tablet in the next six months. [*]
  • Most students believe print textbooks will be obsolete in five years. [*]

(I’d love to get my eyeballs on more recent data on tablet ownership. Anyone?)

Millennials are more efficient…

In usability tests on reasonably well-designed applications, young people are much better than average at completing tasks (in one case, 100% success rate, compared to a 75% success rate for the general adult population). [*]

…and more impatient.

Despite (or because of) this higher level of efficiency, they have dramatically lower patience levels than the general adult population when pages are slow or behave clumsily. [*]

Final thoughts: If this information is self-evident, why are so many sites and applications still so slow?

Some of this research might seem self-evident to you (it certainly did to me), but it still bears mentioning. Why? Because we know these four things:

  1. Millennials are set to surge into mainstream consumerism of media, products, and workplace applications.
  2. Millennials would rather do most things online.
  3. Millennials are extremely good at completing online tasks (such as, say, filling and checking out a shopping cart).
  4. Millennials are one of the fastest-growing groups of mobile adopters.

And yet despite knowing these things, most sites today are still far too slow and far too clunky — especially for mobile users.

Site owners and app vendors have a clear opportunity to appeal to this very appealing demographic simply by making their pages faster. And most site owners and app vendors are missing out.

*This ended up being more challenging than I expected. Trying to find up-to-date research is next to impossible, due to the fact that technology and young people have two things in common: they’re both incredibly fluid and adaptive. In other words, they both change and evolve extremely quickly. (e.g. Studies that are just a couple of years old cite the incredible popularity of MySpace.)

Related posts:

By: Tammy Everts Category: Tags: , , , ,

15 things you can (and should) do to make your site faster for mobile users

29 Aug 2012

9 Comments

August is Speed Awareness Month, and performance experts throughout our community are contributing their perspectives. My post about must-have mobile optimization techniques just went live. Here’s the intro…

As you probably know, mobile users face unique performance challenges:

  • Slower networks (which is why reducing requests and payload is important)
  • Mobile browsers are slower to parse HTML and slower to execute JavaScript (so optimizing client-side processing is crucial)
  • Mobile browser caches are much smaller than those of desktop browsers (requiring new approaches to leveraging local storage of reusable resources)

There are a number of fundamental and advanced tactics that you can use to take on these challenges and win. I’m going to give an overview of each one here, but to get more insight and specific how-to details, I urge you to download Strangeloop’s whitepaper on mobile website optimization.

1. Consolidate resources

By now, this is a standard practice, so I’m not going to explain it here, but I want to point out that resource consolidation can be a double-edged sword for mobile browsers. Reducing requests speeds up page load the first time, but larger consolidated resources may not be cached efficiently. So if you’re using this technique, make sure to balance it with techniques to optimize local storage (coming up next).

2. Use browser caching and localStorage

Caching is an essential technique for achieving acceptable performance, but desktop and mobile caches are not created equal. Mobile browser caches are usually much smaller than on desktop machines, causing items to be flushed quickly, so traditional browser caching doesn’t work well for mobile devices. Luckily, the HTML5 Web Storage specification, which has been implemented in all major desktop and mobile browsers, provides a great alternative to relying solely on browser caching.

3. Embed resources in HTML for first-time use

If a resource doesn’t have a high likelihood of already being cached (such as in a mobile context), it is often best to embed that resource in the page’s HTML — a technique called “inlining” — rather than storing it externally and linking to it.

The disadvantage of inlining is that page size can become very large, so it’s crucial for a web application that uses this strategy to be able to track when the resource is needed and when it is already cached on the client. In addition, the application must generate code to store the resource on the client after sending it inline the first time. For this reason, using HTML5 localStorage on mobile devices (as described above) is a great companion to inlining.

4. Use HTML5 server-sent events

The HTML5 EventSource object and Server-Sent events enable JavaScript code in the browser to open a much more efficient channel from the server to the browser, which the server can then use to send data as it becomes available, eliminating the HTTP overhead of creating multiple polling requests.

5. Eliminate redirects

When users try to navigate to a standard desktop site from a mobile device, this often generates an extra round trip to the client and back to the mobile site, consuming several hundred milliseconds over mobile networks. For obvious reasons, it’s faster to deliver the mobile web page directly in response to the original request, rather than delivering a redirect message that then requests the mobile page. And as a courtesy to users who prefer to view the desktop site on their mobile device, you can provide a link on the mobile site that signals your application to suppress this behavior.

…

Those are the first five tips. You’ll have to go to the original post to read the other ten. :)

Related posts:

By: Tammy Everts Category: Tags: , ,
1 2 3 4 5 6 7 8 »