First Annual State of the Union for Mobile Ecommerce Performance [SLIDES]

9 Oct 2012

2 Comments

I’m a bit late posting these, but here’s the deck from my session at Velocity EU, where I talked about the methodology and findings in Strangeloop’s new mobile performance study.

In our study (available for download here), we looked at the performance of 200 top ecommerce sites on a handful of mobile devices, over 3G and LTE. Our key areas of exploration included:

Usability

  • How many companies have mobile-specific sites?
  • Who do they serve them to?
  • How easy is it to access the full site from a mobile device?

Speed

  • How fast are pages on 3G?
  • How fast are pages on LTE?

Devices

  • Which is faster: an Android or an iPhone?
  • Does the iPhone 5 outperform Android?
  • Which is faster: an iPad or Android tablet?

As always, if you have questions about any of the content of these slides, let me know.

Related posts:

By: Tammy Everts Category: Ecommerce, Mobile, Research

What we learned from testing the mobile load times of 200 top retail sites over 3G and LTE

3 Oct 2012

6 Comments

Earlier today, I had an exciting opportunity to present some of the findings of Strangeloop’s newest study, the 2012 State of Mobile Ecommerce Performance at Velocity EU. I’ll be posting those slides shortly, but first want to take a moment to talk about this study and why we’re so excited about it.

Why test over cellular networks?

Last spring, we were sitting in the office talking about mobile performance measurement and why it’s so hard to get reliable real-world numbers, especially over cellular networks. This seemed like a glaring knowledge gap, but it’s an understandable gap given the massive variability in network performance. Last year, I did an informal latency survey and found that latency can fluctuate wildly, even in the same location at the same time.

3G latency

Methodology: How to DIY your own RUM tests for mobile

So we set out to develop a reliable methodology for measuring mobile performance over cellular networks, which takes this variability into account. We also wanted to create a methodology that anyone could recreate fairly easily.

Here’s how we did it:

  • We focused on 200 leading retail sites, as ranked by Alexa.com.
  • We selected six testing devices: iPhone 4, iPhone 5, Samsung Galaxy S smartphone, Samsung Galaxy S3 smartphone, iPad 2, and Samsung Galaxy tablet.
  • iPhone 4, Galaxy S, iPad 2 and Galaxy tablet were tested over 3G.
  • iPhone 5 and Galaxy S3 were tested over LTE.
  • We tested each site’s home page 30 times per device, using the device’s native browser. By testing this many times and capturing the median result, we hoped to take into account latency variability.
  • For all tests, the devices were positioned in the same location, in an attempt to mitigate the latency impact caused by location changes.
  • For all tests, wifi was turned off, devices and radios were at full power, and screens were not allowed to lock during testing.
  • The tests in this study were conducted using a RUM beacon inserted into the page that captured the onload time. The cache and cookies were cleared automatically between each test.

The results

As I shared at Velocity, the results were a mix of the predictable and the surprising:

Predictable: Pages are slow over 3G.

It’s not that shocking to learn that the median pages took 11 or more seconds to load over 3G.

Surprising: A significant number of pages took 20+ seconds to load.

While we expected pages to be slow, we didn’t expect that 9% of the pages we tested took more than 20 seconds to load on the iPhone.

Predictable: LTE was faster than 3G.

No explanation needed here.

Surprising: LTE was only 27% faster than 3G.

I’ve heard claims that LTE is, on average, ten times faster than 3G. Our results suggest that LTE performance gains might be more unpredictable than this.

Predictable: Pages loaded 22% faster on the iPad than on the Galaxy tablet.

iPad fans might scoff that we even bothered to test this.

Surprising: The Galaxy S3 phone loaded pages 9% faster than the iPhone 5.

It’ll be interesting to hear what iPhone fans make of this. (Full disclosure: I have an iPhone 5 and have to admit I was kind of affronted to see these results.)

Check out our infographics of the report’s highlights, below, and I encourage you to read the findings for yourself. If you have any questions about our findings or approach, don’t hesitate to ask.

Load times of 200 leading retail sites on mobile devices over 3G and LTE networks

Related posts:

By: Tammy Everts Category: Ecommerce, Infographics, Mobile, Performance measurement, Research Tags: , , , , , , ,

O’Reilly webcast: Mobile web performance trends and predictions [SLIDES]

27 Sep 2012

4 Comments

Earlier this week, I had the privilege of participating in an O’Reilly webcast as a preview for my session at Velocity EU next week, where I’ll be presenting the results of Strangeloop’s first annual state of the union on mobile ecommerce performance (not to be confused with our quarterly SoTU on desktop performance, which was most recently released last week).

If you missed the webcast on Tuesday, here are the slides:

We covered a lot of ground. Here’s an overview:

  • 3-5: The mobile market
  • 6-12: Case studies: Why is faster better?
  • 13-23: Measuring mobile performance: tools and tips
  • 24-29: Visualizing performance
  • 30: How does data work on a mobile network?
  • 31-91: Best practices in action
  • 92-102: Mobile caching (it’s a good thing)
  • 103-111: Evolution: Where mobile is heading
  • 112-115: Sneak peek: 2012 State of Mobile Ecommerce Performance

As you can see, I saved the preview for near the end. But I saved the best slide for last… my costume for our “Celebrity Dress-Alike Day” at Strangeloop on Tuesday:

If you have any questions about these slides, drop me a note in the comments. And if you’re going to Velocity next week, I hope to see you there!

Related posts:

By: Tammy Everts Category: Case studies, Ecommerce, Mobile Tags: , , , , , ,

New findings: Ecommerce sites are 9% slower than in 2011

18 Sep 2012

12 Comments

Two years ago, when Strangeloop started tracking the load times of 2,000 top North American ecommerce sites, we had a hunch we’d spot some interesting trends over time. We did not expect, however, to see that pages are continuing to get slower rather than faster. Yet according to the Fall 2012 release of our quarterly Ecommerce Page Speed and Web Performance State of the Union, which came out today, that’s exactly what’s happening.

Not only are pages slower, they’re dramatically slower. Since November 2011, when we last tested these sites, the median home page has taken a 9% performance hit, with load time increasing from 5.94 seconds to 6.5 seconds. This flies in the face of conventional belief that, thanks to faster browsers, networks, and devices, the average end user is enjoying a premium online experience. This is clearly not the case. As user expectations continue to grow, the gap between expectations and reality continues to widen.

Ecommerce Page Speed and Web Performance State of the Union [Fall 2012]

These graphics (higher res version here) illustrate a few of our  findings. I encourage you to download the report to read the rest. Without giving it all away, here are a few highlights:

  • Internet Explorer 10 served pages faster than other browsers, most notably 8% faster than Chrome 20. We tested each page across a number of browsers, including the latest versions of IE, Chrome, and Firefox. It’s important to bear in mind that these were simple tests that didn’t take into account the many nuances of browser performance (which is discussed further in the report), but we considered these results interesting enough to share.
  • Top sites are 10% slower than the pack as a whole. While the median site took 6.5 seconds to load, we saw even poorer results when we looked at the top 100 sites (ranked by revenue and profitability), with the median Alexa 100 home page having a load time of 7.14 seconds. Check out the report for our thoughts on this.
  • Many sites are still not following core performance best practices. We found that 30% of sites tested did not use compression, and 12% did not use keep-alives. As I’ve talked about elsewhere, these two fairly simple techniques can yield big results, including up to 52% improvement in start render time.

Why you should care about these findings

To my knowledge, Strangeloop’s state of the union reports (which we’re now releasing on a quarterly basis) are the only ongoing surveys that measure performance from the perspective of real users. By using WebPagetest, we can simulate performance across browsers and realistic latencies, and get a real-world look at how websites actually behave. It’s easy for site owners to fall into the trap of thinking that their sites are fast for everyone, because site owners are typically seeing benchmark tests run out of datacenters.

I want to emphasize that reports like this one are not a substitute for the real user monitoring you should be performing on your site on an ongoing basis. Instead, consider it a snapshot that we can collectively hold up as a mirror of big-picture ecommerce performance.

As always, I welcome your feedback and questions.

Download the report: State of the Union: Ecommerce Page Speed and Website Performance [Fall 2012]

Download a high-res version of the infographics above (and feel free to re-post): Poster: Ecommerce Page Speed and Website Performance [Fall 2012]

Related posts:

By: Tammy Everts Category: Browsers, Ecommerce, Page speed tests, Performance measurement Tags: , , , , , , , , , , , ,

Browser innovation and the 14 rules for faster loading websites: Revisiting Steve’s work (part 2)

13 Sep 2012

4 Comments

A few months back, I posted part 1 of a 3-part post to answer the question:

“Aren’t many of the web performance rules described by Steve Souders in 2007 already outdated or made obsolete by browser innovation?”

Despite the fact that my company was founded on the premise that Steve’s rules are our bible, I still place a great deal of value on testing core assumptions. Fortunately, testing the first five rules revealed that these performance rules have stood the test of time and are as valid today as they ever were.

Today (finally), I want to look at the next five rules:

Methodology

1. As with my previous post, I used these test cases of the performance rules that Steve created five years ago, and which are still available. Using the Wayback Machine, I timestamped them to October 2007.

Steve Souders: 14 rules for high performing web sites2. Next, I ran each test case on Internet Explorer 6 on WebPagetest to get a sense of what Steve would have seen for performance rules 6 through 10. I did three runs of each test and used the median result.

3. Then I ran each test case on Chrome 19 to see what impact, if any, the rules have on a modern browser. Again, I performed three runs of each test and recorded the median result.

4. Using the above approach, I was able to see and compare the before-and-after results for each rule for both browsers, and then calculate the benefit. You can see the detailed test results, along with links to the tests, in this spreadsheet.

Important: The goal here is not to look at how fast Chrome 19 is versus Internet Explorer 6. We have looked at this in the past. This time I want to look at the relative benefit of each performance rule.

Findings

Here’s a high-level look at the results:

Rule 6: Move scripts to the bottom of the page

What this rule means: It’s better to move scripts from the top to as low in the page as possible. One reason is to enable progressive rendering, but another is to achieve greater download parallelization.

With scripts, progressive rendering is blocked for all content below the script. Moving scripts as low in the page as possible means there’s more content above the script that is rendered sooner. The second problem caused by scripts is blocking parallel downloads. While a script is downloading, the browser won’t start any other downloads, even on different hostnames. [From High Performance Web Sites]

Testing the rule: Steve experimented with placing scripts in various places on a page in order to see their performance impact. Using the methodology above, I was able to see the performance benefit (defined by the improvement in document complete time) of applying two of these techniques to IE6 and Chrome 19:

Rule
 Benefit in IE6 Benefit in Chrome 19
Scripts at the bottom 0.13% 18%
Deferred scripts
 31% 18%

Rule 7: Avoid CSS expressions

What this rule means: CSS expressions are an older technique, supported in Internet Explorer up until the release of IE8, which let you set CSS properties dynamically. The problem with CSS expressions was that they were evaluated more often than most people realized. As Steve pointed out in 2007, moving the mouse around the page can easily generate more than 10,000 evaluations.

This technique has fallen out of practice, largely because of findings like Steve’s. Nobody with any common sense is using this technique anymore, so it’s not relevant.

There may, however, be a modern version of this same issue. I’ve heard reports abut similar issues with JavaScript performance due to easy-to-use JS libraries such as JQuery. Similar to CSS expressions, it’s very easy to hook up functions to JavaScript events like moving the mouse and scrolling, causing similar tens of thousands of calculations to happen. I’m not sure how often this is actually happening out in the real world, but it’s something to be aware of.

Testing the rule: N/A

Rule 8: Make JavaScript and CSS external

What this rule means: If users on your site have multiple page views per session and many of your pages re-use the same scripts and stylesheets, you could potentially benefit from cached external files.

Pages that have few (perhaps only one) page view per session may find that inlining JavaScript and CSS results in faster end-user response times.

For front pages that are typically the first of many page views, there are techniques that leverage the reduction of HTTP requests that inlining provides, as well as the caching benefits achieved through using external files. One such technique is to inline JavaScript and CSS in the front page, but dynamically download the external files after the page has finished loading. Subsequent pages would reference the external files that should already be in the browser’s cache. [From High Performance Web Sites]

Testing the rule: Below are the performance gains (defined as the improvement in document complete time in the repeat view) for both browsers:

Rule
 Benefit in IE6 Benefit in Chrome 19
Cacheable external JS and CSS 37% 40%
Post-onload download 42% 45%
Dynamic inlining 38% 44%

Rule 9: Reduce DNS lookups

What this rule means: There’s no test case for this rule — because it’s a given that fewer DNS connections is a good thing — but I still want to review it here in order to talk about why it’s a good thing.

DNS lookup is when the browser looks up the domain of the object being requested by the browser. Think of this as asking the “phone book” of the internet to find someone’s phone number using their first and last name. DNS lookups are cached for better performance — on caching servers maintained by your ISP or LAN, on your operating system’s DNS cache, and in your browser cache.

Steve offers this advice: Reducing the number of unique hostnames has the potential to reduce the amount of parallel downloading that takes place in the page. Avoiding DNS lookups cuts response times, but reducing parallel downloads may increase response times. My guideline is to split these components across at least two but no more than four hostnames. This results in a good compromise between reducing DNS lookups and allowing a high degree of parallel downloads.

It’s important to know that some newer browsers (at least Chrome does) do DNS prefetching as well.

Testing the rule: N/A

Rule 10: Minify JavaScript

What this rule means: Minification is the practice of removing unnecessary characters — such as comments and white space — from code. This improves JavaScript response time because the size of the downloaded file is reduced. Minification is fairly straightforward, unlike its sister practice: obfuscation.

Like minification, obfuscation removes comments and white space, but it also munges the code — converting function and variable names into smaller strings, thereby making the code more compact as well as harder to read. While munging wasn’t originally conceived of as a performance practice, it was found that it can help performance because it reduces the code size beyond what is achieved by minification. (Warning: Obfuscation can lead to bugs, unlike minification, which is relatively problem free.)

In addition to minifying external scripts, you can also minify inlined script blocks. Even if you’re already gzipping your scripts, minifying them will still reduce the size by at least 5%.

Testing the rule: Below are the performance gains for both minification and obfuscation of different-sized files:

Rule
 Benefit in IE6 Benefit in Chrome 19
Small script minified 14% 16%
Small script obfuscated 14% 15%
Large script minified 22% 21%
Large script obfuscated 23% 21%

Conclusion

Once again, it’s reassuring to see how many of the performance rules have stood the test of time. I was really struck by how similar the results were for many of the test cases. I had expected that the rules would still be relevant, but to be perfectly honest, I had also expected that browser evolution would have resulted in more disparity between the results.

It was also interesting to see how a rule — avoid CSS expressions — has become so entrenched over time that it’s no longer an issue.

This has been a really interesting exercise so far. I’m looking forward to part 3.

Related posts:

By: Tammy Everts Category: Research, Standards and benchmarks Tags: , , , , ,
« 6 7 8 9 10 11 12 13 »