Standards and benchmarks

Browser innovation and the 14 rules for faster loading websites: Revisiting Steve’s work (part 2)

13 Sep 2012

4 Comments

A few months back, I posted part 1 of a 3-part post to answer the question:

“Aren’t many of the web performance rules described by Steve Souders in 2007 already outdated or made obsolete by browser innovation?”

Despite the fact that my company was founded on the premise that Steve’s rules are our bible, I still place a great deal of value on testing core assumptions. Fortunately, testing the first five rules revealed that these performance rules have stood the test of time and are as valid today as they ever were.

Today (finally), I want to look at the next five rules:

Methodology

1. As with my previous post, I used these test cases of the performance rules that Steve created five years ago, and which are still available. Using the Wayback Machine, I timestamped them to October 2007.

Steve Souders: 14 rules for high performing web sites2. Next, I ran each test case on Internet Explorer 6 on WebPagetest to get a sense of what Steve would have seen for performance rules 6 through 10. I did three runs of each test and used the median result.

3. Then I ran each test case on Chrome 19 to see what impact, if any, the rules have on a modern browser. Again, I performed three runs of each test and recorded the median result.

4. Using the above approach, I was able to see and compare the before-and-after results for each rule for both browsers, and then calculate the benefit. You can see the detailed test results, along with links to the tests, in this spreadsheet.

Important: The goal here is not to look at how fast Chrome 19 is versus Internet Explorer 6. We have looked at this in the past. This time I want to look at the relative benefit of each performance rule.

Findings

Here’s a high-level look at the results:

Rule 6: Move scripts to the bottom of the page

What this rule means: It’s better to move scripts from the top to as low in the page as possible. One reason is to enable progressive rendering, but another is to achieve greater download parallelization.

With scripts, progressive rendering is blocked for all content below the script. Moving scripts as low in the page as possible means there’s more content above the script that is rendered sooner. The second problem caused by scripts is blocking parallel downloads. While a script is downloading, the browser won’t start any other downloads, even on different hostnames. [From High Performance Web Sites]

Testing the rule: Steve experimented with placing scripts in various places on a page in order to see their performance impact. Using the methodology above, I was able to see the performance benefit (defined by the improvement in document complete time) of applying two of these techniques to IE6 and Chrome 19:

Rule
 Benefit in IE6 Benefit in Chrome 19
Scripts at the bottom 0.13% 18%
Deferred scripts
 31% 18%

Rule 7: Avoid CSS expressions

What this rule means: CSS expressions are an older technique, supported in Internet Explorer up until the release of IE8, which let you set CSS properties dynamically. The problem with CSS expressions was that they were evaluated more often than most people realized. As Steve pointed out in 2007, moving the mouse around the page can easily generate more than 10,000 evaluations.

This technique has fallen out of practice, largely because of findings like Steve’s. Nobody with any common sense is using this technique anymore, so it’s not relevant.

There may, however, be a modern version of this same issue. I’ve heard reports abut similar issues with JavaScript performance due to easy-to-use JS libraries such as JQuery. Similar to CSS expressions, it’s very easy to hook up functions to JavaScript events like moving the mouse and scrolling, causing similar tens of thousands of calculations to happen. I’m not sure how often this is actually happening out in the real world, but it’s something to be aware of.

Testing the rule: N/A

Rule 8: Make JavaScript and CSS external

What this rule means: If users on your site have multiple page views per session and many of your pages re-use the same scripts and stylesheets, you could potentially benefit from cached external files.

Pages that have few (perhaps only one) page view per session may find that inlining JavaScript and CSS results in faster end-user response times.

For front pages that are typically the first of many page views, there are techniques that leverage the reduction of HTTP requests that inlining provides, as well as the caching benefits achieved through using external files. One such technique is to inline JavaScript and CSS in the front page, but dynamically download the external files after the page has finished loading. Subsequent pages would reference the external files that should already be in the browser’s cache. [From High Performance Web Sites]

Testing the rule: Below are the performance gains (defined as the improvement in document complete time in the repeat view) for both browsers:

Rule
 Benefit in IE6 Benefit in Chrome 19
Cacheable external JS and CSS 37% 40%
Post-onload download 42% 45%
Dynamic inlining 38% 44%

Rule 9: Reduce DNS lookups

What this rule means: There’s no test case for this rule — because it’s a given that fewer DNS connections is a good thing — but I still want to review it here in order to talk about why it’s a good thing.

DNS lookup is when the browser looks up the domain of the object being requested by the browser. Think of this as asking the “phone book” of the internet to find someone’s phone number using their first and last name. DNS lookups are cached for better performance — on caching servers maintained by your ISP or LAN, on your operating system’s DNS cache, and in your browser cache.

Steve offers this advice: Reducing the number of unique hostnames has the potential to reduce the amount of parallel downloading that takes place in the page. Avoiding DNS lookups cuts response times, but reducing parallel downloads may increase response times. My guideline is to split these components across at least two but no more than four hostnames. This results in a good compromise between reducing DNS lookups and allowing a high degree of parallel downloads.

It’s important to know that some newer browsers (at least Chrome does) do DNS prefetching as well.

Testing the rule: N/A

Rule 10: Minify JavaScript

What this rule means: Minification is the practice of removing unnecessary characters — such as comments and white space — from code. This improves JavaScript response time because the size of the downloaded file is reduced. Minification is fairly straightforward, unlike its sister practice: obfuscation.

Like minification, obfuscation removes comments and white space, but it also munges the code — converting function and variable names into smaller strings, thereby making the code more compact as well as harder to read. While munging wasn’t originally conceived of as a performance practice, it was found that it can help performance because it reduces the code size beyond what is achieved by minification. (Warning: Obfuscation can lead to bugs, unlike minification, which is relatively problem free.)

In addition to minifying external scripts, you can also minify inlined script blocks. Even if you’re already gzipping your scripts, minifying them will still reduce the size by at least 5%.

Testing the rule: Below are the performance gains for both minification and obfuscation of different-sized files:

Rule
 Benefit in IE6 Benefit in Chrome 19
Small script minified 14% 16%
Small script obfuscated 14% 15%
Large script minified 22% 21%
Large script obfuscated 23% 21%

Conclusion

Once again, it’s reassuring to see how many of the performance rules have stood the test of time. I was really struck by how similar the results were for many of the test cases. I had expected that the rules would still be relevant, but to be perfectly honest, I had also expected that browser evolution would have resulted in more disparity between the results.

It was also interesting to see how a rule — avoid CSS expressions — has become so entrenched over time that it’s no longer an issue.

This has been a really interesting exercise so far. I’m looking forward to part 3.

Related posts:

By: Tammy Everts Category: Tags: , , , , ,

New findings: How does browser usage vary throughout the day and week?

8 Aug 2012

4 Comments

Last month, I questioned the validity of using Internet Explorer 8 as the default browser for performance testing. In the comments on that post, Stephen Pierzchala raised a good point:

I would be really interested to see a little more granularity in both the Akamai IO and Strangeloop data that shows browser usage by time of day and day of week. While it doesn’t change the aggregate results, it may indicate the type of browser that people choose to use, rather than the browser they are told to use.

We all tend to have semi-educated guesses about the browsers that people choose to use at work/school versus those they use at home, but I wasn’t able to find any studies that answer Stephen’s question. So I took a look through our beacon data to see if I could spot any trends. I got some interesting results that confirmed a few of my assumptions.

Approach

  1. Gathered data for one month of traffic for two large North America-based ecommerce sites.
  2. Pared traffic down to four browsers: Chrome, Firefox, Internet Explorer, and Safari.
  3. Aggregated data for each day of the week and plotted this on a graph for each site.
  4. Aggregated data for each hour of the day and plotted this on a graph for each site.

The results were interesting.

Observations

Browser usage throughout the week

As you can see on this pair of graphs:

  • Internet Explorer was the dominant browser for these sites.
  • Traffic for every browser, with the exception of Safari, either flatlined or decreased over the weekend.
  • Safari traffic spiked dramatically over the weekend.
  • Friday is a big browsing/shopping day across all browsers.

Browser usage throughout the day

These graphs bear a striking similarity to the daily breakdown graphs, in that IE, Firefox, and Chrome use begins to trend downward at the end of the workday, just when Safari use spikes.

Conclusions (aka “more educated guesses”)

While this data represents only two websites, I think there are enough similarities to make these findings a compelling argument for a couple of statements:

  1. Internet Explorer should still be the go-to browser for testing ecommerce sites in North America. The primary reason for this, in my opinion, is that many people are browsing/shopping while at work.
  2. Safari is a serious contender as the browser of choice for home shoppers, particularly those in desirable demographics (in other words, the Apple crowd). This is more of a hunch than an educated guess, but my feeling is that this partially based on the massive recent growth of the iPad market. I say this because everyone I know who uses a MacBook uses Chrome or Firefox instead of the default safari browser, whereas everyone I know who owns an iPad uses Safari. (Like I said, it’s just a hunch.)

I definitely don’t consider this the last word on the subject. The browser wars are far from over, and there’s enough flux in the industry that it’s anyone’s guess as to what the next year might hold. If you know of any comparable studies or statistics, let me know.

Related posts:

By: Tammy Everts Category: Tags: , , , ,

Is it time to give up Internet Explorer 8 as our default test browser?

10 Jul 2012

9 Comments

If you’re a regular user of WebPagetest, you depend on it to give you a reliable sense of how fast pages load across a variety of browsers and simulated latencies. I’m a huge fan and applaud Pat Meenan and the rest of his team in their efforts to maintain its relevance in the face of constantly shifting browser trends.

Knowing how responsive the WebPagetest team is to changing demands, I recently started wondering about why WebPagetest still uses Internet Explorer 8 as its default browser, when it’s commonly believed that Chrome is more widely used and Firefox comes in at a respectable second place. I also wondered if there are new gaps in terms of which browsers we should be testing on. So I started to do a bit of digging.

WebPagetest browsers

Why is this an important question?

While modern browsers are more similar than older browsers when it comes to embracing common standards, all browsers are still not created equal. As I’ve talked about in the past, as web pages become increasingly complex, data-intensive, and dynamic, front-end optimization (FEO) techniques that can make pages faster in some browsers can slow them down, or even break them, in another browser.

If you’re focusing your testing efforts on a single browser, such as IE8, you might just be getting a pinhole view of your site’s performance. And if the majority of your visitors use Chrome and Firefox, you might be getting an irrelevant pinhole view of your site’s performance.

How many people use Webpagetest’s default settings?

But before spending too much time contemplating the question of whether or not IE8 should be the default test browser, first I wanted to establish that this is a relevant question. If most people are using their own custom settings on WebPagetest, the defaults aren’t a huge issue.

To examine the use of default settings, last Friday afternoon I looked at roughly 24,000 public tests on WebPagetest (the sum of a half day’s testing for that day).

As this graph shows, most users are indeed using the default settings:

Global browser usage

Next, I went to Statcounter and grabbed the global browser numbers for July 1-5, and created a comparable graph:

WebPagetest versus global browser stats

Looking at both charts, there are some striking observations about these two sets of data:

  • While just over 13% of global internet users use IE8, roughly two-thirds of WebPagetests are done on IE8.
  • Chrome 20 is the dominant global browser, with a 23% share. IE9 is second, at 17%, and Firefox 13 is third, at almost 15%.
  • While WebPagetests are split almost equally between IE9 and IE7, this doesn’t mirror the purported real world data offered by Statcounter, which shows IE9 to be vastly more popular than IE7.
  • In fact, further to the above point, the number of Safari iPad users is almost double the number of IE7 users, at 2.52% and 1.38% respectively.

Here’s a graph that illustrates the differences more starkly:

Browser stats: WebPagetest use vs. global browser stats

You can see there’s a significant disconnect between the reported global browser usage and the browsers that people in our industry test on.

Based on this data, if we were to create an ideal set of testing options for WebPagetest, we would:

  • Add the capability to test on Chrome 20, and make this the default browser.
  • Add Firefox 13, Safari 5.1, and Safari iPad.
  • Question why so many of us are bothering to test on Firefox 6.

Or would we?

Some of these global usage numbers weren’t sitting well with me, because they don’t reflect the real-world data that I see every day here at Strangeloop. I gathered traffic data for the same time period, July 1-5, for a handful of Strangeloop customers using desktop browsers, representing approximately 350,000 unique visits. I broke the data down by browser version:

Out of curiosity, I compared these numbers to the data gathered by Akamai IO between July 1-5:

Akamai IO browser stats - July 1-5, 2012

While Akamai’s results aren’t identical to Strangeloop’s, they’re similar enough — specifically when it comes to the popularity of IE, Chrome, and Firefox — to validate my belief that both samples are fairly representative of typical internet users in our market.

WebPagetest versus Strangeloop browser stats

So I took Strangeloop’s browser usage numbers and overlaid them on WebPagetest’s numbers:

Browser stats: WebPagetest vs Strangeloop data

While the results don’t dovetail perfectly, you can see that IE8 usage correlates pretty decently. However, there’s a significant gap when it comes to IE9. And Chrome 20, Firefox 13, and Safari — though widely used according to both Strangeloop’s and Akamai’s data — are not yet on WebPagetest’s map.

Conclusion #1: IE8 is still relevant as the default test browser, but we should keep an eye toward using IE9 in the near future.

It would also be great if WebPagetest could add the ability to test for Chrome 20, Firefox 13, and Safari.

Conclusion #2: It could be an extremely useful feature if WebPagetest offered different defaults based on country, or even based on country plus user/application type.

This research raises some interesting new ideas. It’s pretty clear that Strangeloop and Akamai’s data is North American-centric with a bias toward ecommerce customers, and this no doubt accounts for some of the discrepancy noted above. This opens up the question of whether it would be useful to default to different browsers for different countries, and even to different browsers for different types of applications. For example, selecting “Ecommerce Dulles” would test on IE8, while a content-driven site in India should default to Chrome 20.

Takeaway: Know your users… and their browsers.

While this test validates that Internet Explorer 8 is a sound default for the mainstream North American ecommerce market, it may not be the most-used browser for your visitors.

One of our customers, Wine.com, discovered that their highest-spending customers work in the financial services sector, and they were placing orders from work, where they were restricted to IE6 and IE7, which had serious usability implications. And last year, L.L. Bean realized that 4-5% of their affluent older customers were still using IE6.

Before you spend a lot of time running tests, look into your analytics and learn who your users are.

Related posts:

By: Tammy Everts Category: Tags: , , , , , , , ,

The “performance poverty line”: What is it and why does it matter?

5 Jun 2012

6 Comments

Depending on whom you ask, the ideal page load can be 100 milliseconds, 400 milliseconds, 2 seconds… you get the picture.

In our industry, we talk a lot about big, hairy, audacious performance goals, but what about the other end of the spectrum: the performance poverty line?

The performance poverty line is the plateau at which load time ceases to matter because you’ve hit close to rock bottom in terms of business metrics. If your pages are below the performance poverty line, making them a couple of seconds faster doesn’t help your business.

But what is the performance poverty line? Is it quantifiable? Does it vary from business metric to business metric? Is it widely applicable? I went looking for it. It ended up being pretty easy to find.

Methodology

A couple of years ago, I presented a session at Velocity in which I talked about the impact of poor performance on some of the usual business metrics: conversions, bounce rate, and page views per visit. In that presentation, I showed a set of graphs that aggregated user data — millions of transactions recorded in Stangeloop’s monstrously huge analytics database — from five of our ecommerce customers.

To find out the current performance poverty line, I simply revisited these five sites, gathered data from millions of recent transactions, then plotted it on a set of graphs. For the sake of comparison, I also included the 2010 data. It makes for an interesting set of snapshots:

Performance poverty line: bounce rate

Performance poverty line: pages per visit

Performance poverty line: conversion rate

Observation #1: The performance poverty rate is around 8 seconds.

For these five sites, metrics more or less bottom out at around 8 seconds. What’s noteworthy here isn’t just this number, but how incredibly consistent it is across all three metrics.

Observation #2: When pages are slow, business metrics suffer more today than they did two years ago.

This doesn’t apply just to the 8-second mark — it’s consistent at every page speed. For example, look at the final graph. A page that took 6 seconds to load in 2010 suffered a -40% conversion hit. Today, a 6-second page takes a -50% hit. Or look at the second graph, where you see that a relatively fast 4-second page suffers a -30% hit in page views per visit in 2010, but in 2012 a 4-second page takes more than a -40% hit. This indicates a huge change in consumer expectations. Simply put, users are less willing than ever to suffer slow pages.

Takeaway: Try this with your own data.

There’s no point in killing yourself to take your pages from 16 seconds to 8 seconds if you don’t budge your key business metrics. I invite you to determine your site’s performance poverty line with your own data, assuming you have enough data with which to perform statistically relevant analysis. The performance poverty line for your site might be slightly different than for the sites I tested, depending on your content and your audience. Once you’ve identified your poverty line, you’ll know your lowest bar for performance.

When you do determine your site’s poverty line, I’d love to know if it’s close to my results or radically different. Drop me a line and let me know.

Related posts:

By: Tammy Everts Category: Tags: , , ,

Browser innovation and the 14 rules for faster loading websites: Revisiting Steve’s work (part 1)

29 May 2012

13 Comments

(See parts 2 and 3.)

As you may or may not know, there are 14 core optimization rules, set out five years ago by web performance guru Steve Souders in his groundbreaking book High Performance Web Sites. These rules are our industry’s commandments. They’ve been tested and proven and they underpin the work we do every day.

But last week I was in a meeting with a CTO who wondered about the lifespan of these rules. He asked:

“Aren’t many of the performance rules already outdated or made obsolete by browser innovation?”

I have a well-rehearsed answer to this kind of performance heresy, which starts with a clear NO, but after the meeting I started thinking. The High Performance Web Sites research and rules are five years old — back when Internet Explorer 6 was still the dominant browser. Browsers have changed dramatically since then, which raises the question: Is the lens that Steve used five years ago still relevant?

I firmly believe that assumptions should be tested on a regular basis. I took a few hours and started investigating.

Methodology

First, I needed to find some test cases that Steve would have used to test his theories. I was going to ping him directly, but I stumbled across these test cases, which are still available. Using the Wayback Machine, I timestamped them to October 2007.

Steve Souders: 14 rules for high performing web sitesNext, I tested IE6 versus Chrome 19 on WebPagetest to get a sense of what Steve would have seen for the first five performance rules. (I would have done all 14, but ran out of time. Stay tuned for the rest another day.)

Findings

Using the above approach, I was able to see and compare the before-and-after results for each rule for both browsers, and then calculate the benefit. You can see the detailed test results, along with links to the tests, in this spreadsheet.

Important: The goal was not to look at how fast Chrome is versus Internet Explorer. We have looked at this in the past. This time I want to look at the relative benefit of each performance rule.

Here’s a high-level look at the results:

Rule 1: Make fewer HTTP requests

What this rule means: 80% of the end-user response time is spent on the front end. (Related: We re-tested this assumption last year, and it still holds.) Much of this time is spent downloading page objects like images, stylesheets, etc. By reducing the number of these objects, you reduce the number of HTTP requests needed in order to render the page in the browser.

Testing the rule: Steve identified a handful of techniques to help you reduce requests. Using the methodology above, I was able to see the performance benefit (defined by the improvement in document complete time) of applying two of these techniques to IE6 and Chrome 19:

Rule
 Benefit in IE6 Benefit in Chrome 19
Image map 25% 20%
CSS sprites
 25% 18%

Rule 2: Use a content delivery network

What this rule means: Latency is a major performance challenge. (Read more about latency here.) Content delivery networks (CDNs) mitigate latency by caching page content in distributed servers across a region or worldwide, thereby bringing content closer to users and reducing the round trip time.

Testing the rule: Here we can see the performance benefit — and deficit — of using a CDN for both browsers:

Rule
 Benefit in IE6 Benefit in Chrome 19
Use a CDN 14% -2%

Rule 3: Add an Expires header

What this rule means: When a visitor comes to your page for the first time, their browser caches all the objects on that page. After the objects are cached, the browser uses them when required on subsequent pages, saving the browser from having to make HTTP requests for the same objects over and over. This makes those subsequent pages load faster.

Adding Expires headers lets you control how long your page objects are stored in the visitor’s browser cache, so that a visitor can return to your site days, weeks, even months in the future and still experience faster page loads. Expires headers are most often used with images, but they can be used on all objects, from scripts to Flash components.

Testing the rule: Below are the performance gains (defined as the improvement in document complete time in the repeat view) for both browsers:

Rule
 Benefit in IE6 Benefit in Chrome 19
Add an Expires header 70% 59%

Rule 4: Gzip page components

What this rule means: Compression is a core treatment that reduces response times by reducing the size of the HTTP response. Compressing as many file types as possible is an easy way to reduce page weight and make pages faster. At the time Steve developed his rules, Gzip was the most popular and effective compression method available. There are a couple of ways to go with this technique: Gzip just the HTML or Gzip everything.

Testing the rule: Here are the performance gains, again defined as the improvement in document complete time:

Rule
 Benefit in IE6 Benefit in Chrome 19
HTML Gzipped 2% 10%
Everything Gzipped
 24% 44%

Rule 5: Put stylesheets at the top

What this rule means: Moving stylesheets to the document HEAD makes pages load faster because it allows the page to render progressively — that is, it allows the browser to display whatever content it has as soon as possible. At the time Steve wrote the rules, putting stylesheets at the bottom of a page prohibited progressive rendering in many browsers, particularly Internet Explorer.

Testing the rule: Below are the performance gains, this time defined as the improvement in start render time:

Rule
 Benefit in IE6 Benefit in Chrome 19
Put stylesheets at the top 74% 59%

Interpretation

Here’s my take on these findings:

Reducing roundtrips still matters.

As the results for rules 1 and 3 demonstrate, it doesn’t matter how much better modern browsers are at rendering page objects; fewer calls to the server still make a huge difference. When I see that adding an Expires header leads to a 59% improvement in Chrome 19, that tells me this technique is still incredibly relevant.

A CDN helps in some situations, but not all.

I’ve written about this before, but it bears repeating. Your benefits will vary depending on which CDN you choose, as well as things like how your CDN stores content and how far their PoPs are from users. A CDN is a must for many sites, but it’s not a standalone performance solution.

Compression still helps. A lot.

Not only did Gzipping offer benefits in Chrome, it offered even greater benefits than in IE6. This, to me, is a really compelling finding, because it shows that despite the advances made in modern browsers, modern pages have also changed a lot — meaning that more than ever they can benefit from compression.

The location of stylesheets is important, but it depends on the page composition.

Start render really matters and the location of the stylesheets is critical.

Conclusion

It’s reassuring to know that the performance rules — or, at least, these five rules — have stood the test of time. They still apply and are as relevant today as they were five years ago.

(See parts 2 and 3.)

Related posts:

By: Tammy Everts Category: Tags: , , , , , , ,
1 2 3 4 5 6 7 8