Is it time to give up Internet Explorer 8 as our default test browser?

If you’re a regular user of WebPagetest, you depend on it to give you a reliable sense of how fast pages load across a variety of browsers and simulated latencies. I’m a huge fan and applaud Pat Meenan and the rest of his team in their efforts to maintain its relevance in the face of constantly shifting browser trends.

Knowing how responsive the WebPagetest team is to changing demands, I recently started wondering about why WebPagetest still uses Internet Explorer 8 as its default browser, when it’s commonly believed that Chrome is more widely used and Firefox comes in at a respectable second place. I also wondered if there are new gaps in terms of which browsers we should be testing on. So I started to do a bit of digging.

WebPagetest browsers

Why is this an important question?

While modern browsers are more similar than older browsers when it comes to embracing common standards, all browsers are still not created equal. As I’ve talked about in the past, as web pages become increasingly complex, data-intensive, and dynamic, front-end optimization (FEO) techniques that can make pages faster in some browsers can slow them down, or even break them, in another browser.

If you’re focusing your testing efforts on a single browser, such as IE8, you might just be getting a pinhole view of your site’s performance. And if the majority of your visitors use Chrome and Firefox, you might be getting an irrelevant pinhole view of your site’s performance.

How many people use Webpagetest’s default settings?

But before spending too much time contemplating the question of whether or not IE8 should be the default test browser, first I wanted to establish that this is a relevant question. If most people are using their own custom settings on WebPagetest, the defaults aren’t a huge issue.

To examine the use of default settings, last Friday afternoon I looked at roughly 24,000 public tests on WebPagetest (the sum of a half day’s testing for that day).

As this graph shows, most users are indeed using the default settings:

Global browser usage

Next, I went to Statcounter and grabbed the global browser numbers for July 1-5, and created a comparable graph:

WebPagetest versus global browser stats

Looking at both charts, there are some striking observations about these two sets of data:

  • While just over 13% of global internet users use IE8, roughly two-thirds of WebPagetests are done on IE8.
  • Chrome 20 is the dominant global browser, with a 23% share. IE9 is second, at 17%, and Firefox 13 is third, at almost 15%.
  • While WebPagetests are split almost equally between IE9 and IE7, this doesn’t mirror the purported real world data offered by Statcounter, which shows IE9 to be vastly more popular than IE7.
  • In fact, further to the above point, the number of Safari iPad users is almost double the number of IE7 users, at 2.52% and 1.38% respectively.

Here’s a graph that illustrates the differences more starkly:

Browser stats: WebPagetest use vs. global browser stats

You can see there’s a significant disconnect between the reported global browser usage and the browsers that people in our industry test on.

Based on this data, if we were to create an ideal set of testing options for WebPagetest, we would:

  • Add the capability to test on Chrome 20, and make this the default browser.
  • Add Firefox 13, Safari 5.1, and Safari iPad.
  • Question why so many of us are bothering to test on Firefox 6.

Or would we?

Some of these global usage numbers weren’t sitting well with me, because they don’t reflect the real-world data that I see every day here at Strangeloop. I gathered traffic data for the same time period, July 1-5, for a handful of Strangeloop customers using desktop browsers, representing approximately 350,000 unique visits. I broke the data down by browser version:

Out of curiosity, I compared these numbers to the data gathered by Akamai IO between July 1-5:

Akamai IO browser stats - July 1-5, 2012

While Akamai’s results aren’t identical to Strangeloop’s, they’re similar enough — specifically when it comes to the popularity of IE, Chrome, and Firefox — to validate my belief that both samples are fairly representative of typical internet users in our market.

WebPagetest versus Strangeloop browser stats

So I took Strangeloop’s browser usage numbers and overlaid them on WebPagetest’s numbers:

Browser stats: WebPagetest vs Strangeloop data

While the results don’t dovetail perfectly, you can see that IE8 usage correlates pretty decently. However, there’s a significant gap when it comes to IE9. And Chrome 20, Firefox 13, and Safari — though widely used according to both Strangeloop’s and Akamai’s data — are not yet on WebPagetest’s map.

Conclusion #1: IE8 is still relevant as the default test browser, but we should keep an eye toward using IE9 in the near future.

It would also be great if WebPagetest could add the ability to test for Chrome 20, Firefox 13, and Safari.

Conclusion #2: It could be an extremely useful feature if WebPagetest offered different defaults based on country, or even based on country plus user/application type.

This research raises some interesting new ideas. It’s pretty clear that Strangeloop and Akamai’s data is North American-centric with a bias toward ecommerce customers, and this no doubt accounts for some of the discrepancy noted above. This opens up the question of whether it would be useful to default to different browsers for different countries, and even to different browsers for different types of applications. For example, selecting “Ecommerce Dulles” would test on IE8, while a content-driven site in India should default to Chrome 20.

Takeaway: Know your users… and their browsers.

While this test validates that Internet Explorer 8 is a sound default for the mainstream North American ecommerce market, it may not be the most-used browser for your visitors.

One of our customers, Wine.com, discovered that their highest-spending customers work in the financial services sector, and they were placing orders from work, where they were restricted to IE6 and IE7, which had serious usability implications. And last year, L.L. Bean realized that 4-5% of their affluent older customers were still using IE6.

Before you spend a lot of time running tests, look into your analytics and learn who your users are.

Related posts: