Is it time to give up Internet Explorer 8 as our default test browser?

If you’re a regular user of WebPagetest, you depend on it to give you a reliable sense of how fast pages load across a variety of browsers and simulated latencies. I’m a huge fan and applaud Pat Meenan and the rest of his team in their efforts to maintain its relevance in the face of constantly shifting browser trends.

Knowing how responsive the WebPagetest team is to changing demands, I recently started wondering about why WebPagetest still uses Internet Explorer 8 as its default browser, when it’s commonly believed that Chrome is more widely used and Firefox comes in at a respectable second place. I also wondered if there are new gaps in terms of which browsers we should be testing on. So I started to do a bit of digging.

WebPagetest browsers

Why is this an important question?

While modern browsers are more similar than older browsers when it comes to embracing common standards, all browsers are still not created equal. As I’ve talked about in the past, as web pages become increasingly complex, data-intensive, and dynamic, front-end optimization (FEO) techniques that can make pages faster in some browsers can slow them down, or even break them, in another browser.

If you’re focusing your testing efforts on a single browser, such as IE8, you might just be getting a pinhole view of your site’s performance. And if the majority of your visitors use Chrome and Firefox, you might be getting an irrelevant pinhole view of your site’s performance.

How many people use Webpagetest’s default settings?

But before spending too much time contemplating the question of whether or not IE8 should be the default test browser, first I wanted to establish that this is a relevant question. If most people are using their own custom settings on WebPagetest, the defaults aren’t a huge issue.

To examine the use of default settings, last Friday afternoon I looked at roughly 24,000 public tests on WebPagetest (the sum of a half day’s testing for that day).

As this graph shows, most users are indeed using the default settings:

Global browser usage

Next, I went to Statcounter and grabbed the global browser numbers for July 1-5, and created a comparable graph:

WebPagetest versus global browser stats

Looking at both charts, there are some striking observations about these two sets of data:

  • While just over 13% of global internet users use IE8, roughly two-thirds of WebPagetests are done on IE8.
  • Chrome 20 is the dominant global browser, with a 23% share. IE9 is second, at 17%, and Firefox 13 is third, at almost 15%.
  • While WebPagetests are split almost equally between IE9 and IE7, this doesn’t mirror the purported real world data offered by Statcounter, which shows IE9 to be vastly more popular than IE7.
  • In fact, further to the above point, the number of Safari iPad users is almost double the number of IE7 users, at 2.52% and 1.38% respectively.

Here’s a graph that illustrates the differences more starkly:

Browser stats: WebPagetest use vs. global browser stats

You can see there’s a significant disconnect between the reported global browser usage and the browsers that people in our industry test on.

Based on this data, if we were to create an ideal set of testing options for WebPagetest, we would:

  • Add the capability to test on Chrome 20, and make this the default browser.
  • Add Firefox 13, Safari 5.1, and Safari iPad.
  • Question why so many of us are bothering to test on Firefox 6.

Or would we?

Some of these global usage numbers weren’t sitting well with me, because they don’t reflect the real-world data that I see every day here at Strangeloop. I gathered traffic data for the same time period, July 1-5, for a handful of Strangeloop customers using desktop browsers, representing approximately 350,000 unique visits. I broke the data down by browser version:

Out of curiosity, I compared these numbers to the data gathered by Akamai IO between July 1-5:

Akamai IO browser stats - July 1-5, 2012

While Akamai’s results aren’t identical to Strangeloop’s, they’re similar enough — specifically when it comes to the popularity of IE, Chrome, and Firefox — to validate my belief that both samples are fairly representative of typical internet users in our market.

WebPagetest versus Strangeloop browser stats

So I took Strangeloop’s browser usage numbers and overlaid them on WebPagetest’s numbers:

Browser stats: WebPagetest vs Strangeloop data

While the results don’t dovetail perfectly, you can see that IE8 usage correlates pretty decently. However, there’s a significant gap when it comes to IE9. And Chrome 20, Firefox 13, and Safari — though widely used according to both Strangeloop’s and Akamai’s data — are not yet on WebPagetest’s map.

Conclusion #1: IE8 is still relevant as the default test browser, but we should keep an eye toward using IE9 in the near future.

It would also be great if WebPagetest could add the ability to test for Chrome 20, Firefox 13, and Safari.

Conclusion #2: It could be an extremely useful feature if WebPagetest offered different defaults based on country, or even based on country plus user/application type.

This research raises some interesting new ideas. It’s pretty clear that Strangeloop and Akamai’s data is North American-centric with a bias toward ecommerce customers, and this no doubt accounts for some of the discrepancy noted above. This opens up the question of whether it would be useful to default to different browsers for different countries, and even to different browsers for different types of applications. For example, selecting “Ecommerce Dulles” would test on IE8, while a content-driven site in India should default to Chrome 20.

Takeaway: Know your users… and their browsers.

While this test validates that Internet Explorer 8 is a sound default for the mainstream North American ecommerce market, it may not be the most-used browser for your visitors.

One of our customers, Wine.com, discovered that their highest-spending customers work in the financial services sector, and they were placing orders from work, where they were restricted to IE6 and IE7, which had serious usability implications. And last year, L.L. Bean realized that 4-5% of their affluent older customers were still using IE6.

Before you spend a lot of time running tests, look into your analytics and learn who your users are.

Related posts:

9 thoughts on “Is it time to give up Internet Explorer 8 as our default test browser?

  1. I would be really interested to see a little more granularity in both the Akamai IO and Strangeloop data that shows browser usage by time of day and day of week. While it doesn’t change the aggregate results, it may indicate the type of browser that people choose to use, rather than the browser they are told to use.

    I also agree that country level, or at least global region, metrics similar to the StatCounter data would help with browser targeting. This could be made a “feature” of WebPageTest, and could be clearly stated when a visitor goes to run a test.

  2. Stephen:
    Being able to show browser usage more granularly (such as day of week or time of day) is on our to-do list for Akamai IO. We’re also working on enabling users to break the data out by geography.
    –David Belson (a member of the team behind Akamai IO)

  3. Great write-up, thanks for putting it together.

    I think your “Firefox 6″ numbers from WebPagetest are actually Firefox 3.6 and it was probably a parsing error.

    Chrome and Firefox on WebPagetest track the stable branches of both and are usually updated within a day of the update being available (so if you test right now you will actually be using Chrome 20 and Firefox 13).

    For the defaults I usually try to shoot for the lower end of the popular browsers under the assumption that if you at least make that not suck then the more modern browsers will tend to do better (rather than leaving a really big blind spot) but testing the relevant browsers for a given site is certainly important.

    Different defaults for different test locations is pretty easy but I think you’ll find that most people just use the default location too ;-)

  4. Thanks for another great write up guys!

    I agree with Pat’s approach of shooting for the lower end browser by default. I tend to favor the approach of optimizing for the worst offender that still has significant market share (in other words, IE previous gen :)).
    I’m blown away by the number of Safari users in your sample. That’s incredibly high based on what I’ve seen with other sample sets – to your point, it’s important to know your users and this is a GREAT example.
    @Stephen – I agree that looking at other dimensions is extremely telling. I’ve seen very interesting patterns looking at browser utilization by geography as well as time of day/week. If you look at the slides Aaron Kulick and I posted a while back (http://minus.com/msM8y8nyh/2e) your point is illustrated in slide 12 which shows the utilization of Chrome, Safari jumping up on the weekends.

  5. Joshua, what a great article. Now I’m feeling really good about our decision, taken several months ago, to make the default browser used by Keynote for monitoring the performance of our customers’ websites IE9. At that time, we were asked if we were too early with IE9. At this point, it feels like the right choice. I’m glad we do Firefox as well, and wish we had Chrome monitoring covered as well, but… I’m on it! Awesome post, I’ll retweet.

  6. Pingback: Community News: How to Survive the Ops Weekend from Hell | New Relic blog

  7. In my company browsers market share differs quite a lot from region to region and client to client, for example Chrome is very popular in Russia 35%, IE8 has 5% same IE9. Another client in Switzerland has 25% for IE8 and 15 for IE9, the rest is Safari, Firefox, Chrome. Different clients/regions tell different stories.

    I wonder what’s the reason for using DSL as main connection type in WPT, similar analysis would very interesting.

  8. Pingback: New findings: How does browser usage vary throughout the day and week?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>