Is it time to give up Internet Explorer 7 as our default test scenario?

Here at Strangeloop, there’s been a lot of debate lately about what our default browser should be when we run performance tests through Webpagetest. As discussed here previously, we had a very strong argument for using Internet Explorer 7. We’re not ready to abandon it yet, but the idea is under the microscope.

Equally important, we’ve also been debating if the default bandwidth — these days, we test using the DSL setting — is accurate.

Akamai came out with its state of the internet report yesterday, which has made me think about this issue some more.

From Akamai’s report:

“The overall average connection speed for the U.S. as a whole in the third quarter of 2010 was 5.0 Mbps. Delaware continued to maintain its standing as the state with the fastest average connection speed. The overall average peak connection speed in the U.S. during the third quarter was 20 Mbps.

“In looking at high broadband adoption in the U.S. during the third quarter, trending was mostly positive. Quarterly increases in high broadband adoption of 10% or more were seen in 23 states and the District of Columbia, with New Mexico topping the list at 60% growth. In reviewing year-over-year changes in U.S. broadband adoption, four states (Alaska, Minnesota, Montana, and Alabama) grew more than 100% year-over-year, with Alaska’s massive 191% growth leading the way.”

After reading this, I wondered about a few things:

  • First and foremost, I wondered how many people use the default settings. Maybe people are using their own custom settings, so the defaults are not actually a huge issue.
  • What would an experiment look like if we if loaded the Akamai site using different bandwidths and latencies?
  • What would a graph look like in Webpagetest for the same site if I worked my way up from 1.5Mbs (the standard WPT setting for DSL) up to 5.5 Mbs?
  • What would happen if I tested variance in latency as well — say, comparing 15ms to 50ms?
  • While I was at it, what about testing all these parameters in IE8?

How many people use Webpagetest’s default settings?

I needed to address this question first, to validate looking into the rest of my questions. To look at the use of default settings, I simply looked at the last 8,847 public tests (the sum of one day’s testing).

As these graphs show, most users are indeed using the default settings (IE7 and DSL):

Default browsers used for page speed tests

Default connections used for page speed tests

With this established, I was ready to move on to the rest of my questions. Here’s what I found, this time displayed in handy line graph form.

Four observations

First, a couple of caveats. We only ran three tests per site, then used the median results. And it goes without saying that this is only one website; all sites will perform differently, so your mileage may vary.

  1. More bandwidth does not have a significant impact on load time when we test using the current IE7 default, and has almost no impact on start render. Mike Belshe wrote about this last May, and this exercise seems to confirm his findings.
  2. When it comes to latency, we see the same pattern in IE7 that Mike did: when we decrease round trip time, we improve performance.
  3. But unlike Mike’s experiment, we see no pattern in IE8 that links latency and performance when it comes to load time; however, we do see a strong correlation in start render.
  4. Most important, as the side-by-side videos below demonstrate, when we look at perceived load time by watching the site through a real user’s eyes, we see very little difference in the user experience as we increase bandwidth or change latency in IE7 or IE8.

Test results with IE7, 15ms latency:

Test results with IE7, 50ms latency:

Test results with IE8, 15ms latency:

Test results with IE8, 50ms latency:

Conclusions

Webpagetest’s default testing scenario of IE7 with DSL (50ms latency and 1.5mbs) probably needs to be updated. We should probably move to IE8, 2Mbs and 30ms of latency (this is just my hunch) to ensure that our tests reflect the average experience of real visitors.

What do you think? Are there any patterns I’ve missed? What are your thoughts about the ideal testing scenario?

Final note

Circling back to the point that I mentioned at the top of this post: I’m not advocating for overnight change. As one of the companies that hosts Webpagetest, I know that a move to IE8 is not necessarily easy. I raise these questions as fuel for debate.

Edited to add:

Pulling this out of the comments: Patrick at Webpagetest wants to know what you think the new default browser should be. Chime in on the discussion boards.

Related posts:

5 thoughts on “Is it time to give up Internet Explorer 7 as our default test scenario?

  1. Pingback: Tweets that mention Is it time to give up Internet Explorer 7 as our default test scenario? — Web Performance Today -- Topsy.com

  2. Make sure to vote on what you’d prefer the default browser to be here: http://www.webpagetest.org/forums/showthread.php?tid=431

    I know of at least one other user that agrees :-)

    I don’t think the graphs match your conclusions though. For this one site, in all cases except for IE 7 50ms it looks like bandwidth does significantly improve load time.

    What I’d really like to see before making any decisions though are the percentile distributions (and market share). It would suck to target an “average” to the detriment of 40% of your customers for example and generally if you shoot for the lower speeds and browsers things will not be slower with the newer browsers and faster connectivity (while the opposite is not always the case).

  3. Thanks for pointing that out, Patrick. I *meant* to say: “More bandwidth does not have a significant impact on load time when we test using the current IE7 default.” Duly clarified in my post.

    It’s interesting, though, that even with other browser settings, there are diminishing rates of return and the graph pretty much flattens at 3.5 seconds.

  4. Pingback: Announcing the Strangeloop Speed:Revenue Calculator — Web Performance Today

  5. Pingback: Early findings: 97% of mobile end-user response time happens at the front end — Web Performance Today

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>