Here at Strangeloop, there’s been a lot of debate lately about what our default browser should be when we run performance tests through Webpagetest. As discussed here previously, we had a very strong argument for using Internet Explorer 7. We’re not ready to abandon it yet, but the idea is under the microscope.
Equally important, we’ve also been debating if the default bandwidth — these days, we test using the DSL setting — is accurate.
Akamai came out with its state of the internet report yesterday, which has made me think about this issue some more.
From Akamai’s report:
“The overall average connection speed for the U.S. as a whole in the third quarter of 2010 was 5.0 Mbps. Delaware continued to maintain its standing as the state with the fastest average connection speed. The overall average peak connection speed in the U.S. during the third quarter was 20 Mbps.
“In looking at high broadband adoption in the U.S. during the third quarter, trending was mostly positive. Quarterly increases in high broadband adoption of 10% or more were seen in 23 states and the District of Columbia, with New Mexico topping the list at 60% growth. In reviewing year-over-year changes in U.S. broadband adoption, four states (Alaska, Minnesota, Montana, and Alabama) grew more than 100% year-over-year, with Alaska’s massive 191% growth leading the way.”
After reading this, I wondered about a few things:
- First and foremost, I wondered how many people use the default settings. Maybe people are using their own custom settings, so the defaults are not actually a huge issue.
- What would an experiment look like if we if loaded the Akamai site using different bandwidths and latencies?
- What would a graph look like in Webpagetest for the same site if I worked my way up from 1.5Mbs (the standard WPT setting for DSL) up to 5.5 Mbs?
- What would happen if I tested variance in latency as well — say, comparing 15ms to 50ms?
- While I was at it, what about testing all these parameters in IE8?
How many people use Webpagetest’s default settings?
I needed to address this question first, to validate looking into the rest of my questions. To look at the use of default settings, I simply looked at the last 8,847 public tests (the sum of one day’s testing).
As these graphs show, most users are indeed using the default settings (IE7 and DSL):
With this established, I was ready to move on to the rest of my questions. Here’s what I found, this time displayed in handy line graph form.
First, a couple of caveats. We only ran three tests per site, then used the median results. And it goes without saying that this is only one website; all sites will perform differently, so your mileage may vary.
- More bandwidth does not have a significant impact on load time when we test using the current IE7 default, and has almost no impact on start render. Mike Belshe wrote about this last May, and this exercise seems to confirm his findings.
- When it comes to latency, we see the same pattern in IE7 that Mike did: when we decrease round trip time, we improve performance.
- But unlike Mike’s experiment, we see no pattern in IE8 that links latency and performance when it comes to load time; however, we do see a strong correlation in start render.
- Most important, as the side-by-side videos below demonstrate, when we look at perceived load time by watching the site through a real user’s eyes, we see very little difference in the user experience as we increase bandwidth or change latency in IE7 or IE8.
Test results with IE7, 15ms latency:
Test results with IE7, 50ms latency:
Test results with IE8, 15ms latency:
Test results with IE8, 50ms latency:
Webpagetest’s default testing scenario of IE7 with DSL (50ms latency and 1.5mbs) probably needs to be updated. We should probably move to IE8, 2Mbs and 30ms of latency (this is just my hunch) to ensure that our tests reflect the average experience of real visitors.
What do you think? Are there any patterns I’ve missed? What are your thoughts about the ideal testing scenario?
Circling back to the point that I mentioned at the top of this post: I’m not advocating for overnight change. As one of the companies that hosts Webpagetest, I know that a move to IE8 is not necessarily easy. I raise these questions as fuel for debate.
Edited to add:
Pulling this out of the comments: Patrick at Webpagetest wants to know what you think the new default browser should be. Chime in on the discussion boards.