In defence of Blaze

I follow a lot of blogs, but I rarely read comments, especially on contentious posts, because people can quickly lose all sense of decency and get nasty. A recent example of this: a couple of weeks ago, Blaze released its study of mobile browser performance — announcing that the iPhone browser is 52% slower than Android — and came under heavy fire from many people.

As it turned out, some errors were made in this study. I won’t get into the particulars here (Blaze has been forthright about it in their own post), but these errors triggered a flood of vitriol, which I think the folks at Blaze have handled exceptionally well.

Measuring real-world web performance is a very, very difficult task. There are countless variables involved, and these variables are in a state of constant flux. I’m not saying this to issue a “get out of jail free” card for any company that takes on an ambitious performance study.  However, I am saying this to point out that this is why there are so few ambitious web performance studies. But here’s the thing: our industry needs these ambitious studies. We need to take studies like Blaze’s for what they are: a first step in our strategy for gathering data.

As a community, I think we need to stand up and support the fact that companies like Blaze make the effort to put studies like this together, providing the opportunity for the scientific process to take hold. In their original post, they spelled out their methodology quite clearly, thus making it easy for people like you and me to replicate their study – and if we choose, to start with a different hypothesis and prove Blaze’s hypothesis wrong.

I don’t believe that the people at Blaze had any intention of misrepresenting the truth, as they’ve been accused of by some. They’ve admitted their mistakes, corrected their post, and broadcast their corrections through the same social media channels they used to broadcast their original study. This deserves respect.

Reading the negative comments and press directed at Blaze makes me a bit skittish about releasing data myself. I’m sure I’m not alone in feeling this way. Our industry has come a long way in just a few years. We need an open and collaborative environment in order for us all to continue to make forward progress.

Related posts:

• Review: Blaze Mobile performance measurement tool
• How to perform a 5-minute page speed/revenue analysis of your ecommerce site
• Is it time to give up Internet Explorer 7 as our default test scenario?
  By: Tammy Everts Category: Page speed tests, Tools

Above-the-fold time (AFT): Useful, but not yet a substitute for user-centric analysis

It was great to see above-the-fold time (AFT) on the agenda at yesterday’s Velocity Online. It’s been pretty widely acknowledged, here and on other blogs, that load time and doc complete time don’t fully cut it as measurement numbers. Our industry needs a user-centered approach to measuring page performance, one that tells site owners when visitors are able to see and interact with a significant amount of page content. AFT is a promising concept.

Coming up with a universal algorithm for such a nuanced measurement — an algorithm that can be applied to any site to consistently and accurately measure performance — is a gargantuan undertaking. Pat Meenan and the rest of the gang at Google who are working on WebPagetest deserve huge respect for being the first to tackle this hairy challenge.

Defining “Above the Fold”

In yesterday’s session on above-the-fold time, led by Google’s Jake Brutlag, AFT was defined as the moment when content “above the fold” (aka “what you see in your browser window”) stops changing and reaches its final state.

Put in its simplest terms, WebPagetest’s AFT algorithm performs the following calculations:

• Differentiates between content and ads by classifying static pixels as content and dynamic pixels as ads
• Identifies static pixels as those that change less than 5 times within a defined cutoff point
• Determines AFT as the moment after the last change (within the above-mentioned cutoff) of a static pixel

Testing AFT on real live sites

I was interested in seeing how WebPagetest’s new AFT option performs, so I ran tests on the top 20 Alexa-ranked retail sites.*

Interpreting the results

As Jake pointed out in yesterday’s session, WebPagetest’s AFT measurement often corresponds approximately with onload/document complete time or fully loaded time. You can see this in the page tests for Overstock.com and Bodybuilding.com.

In contrast, I’ve highlighted all the instances where AFT exceeds load time. While in many cases, the difference isn’t huge, in some cases, the difference is considerable.

Macys.com, for example, has a load time of 6.760s, and an AFT of 16.8s. If you take a look at the waterfall and filmstrip view for Macys, you can see that one of the reasons for this discrepancy is an animated graphic in the main promo banner, which rotates every 3 seconds.

The test results for Amazon also showed a huge variance: 2.701s load time and 5.8s AFT. Looking at the waterfall and filmstrip reveals an animated graphic to be one of the culprits.

Some optimization techniques will give inflated AFT measurements

I applaud the folks at WebPagetest for taking this brave first step in tackling above-the-fold time. The main problem that I see with the current iteration of the AFT algorithm is that it doesn’t fully take into account two effective performance optimization techniques that are growing in popularity:

• Deferral – An example of this is deferring objects like the Facebook “like” button to load last. It’s a great optimization technique because the button is a piece of third-party code that can slow down a page, and it’s not crucial to how a page functions. However, because it can appear above the fold, it can result in an AFT measurement that is out of line with how a visitor actually perceives the usability of a page’s content.
• Progressive rendering — Developers are being encouraged to use this technique, which delivers fast, low-quality images right away and then substitutes them with high-quality images at the end of the page load. Like deferral, this technique focuses on how a user perceives a page’s performance. But like deferral, this technique can give a deceptively slow AFT measurement.

In Jake’s presentation wrap-up, he was very clear about the current limitations of AFT:

• Only applicable to lab settings
• Does not reflect user-perceived latency based on functionality
• AFT heuristics always need further iteration

In other words, while AFT is useful as a visual validation of other metrics, it is not yet a substitute for understanding functional readiness — how users actually see and use a page.

*Tests conducted on IE9, via the WebPagetest server in Dulles, VA.

Related posts:

• Review: Blaze mobile performance measurement tool
• Front-end performance optimization: It isn’t over until it’s over. And it’s never over.
• New data and development at Webpagetest