Above-the-fold time (AFT): Useful, but not yet a substitute for user-centric analysis
16 Mar 2011
It was great to see above-the-fold time (AFT) on the agenda at yesterday’s Velocity Online. It’s been pretty widely acknowledged, here and on other blogs, that load time and doc complete time don’t fully cut it as measurement numbers. Our industry needs a user-centered approach to measuring page performance, one that tells site owners when visitors are able to see and interact with a significant amount of page content. AFT is a promising concept.
Coming up with a universal algorithm for such a nuanced measurement — an algorithm that can be applied to any site to consistently and accurately measure performance — is a gargantuan undertaking. Pat Meenan and the rest of the gang at Google who are working on WebPagetest deserve huge respect for being the first to tackle this hairy challenge.
Defining “Above the Fold”
In yesterday’s session on above-the-fold time, led by Google’s Jake Brutlag, AFT was defined as the moment when content “above the fold” (aka “what you see in your browser window”) stops changing and reaches its final state.
Put in its simplest terms, WebPagetest’s AFT algorithm performs the following calculations:
- Differentiates between content and ads by classifying static pixels as content and dynamic pixels as ads
- Identifies static pixels as those that change less than 5 times within a defined cutoff point
- Determines AFT as the moment after the last change (within the above-mentioned cutoff) of a static pixel
Testing AFT on real live sites
I was interested in seeing how WebPagetest’s new AFT option performs, so I ran tests on the top 20 Alexa-ranked retail sites.*
| Website | Load time | Above the fold | Difference (%) |
| Amazon | 2.701 | 5.8 | 218% |
| eBay | 2.312 | n/a | n/a |
| Netflix | 5.172 | 8.2 | 159% |
| Amazon UK | 3.225 | 4.7 | 146% |
| Walmart | 4.883 | 6.4 | 131% |
| Etsy | 6.210 | 11.8 | 190% |
| BestBuy | 3.511 | 4.5 | 128% |
| Target | 5.031 | 3.9 | 78% |
| Wells Fargo | 2.061 | 2.9 | 141% |
| Fox Sports | 6.132 | n/a | n/a |
| Overstock | 3.036 | 2.9 | 96% |
| Home Depot | 24.175 | 26.4 | 109% |
| Barnes & Noble | 6.496 | n/a | n/a |
| Macys | 6.760 | 16.8 | 249% |
| Sears | 4.031 | 4.7 | 117% |
| Zappos | 7.683 | 8.3 | 108% |
| Ticketmaster | 8.566 | 17.6 | 205% |
| Costco | 4.720 | 5.8 | 123% |
| Bodybuilding | 8.857 | 8.8 | 99% |
| Auto Trader | 5.420 | 7.4 | 137% |
Interpreting the results
As Jake pointed out in yesterday’s session, WebPagetest’s AFT measurement often corresponds approximately with onload/document complete time or fully loaded time. You can see this in the page tests for Overstock.com and Bodybuilding.com.
In contrast, I’ve highlighted all the instances where AFT exceeds load time. While in many cases, the difference isn’t huge, in some cases, the difference is considerable.
Macys.com, for example, has a load time of 6.760s, and an AFT of 16.8s. If you take a look at the waterfall and filmstrip view for Macys, you can see that one of the reasons for this discrepancy is an animated graphic in the main promo banner, which rotates every 3 seconds.
The test results for Amazon also showed a huge variance: 2.701s load time and 5.8s AFT. Looking at the waterfall and filmstrip reveals an animated graphic to be one of the culprits.
Some optimization techniques will give inflated AFT measurements
I applaud the folks at WebPagetest for taking this brave first step in tackling above-the-fold time. The main problem that I see with the current iteration of the AFT algorithm is that it doesn’t fully take into account two effective performance optimization techniques that are growing in popularity:
- Deferral – An example of this is deferring objects like the Facebook “like” button to load last. It’s a great optimization technique because the button is a piece of third-party code that can slow down a page, and it’s not crucial to how a page functions. However, because it can appear above the fold, it can result in an AFT measurement that is out of line with how a visitor actually perceives the usability of a page’s content.
- Progressive rendering — Developers are being encouraged to use this technique, which delivers fast, low-quality images right away and then substitutes them with high-quality images at the end of the page load. Like deferral, this technique focuses on how a user perceives a page’s performance. But like deferral, this technique can give a deceptively slow AFT measurement.
In Jake’s presentation wrap-up, he was very clear about the current limitations of AFT:
- Only applicable to lab settings
- Does not reflect user-perceived latency based on functionality
- AFT heuristics always need further iteration
In other words, while AFT is useful as a visual validation of other metrics, it is not yet a substitute for understanding functional readiness — how users actually see and use a page.
*Tests conducted on IE9, via the WebPagetest server in Dulles, VA.
Mar 16, 2011 @ 13:45:46
Fascinating. I’m going to blog about this. AFT is impossible to measure without “custom timing events”. These are events where you can test at the microsecond level from “inside the browser”. We’ve already architected these kind of events into our mobile browser. For the first time you can see inside at what’s going on when the browser loads a page regardless of the optimization techniques you use.
Cheers,
Peter
Mar 17, 2011 @ 09:50:00
Great post. It’s true – this is a complex problem. It’s important to recognize this is different from page load time – so comparing them to see how similar they are is not the point – in fact that’s the opposite of what we want. (Your use of “variance” infers a discrepency – simply saying “difference” would be better – they are different!)
I disagree that the optimization techniques you site aren’t accounted for. Deferring 3rd party content is a great technique – but the motivation for doing this (you don’t control 3rd party content so you don’t want it blocking your main content) doesn’t mean that content isn’t meaningful for the page. AFT tries to approximate when the user would feel the page is ready. There’s no definite answer whether a Facebook like button, YouTube video, or recent Twitter updates list is main content on the page – the answer varies by website and user. The issue has less to do with deferral and is more about determining the “main content” on the page.
Similarly, progressive rendering is more about attaching (JavaScript) functionality to the page after the page has rendered – so there is no issue with AFT. I’m not aware of many sites doing a low res high res image switch – but again, this would get back to the definition of when the user perceives the page to be ready. I personally would be unsettled by low res images that got overlaid with new images, and would consider that a page that wasn’t ready. (Whereas, animated gifs or rotating ads don’t affect my perception of page readiness – so it’s nice that AFT handles those.)
I’m not saying there aren’t issues with AFT. Again – it’s a very complex problem. But we need to focus on the primary issue – defining a page’s “main content”. Macys.com is a great example – is that rotating banner are those rotating clothes something the user would wait for? I’m guessing no and hopefully we can iterate on AFT and find ways to improve it.
Mar 17, 2011 @ 16:17:01
Thanks for your comments, Steve. To respond:
Your use of “variance” infers a discrepency – simply saying “difference” would be better – they are different!
Good point. I probably used the word “variance” a bit too loosely, and “difference” would have been more accurate.
The issue has less to do with deferral and is more about determining the “main content” on the page.
Agreed. I think this point works with, rather than against, the points I raised, and it helps us get closer to addressing the kernel of the issue. Deferral — properly implemented, with sensitivity into what the majority of users want from the page – should reflect what is considered “main content” and what isn’t. If a tester could somehow refine their page test so that AFT recognizes the strategic use of deferral (for ads and other image related items for example), we’d get a more truly user-centric sense of AFT usefulness.
Re progressive rendering: We’re very excited about the correlations we’re seeing between progressive image rendering and key business metrics. I’d be happy to share some of our customer stories with you some time.
At the end of the day, I think we all agree that trying to approximate when a user will feel that a page is ready – in other words, marrying hard numbers with highly subjective perceptual ephemera – is an insanely tricky job. I can’t emphasize enough how awesome it is that WebPagetest has taken on this task, and how much I’m looking forward to watching AFT evolve.
Mar 21, 2011 @ 11:35:28
Four more things to consider:
1) AFT has to relate to all things web, which includes Mobile and all the different form factors
2) Some people are already talking about OnFocus as being the point when the page is usable (this can appear above the fold). Remember there is NO such thing as OnFocus in Androids Mobile browser
3) Scale. What if someone scales the entire page to fit in your Mobile browser screen. Then there’s no AFT.
4)Deferring items. This needs very careful consideration. There are already examples of web pages in the public domain that defer “items by 2 seconds” or more. The Mobile browser ignores those 2 seconds.
AFT is really tough to measure without exceptionally clear definitions.
Cheers,
Peter
Experimental new Webpagetest feature lets you test sites on Chrome — Web Performance Today
Mar 23, 2011 @ 15:13:26
[...] ← Above-the-fold time (AFT): Useful, but not yet a substitute for user-centric analysis [...]
Otimizações na Web e o carregamento assíncrono | blog.caelum.com.br
Apr 28, 2011 @ 06:17:18
[...] apenas quando o usuário fizer scroll para vê-las. Há inclusive quem defenda uma nova métrica, o Above The Fold (AFT) Time apresentado na Velocity Online 2011 mês passado, que leve em conta apenas os objetos necessários para a primeira impressão do [...]
May 09, 2011 @ 02:48:30
Very interesting Alexa ranking list. Proves what I have been thinking about Above the Fold. Thanks for post
Are you focusing on first and repeat views? Then you may be ignoring 96% of your page traffic. — Web Performance Today
May 10, 2011 @ 21:34:41
[...] Above-the-fold time (AFT): Useful, but not yet a substitute for user-centric analysis [...]
The Problem with the YSlow | Indolering
Jul 04, 2011 @ 21:48:38
[...] load time scoring (both Google’s search scoring and the page speed tools) do not measure perceptual load time. If the user can interact with the content, s/he will not care if the footer or some fancy JS [...]