8 May 2012
For years, we in the performance community have been preaching that site owners need to use real end user monitoring (RUM) tools or tools like Webpagetest.org to get a real-world picture of performance. I’m happy to report that our efforts are paying off. I’m now seeing many top-tier customers using RUM from companies like Compuware and New Relic or implementing private instances of WebPagetest.
But there’s a new problem: Now that we’re seeing wide-scale adoption of front-end optimization best practices, misapplication of these best practices could be delivering “false positives” — sites that test well but look bad for real users.
There are a lot of ways to optimize a page at the front end, and while some of these techniques can give you great-looking RUM and WebPagetest results on paper, they can also mess with how your page actually looks and performs. These days, I’m seeing more and more sites that test well but perform poorly.
I see the same pair of issues over and over. When you’re reviewing your RUM or WebPagetest results, keep an eye on these two things to ensure that how your site looks on paper matches how it looks in the real world:
1. Find out what is being deferred, and make sure that this level of deferral works for your business.
One of the things I’m seeing more and more is something called “blanket deferral”, in which almost every page resource is deemed a candidate for deferral after the onload event. Moving things after the onload event is a great technique for loading useful content first, but it has to be used strategically. If you move everything after the onload event, then your load time numbers become meaningless.
I recently had the people at an ad-driven site tell me they were shown a 50% performance gain through FEO, but when they reviewed the results, the ads didn’t show up until 2 seconds after onload. For some types of sites that level of deferral might work, but for these guys it would’ve meant a huge drop in clicks and revenue.
The best way to see if your test results are based on blanket deferral is to capture a video, using a tool like WebPagetest, to confirm that the onload event actually corresponds to how the site loads. After you capture the video (it’s easy to do: here’s a quick how-to), then scroll across the frames. This will also scroll across the waterfall, like this:
Cross referencing the waterfall and the pictures will help you sniff out tests that just don’t feel right. When you’re looking at them, ask yourself:
- Are key sections blank when I hit the blue line?
- Does something actually render at the red line?
If the answer to either of these questions is “no”, then your test may be showing you that someone has implemented blanket deferral on this page.
2. How do your pages look within a flow?
Up to 96% of your page views will not be seen as either a first view or repeat view, as characterized by WebPagetest. Most of the page traffic on your site is “flow” traffic, in other words, pages that are seen after the visitor has already visited at least one other page on your site. If you’re focusing solely on how your pages look for first-time and repeat views, you’re not getting a realistic sense of how your pages load for most of your traffic.
Take, for example, a product page on a typical ecommerce site. That page may be accessed directly from a search engine, but in many cases that page will be accessed after first visiting at least one or two pages on your site.
Why do you need to care about flow traffic? Because badly implemented FEO can slow down a flow.
How to measure flow timing via RUM: Try to extract a graph like this to show you how fast pages are based on the session page:
How to measure flow timing via WebPagetest: Using WebPagetest, run a script to mimic a typical flow. Go to the site and click on the “Script” tab under “Advanced Settings”:
Enter a script to navigate to the page in question — something like this:
Either of these tests will give you a more insightful look into how your pages perform for most visitors than a simple test that shows “first view” and “repeat view” results.
One of the main tenets of our industry is that we can only manage what we measure. Another main tenet is that real-user testing is crucial, because the entire point of what we’re doing is to deliver the best possible online experience.
We need to make sure that we’re not gaming our measurements — however unintentionally — by some of the very front-end optimization techniques we tout. If we’re optimizing pages to make them faster but ultimately delivering a poorer user experience, then we’re not doing anybody a service.