A couple of sort-of-related items of note:
This week, Pat Meenan ran a series of tests to determine what impact, if any, video capture has on Webpagetest‘s ability to measure performance consistently. This was after a conversation (discussed here) with Tony Perkins, when Tony found that video capture skewed test results during an experimental Micro instance.
From Pat’s post:
One thing I’ve always been concerned about is people taking a single measurement point and using that as representative of the performance of their site. Even for a well-performing platform (where the back-end is consistent) there are a lot of variations and sometimes significant outliers (I usually recommend taking at least 3 measurements and throwing out any that look obviously wrong). […]
I have […] been asked quite a few times about the impact of capturing video on the actual test results (and I’ve generally steered people away from comparing video-capture results against non-video results). I jumped through a lot of hoops in Pagetest to minimize the impact and I measured it as “effectively 0” on my dev systems but never did any large-scale testing against the public test systems to see how they perform.
Pat ran 100 tests each of www.aol.com, www.amazon.com and www.yahoo.com, with and without video capture. He posted his findings (along with some very nice graphs) on his blog.
On a related note, earlier this week Tony released the latest version of the monitoring script he’s been working on for the past few months (blogged here). This script — now dubbed WPT Monitor 0.2.0 — is designed to sit in front of a private instance of Webpagetest and automatically test multiple landing pages at set intervals in perpetuity, and then aggregate the results. At any given time, the script lets you generate an easy-to-read chart that shows overall performance for one day or for any time span you specify. The script is available for download and discussion here on Webpagetest.
I want to take a moment to thank Pat and Tony for all their hard work in creating and refining these tools, and for their incredible generosity in sharing them with the rest of us. It’s not an overstatement to say that the launch of tools like these are watershed moments in our industry.