I’ve been working on a number of interesting research projects lately. As part of my research, I have been conducting thousands of page tests with and without video (a Webpagetest beta feature, which I’ve written about here and here). Following a conversation with Patrick Meenan late last week — in which he warned that capturing a video might have a significant impact on the overall test results, thus making test results taken in conjunction with a video uncomparable to test results for the same URL without the video — I became concerned about the validity of my research.
On Friday, I had a casual conversation with my friend Tony Perkins, the creator of the Webpagetest monitoring tool I wrote about here, about this topic. Tony has his own private Webpagetest infrastructure and has been experimenting with different memory and CPU sizes. Over the weekend, he decided to test the impact that including video has on the gathering of performance numbers. Using an experimental instance, he ran just over 100 tests on the same script, with and without video capture.
Patrick’s warning was confirmed: There was a significant difference between the tests.
|Time to first byte (in seconds)||5.235||3.657|
What was the performance culprit?
After some discussion between Tony and Patrick (I was merely the fascinated eavesdropper), the problem ended up being, to a great extent, the environment in which Tony was running his tests: an EC2 Micro instance of Windows 2003 running in VA.* When you think about it, this isn’t all that surprising, given that Micro instances were designed as a low-cost alternative to Small instances for lower throughput applications and websites.
Tony tested the same setup with EC2 Small instances to see if the impact is less, and it was. The results were almost identical:
|Time to first byte (in seconds)||2.627||2.901|
Your CPU and available memory have an impact on your page test results. If you are setting up your own test systems, be aware of CPU and memory limitations.
As someone who loves capturing videos on Webpagetest, I have now changed my process and you should as well. I will no longer compare results taken with a video to those without a video. Even though Tony’s numbers above show comparable results when the machines have enough memory and CPU, I prefer to remain a cautious skeptic.
Patrick also offered this comment:
“I think it also turned out that Micro instances just aren’t good for testing with period, not just video. The numbers were all over the place even without video (and the impact was probably just because of the wild fluctuations). It does get to the point of having a “statistically significant” sample when you’re quoting performance numbers, though, and to understand what that number is on a given test environment.”
- If you are setting up your own test systems, beware of CPU and memory limitations.
- To be on the safe side, don’t compare video-capture results with non-video results.
- Make sure to use a statistically significant sample when quoting performance numbers.
- Know what that number is on a given test environment.
*Methodology: Using Webpagetest, Tony executed the same script at 15-minute intervals from an EC2 Micro instance of Windows 2003 running in VA. He gathered just over 100 points of data, which were first sorted by doc time and the 90th % captured. The resulting data was further sorted by fully loaded and the 90th % captured. This test was configured as a “multi step” test, meaning that the individual numbers for each step are added together.