Chances are you already know that, in order to get a real sense of how your site performs, you need to test it using tools that simulate the real-world end-user experience. I talked about this a couple of weeks ago, and suggested Webpagetest as a very usable tool that delivers solid results.
After posting on this topic I got a lot of feedback, and one theme recurred. To paraphrase a customer conversation:
“It’s great to have one isolated test, but what we need is a continuous view into performance. I need hundreds of these results everyday — not just one.”
Anyone who has run dozens of tests on Webpagetest knows it is challenging/impossible to run and aggregate hundreds or thousands of web page test results. For instance, I took this picture yesterday as I was trying to put together a list of URLs for a customer that spanned multiple tests. At the height of the task, I was working in Notepad, Excel, and had dozens of WPT windows open.
I don’t think our community has a good answer to this problem. I know many of the metrics vendors are hard at work trying to simulate real-world scenarios or actually using real-world browsers, but these are fraught with challenges (a post that will be coming soon).
But I came across a cool innovation today that I wanted to share.
One of Strangeloop’s clients, Travelocity, has tackled this problem head on. Their Director of Development, Tony Perkins, created his own script that sits in front of his own instance of Webpagetest and automatically tests the same six landing pages every 10 minutes in perpetuity, then aggregates the results. At any given time, Tony can generate an easy-to-read chart that shows overall performance for one day or for any time span he specifies. He can even view side-by-side comparisons of how the test pages perform with and without Strangeloop’s Site Optimizer service. Pretty slick.
I say all of this not to be critical of Webpagetest, which is a fantastic tool (obviously, or else my company wouldn’t be a sponsor), and which has gotten even better with yesterday’s release of a new UI and some additional functionality. I say this to point out that front-end performance measurement tools that use real browsers and simulate the real world — particularly free, independent tools like WPT — are still in their nascency. Tony and Travelocity have proven that WPT is extensible, which is pretty exciting news for anyone who cares about fine-tuning tools so that they can get the exact data they need.
I asked Tony if he has any plans to make his script open source, and am happy to report that he does. I’ll pass along more info on this as it comes available.