Travelocity makes a good tool even better with new script for Webpagetest
12 Aug 2010
Chances are you already know that, in order to get a real sense of how your site performs, you need to test it using tools that simulate the real-world end-user experience. I talked about this a couple of weeks ago, and suggested Webpagetest as a very usable tool that delivers solid results.
After posting on this topic I got a lot of feedback, and one theme recurred. To paraphrase a customer conversation:
“It’s great to have one isolated test, but what we need is a continuous view into performance. I need hundreds of these results everyday — not just one.”
Anyone who has run dozens of tests on Webpagetest knows it is challenging/impossible to run and aggregate hundreds or thousands of web page test results. For instance, I took this picture yesterday as I was trying to put together a list of URLs for a customer that spanned multiple tests. At the height of the task, I was working in Notepad, Excel, and had dozens of WPT windows open.
I don’t think our community has a good answer to this problem. I know many of the metrics vendors are hard at work trying to simulate real-world scenarios or actually using real-world browsers, but these are fraught with challenges (a post that will be coming soon).
But I came across a cool innovation today that I wanted to share.
One of Strangeloop’s clients, Travelocity, has tackled this problem head on. Their Director of Development, Tony Perkins, created his own script that sits in front of his own instance of Webpagetest and automatically tests the same six landing pages every 10 minutes in perpetuity, then aggregates the results. At any given time, Tony can generate an easy-to-read chart that shows overall performance for one day or for any time span he specifies. He can even view side-by-side comparisons of how the test pages perform with and without Strangeloop’s Site Optimizer service. Pretty slick.
I say all of this not to be critical of Webpagetest, which is a fantastic tool (obviously, or else my company wouldn’t be a sponsor), and which has gotten even better with yesterday’s release of a new UI and some additional functionality. I say this to point out that front-end performance measurement tools that use real browsers and simulate the real world — particularly free, independent tools like WPT — are still in their nascency. Tony and Travelocity have proven that WPT is extensible, which is pretty exciting news for anyone who cares about fine-tuning tools so that they can get the exact data they need.
I asked Tony if he has any plans to make his script open source, and am happy to report that he does. I’ll pass along more info on this as it comes available.
Tweets that mention Travelocity makes a good tool even better with new script for Webpagetest — Web Performance Today -- Topsy.com
Aug 12, 2010 @ 13:36:18
[...] This post was mentioned on Twitter by Aaron Kulick, Aaron Kulick. Aaron Kulick said: RT @joshuabixby: New blog post: Travelocity makes a good tool even better with new script for #Webpagetest http://bit.ly/b3ECya [...]
Aug 12, 2010 @ 14:35:25
FYI, WebPagetest supports an automation API that you can build tools against: http://www.webperformancecentral.com/wiki/WebPagetest/Automating_Pagetest
I don’t encourage doing that against the public instance because you could easily bury the testing infrastructure but it’s a good starting point for automating a private instance (or if you are doing one-off batch jobs where you want to collect stats and pull them together automatically). I had to do this very recently where I wanted to test a set of urls from all of the test locations and automatically pull out the median run and the times.
I’d love to add support for the system itself to be able to repeat tests at a given interval and database the results (with a pretty reporting front-end). Tony’s work could serve as a great starting point for that (again, more valuable for private instances).
Travelocity makes a good tool even better with new script for … « Web Script Genie
Aug 13, 2010 @ 02:30:47
[...] posted here: Travelocity makes a good tool even better with new script for … Related Posts:Travelocity makes a good tool even better with new script for …Are these [...]
Aug 13, 2010 @ 08:08:08
What’d be nice is a tool that doesn’t just measure from a test server, but collects data from your real visitors. I’m aware of Boomerang, but that just seems like the nuts and bolts of the solution. Are you aware of any tools that would allow me to paste a snippet of code on my page and then I could login and see average page load times? Google Analytics isn’t specific enough…
How Important Is Transaction Monitoring?-business
Aug 15, 2010 @ 07:25:23
[...] Travelocity makes a good tool even better with new script for … [...]
Aug 16, 2010 @ 17:06:10
@Shane: Good question. You’ve prompted me to start doing some digging in this area. Stay tuned for the results!
Aug 16, 2010 @ 17:07:01
@Patrick: Thanks for the API tip. Good to know!
Setting up your own web page test systems? Beware of your CPU and memory limitations. — Web Performance Today
Mar 23, 2011 @ 16:20:07
[...] with my friend Tony Perkins, the creator of the Webpagetest monitoring tool I wrote about here, about this topic. Tony has his own private Webpagetest infrastructure and has been experimenting [...]