Our industry has been in desperate need of a solid tool that measures real-world mobile website performance. Blaze Mobile — which is based on the Webpagetest framework — may be just what we’ve been waiting for.
The tool is still in beta, and the folks at Blaze has a list of known issues they’re working on. It’s up to the rest of us to take it for a test drive, see how it performs, and offer our feedback.
I thought it would be interesting to start by testing the current top 5 sites in Keynote’s mobile commerce index and benchmark Blaze’s results alongside Keynote’s.
Blaze lets you test for iPhone and Android, so I tested on both those platforms.
You can choose between one, two, and three test runs per URL. I opted for three runs. The averages are below.
The tool also lets you capture video results, so I tested with and without that option. As discussed here, capturing video can have a negative impact on Webpagetest’s page load times. Blaze has stated that their tool has the same issue, so I wanted to see how pronounced the impact of video capture is.
Comparison: Keynote and Android
Comparison: Keynote and iPhone
What patterns emerged?
Even allowing for fluctuations caused at the network and delivery end of things, there are some interesting patterns:
- Across the board, Android load times were significantly faster than Keynote and iPhone load times. In the case of the Barnes & Noble site, the Android load time was about 400% faster.
- Keynote’s load times surprised me by being, overall, slower than Blaze’s. I had expected that, because Keynote takes their measurements at the highest possible network speeds available at the time of the tests, their results would be faster.
- Video capture dramatically slowed down iPhone results, in some cases making them almost twice as slow as load times tested without video capture. The effect of video capture on Android results was much slighter.
I found Blaze Mobile extremely usable. The UI is really clear, and considering the fact that it can only run one test at a time, I was impressed with how fast it was able to deliver results (though that could change if it catches on).
The test results pages are easy to read and include a waterfall and HAR file. Right now, the results page only shows you the load time and page size.
Suggestions for future development:
- It would be great to also see time to first byte and start render. Sure, you can find this on the waterfall, but waterfall interpretation isn’t in everyone’s skill set. (If you want to add it to yours, here’s a beginner’s guide to waterfall charts.)
- I also like Webpagetest’s option to view your test results as a filmstrip. This would be a good feature to see here as well.
- It would be good to be able to run side-by-side tests.
- I’d also like to be able to export the video file.
I like Blaze’s transparent methodology. Keynote provides a bit of this in a footnote on their mobile index page, but I’ve always hankered for more background from them. I’d love to know what devices and operating systems they’re testing with. With Blaze, I get a better sense, not just of how they run their tests, but also the inevitable caveats that come with interpreting the results. As a result, I feel like I can buy in to their results. They feel truthy.
My only major caveat
Until the video capture issue is resolved for the iPhone, I’d consider these results invalid. So make sure you de-select ‘Enable Video Capture’ when you run iPhone tests.
Other than that issue, I’m adding Blaze Mobile to my toolset, and looking forward to where they take it next.