Earlier today, I had an exciting opportunity to present some of the findings of Strangeloop’s newest study, the 2012 State of Mobile Ecommerce Performance at Velocity EU. I’ll be posting those slides shortly, but first want to take a moment to talk about this study and why we’re so excited about it.
Why test over cellular networks?
Last spring, we were sitting in the office talking about mobile performance measurement and why it’s so hard to get reliable real-world numbers, especially over cellular networks. This seemed like a glaring knowledge gap, but it’s an understandable gap given the massive variability in network performance. Last year, I did an informal latency survey and found that latency can fluctuate wildly, even in the same location at the same time.
Methodology: How to DIY your own RUM tests for mobile
So we set out to develop a reliable methodology for measuring mobile performance over cellular networks, which takes this variability into account. We also wanted to create a methodology that anyone could recreate fairly easily.
Here’s how we did it:
- We focused on 200 leading retail sites, as ranked by Alexa.com.
- We selected six testing devices: iPhone 4, iPhone 5, Samsung Galaxy S smartphone, Samsung Galaxy S3 smartphone, iPad 2, and Samsung Galaxy tablet.
- iPhone 4, Galaxy S, iPad 2 and Galaxy tablet were tested over 3G.
- iPhone 5 and Galaxy S3 were tested over LTE.
- We tested each site’s home page 30 times per device, using the device’s native browser. By testing this many times and capturing the median result, we hoped to take into account latency variability.
- For all tests, the devices were positioned in the same location, in an attempt to mitigate the latency impact caused by location changes.
- For all tests, wifi was turned off, devices and radios were at full power, and screens were not allowed to lock during testing.
- The tests in this study were conducted using a RUM beacon inserted into the page that captured the onload time. The cache and cookies were cleared automatically between each test.
As I shared at Velocity, the results were a mix of the predictable and the surprising:
Predictable: Pages are slow over 3G.
It’s not that shocking to learn that the median pages took 11 or more seconds to load over 3G.
Surprising: A significant number of pages took 20+ seconds to load.
While we expected pages to be slow, we didn’t expect that 9% of the pages we tested took more than 20 seconds to load on the iPhone.
Predictable: LTE was faster than 3G.
No explanation needed here.
Surprising: LTE was only 27% faster than 3G.
I’ve heard claims that LTE is, on average, ten times faster than 3G. Our results suggest that LTE performance gains might be more unpredictable than this.
Predictable: Pages loaded 22% faster on the iPad than on the Galaxy tablet.
iPad fans might scoff that we even bothered to test this.
Surprising: The Galaxy S3 phone loaded pages 9% faster than the iPhone 5.
It’ll be interesting to hear what iPhone fans make of this. (Full disclosure: I have an iPhone 5 and have to admit I was kind of affronted to see these results.)