Advanced Mobile Optimization: How does it work? How do we measure success? [slides]

27 Jan 2012

0 Comment

It’s been a busy couple of weeks, but I finally got around to posting the slides from my talk about advanced mobile optimization at the San Francisco & Silicon Valley Web Performance Meetup.

Advanced Mobile Optimization: How does it work? How do we measure success?
View more presentations from Strangeloop

I always enjoy coming to these Meetups, and this time was no exception. Thanks again to Aaron Kulick for inviting me, to LinkedIn for hosting, and to the extremely keen and knowledgeable crowd who turned out. :)

Related posts:

By: Josh Bixby Category: Mobile Tags: , , , , ,

New report: What we’ve learned from two years of watching the top 2,000 e-commerce websites

26 Jan 2012

2 Comments

Yesterday was a busy day for us here at Strangeloop. We released our second annual “state of the union” report on the page speed and performance of 2,000 leading e-commerce websites.

Infographics: 2012 State of the Union for E-Commerce Page Speed and Website Performance

We came up with the idea of conducting these reports over a year ago, when we realized there was no way to track performance changes and trends — from a real user’s perspective — on an ongoing basis. As I mentioned to CNet’s Stephen Shankland, when we talked about the report the other day, we can’t rely on benchmarking tests run out of data centers to give us any relevant sense of how our sites are actually performing out in the real world. They’ve got ridiculous bandwidth, and they’re parked right next to the content delivery machines, so they’ve got zero latency. By using WebPagetest — which simulates how fast a site loads in a real user’s browser — to measure the performance of the same set of 2,000 sites year after year, we can compare real-world performance snapshots and see what’s changed, and what’s caused the changes.

I don’t want to give away all our findings, the highlights of which are illustrated in the infographics above, but a couple of interesting things we discovered:

  • Pages have gotten bigger – from an average of 86 requests last year to 98 requests this year. This kind of growth is consistent with trends identified via the HTTP Archive.
  • Repeat views are a whopping 20% slower than they were last year, probably due to the the number of page objects.
  • Page Speed scores have gotten significantly worse, from 83% last year to 75% now. I was just talking to Pat Meenan, who created WebPagetest, and he pointed out that a probable reason for the drop is because the newest version of Page Speed checks for more optimizations, which has resulted in lower scores across the board. (Side note: It’s important to know that Page Speed doesn’t take into account advanced front-end content optimization techniques — nor could it, because there are way too many of them to track and measure. As a for-instance, we’ve had experiences here at Strangeloop where we’ve taken a page with a perfect Page Speed score, accelerated it with Site Optimizer, cut load time in half, and ended up with a new Page Speed score of 74%.)

While the findings are a mixed bag of good and bad news, to me the most exciting takeaway is the  attention this report has received from major media outlets like CBS News. It goes to confirm what became evident over Black Friday: site speed has become a mainstream issue.

Download the report: 2012 State of the Union: E-Commerce Page Speed and Website Performance

Download a high-res version of the infographics above (feel free to re-post): Poster: E-Commerce Page Speed and Website Performance (2012)

Related links:

By: Josh Bixby Category: Ecommerce, Research, Standards and benchmarks Tags: , , , , , , , , , , , ,

Interesting new findings about page views, time on site, and bounce rate across desktop and mobile browsers

20 Jan 2012

0 Comment

Last month, I talked with Mac Slocum at O’Reilly Radar about mobile performance, and he asked me a couple of interesting question:

  • Are mobile users more or less tolerant of delays than desktop users?
  • Are users of one type of system more accepting of delays than users of another?

These questions are a gateway to a fascinating area of research, because they lead into a topic that we all have pet theories about (i.e. Chrome users are more tech savvy than the average person, while Internet Explorer users are less) but have little statistical evidence to back up.

I told Mac that I planned to do more digging and report back, so here I am.

Methodology

  1. I took five e-commerce sites (full sites, not mobile versions) that Strangeloop is currently accelerating and pulled their entire transaction volume over the past month — totaling hundreds of millions of unique visits via desktop and mobile. While desktop transactions outnumbered mobile transactions, the mobile numbers were still statistically significant: the smallest set of mobile numbers comprised around 200,000 unique visits and the largest set comprised about 20 million unique visits.
  2. I extracted the following data: page views, time on site, and bounce rate.
  3. I sorted the data into the following browser/OS groups: Internet Explorer, Firefox, Chrome, iPad, iPhone, and Android (phone).
  4. I calculated the averages for each metric and browser.
  5. I graphed the numbers and looked for trends. Some interesting patterns emerged:

Desktop vs. mobile performance: average pageviewsDesktop vs mobile performance: average time on siteDesktop vs mobile browser performance: average bounce rate

Three Key Findings

Finding #1: Internet Explorer users consistently view more pages, spend more time on site, and have a lower bounce rate than Firefox and Chrome users.

The average number of page views for IE users was 6.134, as opposed to 5.17 for Firefox users and 5.14 for Chrome users. IE users spent between 30-45 seconds longer on the site than other users, and their bounce rate was lower by 5 or 6 percentage points — a pretty significant difference. Could all of this substantiate the belief (among non-IE users, at least) that IE fans are less tech savvy and therefore slower and more ponderous web users than the rest of us? Or perhaps it’s a hardware issue — are IE users more likely to be using older systems with less processing power?

But what about that lower bounce rate? At around 35%, it’s a pretty strong number, especially compared to 41% for Firefox users and 42% for Chrome. A lower bounce rate generally signifies that people who come to your site find it relevant and worth sticking around to check out. Are IE users better searchers and more likely to arrive at the right destination, or are they simply more easily satisfied than other users?

Finding #2: iPad users are more similar to desktop users than they are to smartphone users.

While iPad users view somewhat fewer pages per visit than desktop users (4.54 versus 5.14, 5.17, and 6.13), their average time on site and bounce rate were commensurate with the desktop crowd. This isn’t a huge surprise. We know that most iPad users are browsing in the comfort of their home, and they consider their iPad to be more like a small laptop than an oversized phone. What’s interesting here is that, even though iPad performance lags behind desktop (it is a mobile device, after all, and it suffers from many of the same performance constraints as a smartphone: from low processor power to touchscreen lag), iPad users seem willing to stick around for a longer desktop-like experience.

Finding #3: iPhone users consistently view fewer pages, spend less time on site, and have a higher bounce rate.

At the opposite end of the spectrum from Internet Explorer users we find iPhone users. In every sample group, iPhone users, on average, spent significantly less time on site (2:31 vs 3:20) and viewed fewer pages (2.41 vs 3.1) than Android, and had a higher bounce rate (60.76% vs 57.17%). The shorter time spent on site could be attributed to the (arguable) fact that iPhones are better-powered than other devices, but that doesn’t account for the page views and bounce rate. Do these validate all the stereotypes about iPhone users: that they — or should I say, we — are impatient, savvy web users who will bounce from a site if we can’t find what we want right away and aggressively search elsewhere? Or that we know what we want and can expedite a transaction faster and more efficiently than other users?

Takeaways

This research doesn’t directly answer Mac’s questions, but it does come at them sideways and raise some interesting — to me, anyway — questions about the types of people who use different technologies, and how and why they use them. We can’t answer these questions today, but these findings are food for thought and debate.

Your thoughts?

Related posts:

By: Josh Bixby Category: Ecommerce, Mobile, Research Tags: , , , , , , , , , ,

When it comes to mobile development, does bandwidth still matter?

12 Jan 2012

2 Comments

If you work in mobile development, there’s a good chance you already have a strong opinion about responsive design. (If you don’t already have a strong opinion, that will probably change by the end of this year. As buzz phrases go, “responsive design” could be the “cloud computing” of 2012.)

Given all this, I couldn’t not respond to this post about responsive design written by Joaquin Lippincott, president of Metal Toad Media: When It Comes to Mobile Development, Stop Worrying about Bandwidth. He makes some good points about the fact that some responsive design techniques — such as swapping out images for CSS3, and using other CSS3 techniques like gradients and transparencies — are processor-intensive, meaning that while they may deliver superior performance, they can be tough on your device’s CPU.

But this statement stopped me in my tracks:

When it comes to building websites in 2012, bandwidth truly doesn’t matter. [emphasis mine] And it’s going to matter even less in 2013 with the roll out of more 4G networks. Additionally many devices are often on wi-fi, so provider network speed is truly a non-issue. Even on 3G, bandwidth isn’t a big deal – we’re streaming HD video, so don’t sweat 20k images.

To say that bandwidth seriously doesn’t matter, in any context, is obvious bait for any performance geek. True, a growing number of mobile users are browsing via tablets at home over their wifi connection, but this shouldn’t distract us from the fact that mobile use is growing across the board, which means that the total number of people connecting over 3G and 4G is still increasing. And any mobile user who is sensitive to their data cap — and to those monthly bills from their telecom provider — will tell you that, for them, bandwidth truly does still matter.

Not surprisingly, there were comments from a few performance geeks at the end of the post, which ultimately led to Joaquin clarifying his point to mean that developers and designers need to balance the trade-offs between bandwidth and CPU load.

While I think that anyone who builds sites for a living gets the fact that speed is an important usability issue (or as I prefer to put it: speed is THE important usability issue), I hear alarm bells when I hear people casually toss off statements like “bandwidth doesn’t matter”. To me, this post demonstrates the wide gap that still exists between design and performance. It also makes me wonder what kinds of conversations are happening out there between digital media agencies and their clients, who might not have the technical background to understand the nuances of what’s being discussed.

Your thoughts on juggling responsive design and mobile performance? I’d love to hear stories from the trenches.

Related posts:

By: Josh Bixby Category: Mobile Tags: , , ,

Colonoscopies, cold water and pain: How our memory works and how this relates to web performance

5 Jan 2012

1 Comment

Interesting fact: Back in the 1990s, colonoscopies were routinely conducted without anaesthetic or amnesiatic drugs. (Note: I’ve never undergone this procedure myself, but I can only imagine that it’s more than a bit uncomfortable.)

In the 1990s, researchers at the University of Toronto conducted a study in which patients undergoing a colonoscopy were prompted every 60 seconds to indicate the level of pain they were experiencing. Patients rated pain on a scale of zero to 10, in which zero was no pain and 10 was intolerable pain. A total of 154 patients participated in the experiment; the shortest procedure lasted 4 minutes, and the longest was 69 minutes.

In the graphs below, you can see the experience of two representative patients. As you can see, the experience of each patient varied considerably during the procedure, which lasted 8 minutes for patient A and 24 minutes for patient B.

Pain and perception: How this relates to web performance

An interesting question emerges from this: Assuming that both patients used the scale of pain similarly, who actually experienced more pain? Based on these graphs, most of us would assume that Patient B suffered significantly more than Patient A.

Surprise finding: Duration of pain doesn’t correlate with perceived intensity

After the procedure, patients were asked to rate the “total amount of pain” they had experienced during the procedure. Surprisingly, Patient A retained a much worse memory of the experience then Patient B — in fact it was twice as bad.

What emerges from this are two very interesting patterns in the human brain, which have been repeated and noted in many subsequent studies:

  • Peak-end rule: The total amount of pain was well predicted by the average of the level of pain reported at the worst moment of the experience and at its end.
  • Duration neglect: The duration of the procedure had no effect whatsoever on the rating of overall pain.

The fact that duration has no real impact on the overall experience intrigued me, so I dug deeper into this area of research.

I found another study that shed more light on this (a study whose name I really like: the Cold Pressor). In this study, participants were asked to hold their hands in painfully cold water until they are invited to remove it and are offered a warm towel. During immersion, participants were asked to provide a continuous record of the pain they experience.

Each participant endured two cold-hand episodes, with some starting with the short episode and some starting with the long one:

  • The short episode consisted of immersion for 60 seconds at 14 degrees Celsius followed by a warm cloth and a 7-minute break.
  • The long episode (conducted on the other hand) consisted of the short episode + 30 additional seconds in which the water was made 1 degree warmer without the knowledge of the participant (1 degree Celsius is barely perceptible), followed by a warm cloth and a 7-minute break.

After completing both episodes, participants were told they needed to perform a third trial and were given the choice to repeat either the first or the second experience. The participants had no explicit knowledge that the duration of each episode was different, or that the water temperature had changed.

The peak-end rule theory predicted a worse memory of the short episode, and duration neglect predicted that the difference between 60 and 90 seconds will be ignored. This is exactly what they found: 80% of the participants chose to repeat the long episode and endure 30 seconds of slightly reduced, but totally needless, pain.

If participants had been asked “Would you prefer a 60-second or 90-second immersion?” I’m certain everyone would have selected the short episode. When asked, people knew which episode was longer, but they did not use this information to make their decision.

How do the ‘peak-end rule’ and ‘duration neglect’ apply to web performance?

Interestingly, the peak-end rule would tell us that the last page in a flow (i.e. the end of the experience) would have a large impact on a user’s experience and perception. In an e-commerce scenario, the last experience is often the slowest, caused by a long credit card authentication process. For example, I just booked an airline ticket and tracked the page load time across the flow:

Country selector page 1.3 seconds
Home page 3.2 seconds
Search results page 6.4 seconds
Review flight details page 3.3 seconds
Upsell me on stuff I don’t want page 4.2 seconds
Credit card input page 2.1 seconds
Confirmation page 11.3 seconds

Conclusion: We need more studies

In the past, I’ve shared a case study showing how we tracked abandonment throughout a mobile transaction, and how slowing down different pages in the flow affected bounce rate. As a follow-up to that (which I hope to get to soon), I’m very interested in seeing, on a predetermined flow, the relationship between the performance of the last page and overall user satisfaction (measured by repeat site usage and/or cross referenced with the customer survey data that many of our customers religiously gather).

The duration neglect phenomenon is even more interesting. If human memory neglects to take duration into consideration as a key factor in determining the pain or gain of an experience, why is page performance so closely correlated to overall user satisfaction and key metrics like bounce rate, page views and conversion. I think we need to perform more controlled studies on real participants to really understand how duration neglect relates to our world.

I’ve been thinking about designing a few studies to mimic the controlled psychology studies above. If you have an experiment design you’d like to share — or better yet, if you run a psychology department and want to collaborate — let me know.

Related posts:

By: Josh Bixby Category: Human factors, Research
1 2 3 4 5 6 7 8 »