metrics

Advanced Mobile Optimization: How does it work? How do we measure success? [slides]

27 Jan 2012

0 Comment

It’s been a busy couple of weeks, but I finally got around to posting the slides from my talk about advanced mobile optimization at the San Francisco & Silicon Valley Web Performance Meetup.

Advanced Mobile Optimization: How does it work? How do we measure success?
View more presentations from Strangeloop

I always enjoy coming to these Meetups, and this time was no exception. Thanks again to Aaron Kulick for inviting me, to LinkedIn for hosting, and to the extremely keen and knowledgeable crowd who turned out. :)

Related posts:

By: Josh Bixby Category: Mobile Tags: , , , , ,

Interesting new findings about page views, time on site, and bounce rate across desktop and mobile browsers

20 Jan 2012

0 Comment

Last month, I talked with Mac Slocum at O’Reilly Radar about mobile performance, and he asked me a couple of interesting question:

  • Are mobile users more or less tolerant of delays than desktop users?
  • Are users of one type of system more accepting of delays than users of another?

These questions are a gateway to a fascinating area of research, because they lead into a topic that we all have pet theories about (i.e. Chrome users are more tech savvy than the average person, while Internet Explorer users are less) but have little statistical evidence to back up.

I told Mac that I planned to do more digging and report back, so here I am.

Methodology

  1. I took five e-commerce sites (full sites, not mobile versions) that Strangeloop is currently accelerating and pulled their entire transaction volume over the past month — totaling hundreds of millions of unique visits via desktop and mobile. While desktop transactions outnumbered mobile transactions, the mobile numbers were still statistically significant: the smallest set of mobile numbers comprised around 200,000 unique visits and the largest set comprised about 20 million unique visits.
  2. I extracted the following data: page views, time on site, and bounce rate.
  3. I sorted the data into the following browser/OS groups: Internet Explorer, Firefox, Chrome, iPad, iPhone, and Android (phone).
  4. I calculated the averages for each metric and browser.
  5. I graphed the numbers and looked for trends. Some interesting patterns emerged:

Desktop vs. mobile performance: average pageviewsDesktop vs mobile performance: average time on siteDesktop vs mobile browser performance: average bounce rate

Three Key Findings

Finding #1: Internet Explorer users consistently view more pages, spend more time on site, and have a lower bounce rate than Firefox and Chrome users.

The average number of page views for IE users was 6.134, as opposed to 5.17 for Firefox users and 5.14 for Chrome users. IE users spent between 30-45 seconds longer on the site than other users, and their bounce rate was lower by 5 or 6 percentage points — a pretty significant difference. Could all of this substantiate the belief (among non-IE users, at least) that IE fans are less tech savvy and therefore slower and more ponderous web users than the rest of us? Or perhaps it’s a hardware issue — are IE users more likely to be using older systems with less processing power?

But what about that lower bounce rate? At around 35%, it’s a pretty strong number, especially compared to 41% for Firefox users and 42% for Chrome. A lower bounce rate generally signifies that people who come to your site find it relevant and worth sticking around to check out. Are IE users better searchers and more likely to arrive at the right destination, or are they simply more easily satisfied than other users?

Finding #2: iPad users are more similar to desktop users than they are to smartphone users.

While iPad users view somewhat fewer pages per visit than desktop users (4.54 versus 5.14, 5.17, and 6.13), their average time on site and bounce rate were commensurate with the desktop crowd. This isn’t a huge surprise. We know that most iPad users are browsing in the comfort of their home, and they consider their iPad to be more like a small laptop than an oversized phone. What’s interesting here is that, even though iPad performance lags behind desktop (it is a mobile device, after all, and it suffers from many of the same performance constraints as a smartphone: from low processor power to touchscreen lag), iPad users seem willing to stick around for a longer desktop-like experience.

Finding #3: iPhone users consistently view fewer pages, spend less time on site, and have a higher bounce rate.

At the opposite end of the spectrum from Internet Explorer users we find iPhone users. In every sample group, iPhone users, on average, spent significantly less time on site (2:31 vs 3:20) and viewed fewer pages (2.41 vs 3.1) than Android, and had a higher bounce rate (60.76% vs 57.17%). The shorter time spent on site could be attributed to the (arguable) fact that iPhones are better-powered than other devices, but that doesn’t account for the page views and bounce rate. Do these validate all the stereotypes about iPhone users: that they — or should I say, we — are impatient, savvy web users who will bounce from a site if we can’t find what we want right away and aggressively search elsewhere? Or that we know what we want and can expedite a transaction faster and more efficiently than other users?

Takeaways

This research doesn’t directly answer Mac’s questions, but it does come at them sideways and raise some interesting — to me, anyway — questions about the types of people who use different technologies, and how and why they use them. We can’t answer these questions today, but these findings are food for thought and debate.

Your thoughts?

Related posts:

By: Josh Bixby Category: Ecommerce, Mobile, Research Tags: , , , , , , , , , ,

The 20 best web performance links of Q4

15 Dec 2011

1 Comment

Every time I write one of these posts, I’m impressed by the volume and quality of writing that happens in our industry on an ongoing basis. It’s truly an exciting time for web performance. I feel endlessly engaged by the dialogue that happens every day, and honoured to be part of it it.

This roundup (which includes links pulled from the Strangeloop WPO Hub), includes some increasingly refined thinking about mobile optimization, a handful of excellent tutorials and case studies (including some great new presentations from Velocity EU), and some revolutionary browser developments.

But my favourite link is this first one…

The best link of Q4

Retailers need for tech speed
Does it tell us anything new? No. But I’ve lost count of how many people I’ve forwarded this two-minute segment on CNBC’s “Power Lunch” — which discusses the importance of speed for e-commerce sites, particularly for mobile users, during the holiday shopping season. For me, this shows that site speed has finally jumped into the mainstream. I’m excited to see how this attention snowballs in 2012.

Mobile

Mobile UI Performance
This slide deck from Estelle Weyl’s excellent presentation at Velocity EU gives an overview of mobile performance challenges, why we need to address them differently than we deal with desktop sites, and detailed tips on how to do just that.

Performance Automation 101
This slide deck from Jeroen Tjepkema’s Velocity EU presentation explains what performance automation is, how it works, and why it’s the only viable solution for dealing with the challenges of mobile device/browser fragmentation.

HTML5 Techniques for Optimizing Mobile Performance
Great post on HTML5 Rocks: ”In this article, we will discuss the bare minimum of what it takes to create a mobile HTML5 web app. The main point is to unmask the hidden complexities which today’s mobile frameworks try to hide. You will see a minimalistic approach (using core HTML5 APIs) and basic fundamentals that will empower you to write your own framework or contribute to the one you currently use.”

Mobile Performance Manifesto
Love this post from David Calhoun itemizing and describing mobile performance best practices.

Tools

How WebPagetest works
If you’ve ever wondered how exactly WebPagetest gathers performance data from the various browsers it simulates, this is great post from Pat Meenan in which he cracks the hood of WebPagetest and explains all that.

Mobile Perf Bookmarklet
Steve Souders offers one mobile bookmarklet to rule them all: a new “master bookmarklet” that lets you install a handful of common debugger and profiler bookmarklets in your mobile broswer in one step.

Is Synthetic Monitoring Really Going to Die?
Alois Reitbauer asks: “Will User Experience Management using JavaScript agents eventually replace synthetic monitoring or will there be a coexistence of both approaches in the end?” As you might guess, the answer is not cut and dried.

Case studies, how-tos, and other research

Diagnosing Slow Web Servers with Time to First Byte
Much as it pains me to admit it, from time to time performance pains aren’t caused at the front end. Performance expert Andy King gives some good tips on how to use the time to first byte metric, as displayed on a waterfall chart, to help diagnose a slow server.

The art and craft of the async snippet
Stoyan Stefanov examines the topic of asynchronous code “from the perspective of a third party – when you’re the third party, providing a snippet for other developers to include on their pages, be it an ad, a plugin, widget, visits counter, analytics, or anything else.”

Why loading third party scripts async is not good enough
We talk about asynchronously loading third-party snippets as if that’s the sole cure for performance pains, but in this case study, Aaron Peters reminds us that sometimes it’s okay to defer their loading until after onload.

Fast Loading JavaScript
Slide deck from performance consultant Aaron Peters’ great Velocity EU presentation: “A walk-through of several JavaScript loading techniques with a characteristics table for each and at the end a decision tree to help you decide which technique to use.”

How Downtime Financially Impacts Top Ecommerce Websites
Compelling infographic showing how downtime affected the Internet Retailer 500 in 2010. Includes the estimate that downtime resulted in more than $300 million in lost revenue for the IR 500.

Testing for Frontend SPOF
Excellent post from Pat Meenan in which he simulates third-party outage with a blackhole server in order to demonstrate — via WebPagetest-generated video — how that outage slows down or disrupts page load.

Browsers and connectivity

SPDY of the Future Might Blow Your Mind Today
Great post (“definitely for protocol geeks”) by Google software engineer Mike Belshe on SPDY’s evolution and how Kindle Silk is taking it beyond other browsers.

Chrome Fast
Slides from Google software engineer Tony Gentilcore’s excellent presentation at Velocity EU, in which he gives an overview of the Chrome platform and explains what makes Chrome fast.

Report reveals drop between peak and off-peak surfing
No big surprise, but a good reminder (especially if you rely on synthetic testing) that real-world performance is a nebulous thing: UK study finds that web speed is up to 69% slower during evening peak time.

The end of an era: Internet Explorer drops below 50% of Web usage
Mark the month and year. November 2011 was the first time in more than a decade that Internet Explorer’s share of global browser usage dropped below 50%.

Opinion pieces

Your CDN is not a silver bullet for web performance
In the e-commerce and SaaS world, the two most common causes of poor web performance are third-party content and server-side processing. Neither of these bottlenecks are addressed by loading static content from a closer location via a content delivery network.

Why you have less than a second to deliver exceptional performance
dynaTrace’s Alois Reitbauer writes: “Being exceptionally fast is becoming the dogma for developing web applications. But what is exceptionally fast and how hard is it to build a top performing web site?” I like posts like this because they remind us what the fundamental questions are that our industry is trying to address.

If you have any other great links to share, let me know in the comments. And if you’re looking for more great links, we have hundreds — sorted by topic, industry, and type — over in the Strangeloop WPO Hub.

Related posts:

By: Josh Bixby Category: Browsers, Case studies, Ecommerce, Industry news, Mobile, Research, Third-party content, Tools Tags: , , , , , , , , , , , , ,

Case study: The impact of HTML delay on mobile business metrics

23 Nov 2011

4 Comments

When I gave my talk about mobile performance and business KPIs at Velocity Berlin a couple weeks back, one of the areas I got the most questions about later was the experiments we were able to do in which we delayed HTML on a customer’s site and tracked the results over a 12-week period. I thought it might be useful to break some of this out into its own post.

As I mentioned at Velocity, I was insanely jealous of Google and Bing a couple years ago, when they revealed their own in-house experiments with HTML delay. Most of us in the performance community would kill for that kind of experimentation platform. So I was extremely happy and grateful when one of our customers at Strangeloop expressed an interest in figuring out the value of time for their business.

Methodology

We conducted a split test over the course of 12 weeks, in which we segmented mobile traffic into four groups: fully optimized, 200 ms delay, 500 ms delay, and 1000 ms delay. We monitored four metrics: bounce rate, conversion rate, cart size, and page views. We also monitored and analyzed user behavior for 6 weeks after the test ended, to gauge the long-term impact, if any, of slow performance even after users begin to receive an accelerated site.

Results

The results of the 200 ms delay weren’t significant, but the customer and I were both taken by the dramatic impact of the 500 ms and 1000 ms slowdown. Our customer was blown away that they were losing 3.5% percent of their conversions when their site was delayed by just one second on a mobile device. This was a major epiphany for them, and it’s already helped them change their business and how they view mobile.

I’m including this next graph to reinforce the connection between load time and user behavior.

Over the 12 weeks, you can see that, while the HTML delays were constant — 500 ms and 1000 ms — users’ reaction to these delays fluctuated. The drop in bounce rate ranged from around -2% and -12% for users who experienced a 1000 ms delay, and 0% and -6% for those who experienced a 500 ms delay. While the bounce rate may have varied, the one thing that was constant was the fact that the behavior trends are strongly linear for both groups, and the bounce rate for the 1000 ms group was consistently worse.

Finally, and most interestingly to me, we wanted to look beyond just the effect of delay in the timeframe of the experiment. We know that slower pages have an immediate impact on user behavior and customer satisfaction. We wanted to find out if there was any long-term impact on customer satisfaction.

So we looked at our traffic data for the 6 weeks immediately after the experiment — specifically at the behavior of returning visitors. As any site owner will tell you, repeat customers are the bread and butter of an e-commerce vendor. These are the people you need to keep happy. If you look at the graph below, you can see that, even after the experiment was over and the shoppers in the 500 ms and 1000 ms group started to be served the same accelerated site as the baseline group, they were significantly less likely to return to the site. By the end of the 6-week period, you can see that return traffic is slowly improving as visitors seem to finally be recovering from their poor experience.

Conclusions

As I’ve already mentioned, these findings have been a huge revelation to the company that owns this site. It’s had a major impact on how they’re tackling performance on their mobile platform. The most important over-arching takeaways from this experiment were:

  • Mobile shoppers are now fully engaging with e-commerce sites in significant enough numbers that we can analyze their behavior as a group.
  • Even a 500 ms delay has a major impact on metrics. For mobile sites, which can suffer excruciating load times (the latest Keynote index is about 12 seconds), this is a wake-up call that they need to take a hard-line approach to optimizing their pages.
  • The damage of poor performance is lasting.

See the full presentation: Case Studies from the Mobile Frontier: The Relationship Between Faster Mobile Sites and Business KPIs

Related posts:

By: Josh Bixby Category: Case studies, Ecommerce, Mobile Tags: , , , , , , , ,

Why doesn’t our industry talk more about performance and productivity?

18 Oct 2011

5 Comments

Ever wondered how much time people in your company spend waiting for internal apps to load? One of our clients did some math on this, and they got some eye-opening results:

They calculated that, in just one small department of 20 people, those people spent a total of 130 hours per month waiting for the pages of a single internal web-based application to load. That’s 130 hours (in other words, almost an entire month’s worth of work hours for one person) that could have been spent making sales, responding to clients, fighting with the photocopier – basically doing anything more productive than staring at a screen.

In 2008, Aberdeen released a benchmark report called Application Performance Management: The Lifecycle Approach Brings IT and Business Together. This report stated that application performance issues — including internal app performance — could hurt overall corporate revenues by up to 9%.

At the same time, Aberdeen also found that the average organization was using six business-critical applications, with plans to rollout four more, bringing the total up to ten applications by 2010.

Let’s imagine a not-so-crazy scenario

Now let’s apply Aberdeen’s findings to my customer findings at the top of this post. Imagine that this same department of 20 people isn’t using just one application, but instead is using ten. Imagine that four out of ten of these apps are experiencing significant performance problems.

The results:

  • Instead of waiting 130 hours, the people in this department spend a total of 520 hours a month waiting for pages to load.
  • That’s the equivalent of the total number of work hours for 3.5 employees.
  • In monetary terms: that wait time equals about $9,500 per month, or $114,000 a year (based on an average annual salary of $32,700, according to  U.S. Bureau of Labor Statistics, January 2010).
  • Instead of a 20-person department, extrapolate these numbers throughout a larger enterprise, such as Amazon. With more than 33,000 employees in total, let’s assume that half are desktop workers. Following the rationale above, poor app performance could cost Amazon upward of $7.8 million a month, or $94 million a year.

Crazy? No. A bit facile? Yes. But only a bit. While employees are waiting for apps to load, they could be doing other things, such as making calls and multitasking with other slow apps. But bear in mind usability expert Jakob Nielsen’s findings about response time and human behavior: 10 seconds is about the limit for keeping a user’s attention focused on a non-responsive dialogue. After 10 seconds, even the most efficient worker has to struggle to re-focus on the task at hand. Repeated interruptions are a huge detriment to productivity.

We need more performance + productivity case studies.

We all know about the impact that faster page load had on revenue for Amazon and Shopzilla. But there are some inspiring lesser-known case studies that demonstrate the relationship between improving internal app performance and employee productivity.

In one case study, Hydro-Québec was experiencing some serious performance pains with a shared CAD app, with people in remote offices suffering 30- to 60-second delays between every mouse click. Not surprisingly: “Response time was ugly,” according to Daniel Brisebois, Hydro-Québec’s IT advisor. After application response time was improved by 10- to 25-fold, Hydro-Québec reported benefits such as fewer errors, a faster engineering cycle, and enhanced data integrity.

In another case study, this one for the Hilton Grand Vacations Club, a division of the company that sells timeshares to international customers, the company was experiencing major latency issues that caused delays of up to 30 minutes for a vital contract-processing app. Hilton’s senior director of technology applications, Rich Jackson, said that “As you can imagine, with the customer is sitting in front of you while you are waiting on a computer process, it’s not an ideal situation. When you’re processing a contract, time is of the essence. You don’t want delays due to technology issues.” After accelerating the app, Jackson said that “The contract process has been reduced from more than 30 minutes to just a minute or two. It has a huge effect on customer and employee satisfaction.”

These are good stories, but we need more.

Why don’t we WCO folks talk more about performance and productivity?

Off the top of my head:

  • In the early days of our industry, we distanced ourselves from the application acceleration folks in order to carve out our own niche.
  • We share information using tools like WebPagetest and the HTTP Archive, which are only relevant to the public web.
  • Talking about money is a lot sexier than talking about productivity.

In the months to come, I’m going to do my part to make productivity as sexy as money. If you have productivity stories to share, I’d love to hear about them.

Related posts:

By: Josh Bixby Category: Case studies, Productivity Tags: , , , ,
1 2 3 4 5