Research

New report: What we’ve learned from two years of watching the top 2,000 e-commerce websites

26 Jan 2012

2 Comments

Yesterday was a busy day for us here at Strangeloop. We released our second annual “state of the union” report on the page speed and performance of 2,000 leading e-commerce websites.

Infographics: 2012 State of the Union for E-Commerce Page Speed and Website Performance

We came up with the idea of conducting these reports over a year ago, when we realized there was no way to track performance changes and trends — from a real user’s perspective — on an ongoing basis. As I mentioned to CNet’s Stephen Shankland, when we talked about the report the other day, we can’t rely on benchmarking tests run out of data centers to give us any relevant sense of how our sites are actually performing out in the real world. They’ve got ridiculous bandwidth, and they’re parked right next to the content delivery machines, so they’ve got zero latency. By using WebPagetest — which simulates how fast a site loads in a real user’s browser — to measure the performance of the same set of 2,000 sites year after year, we can compare real-world performance snapshots and see what’s changed, and what’s caused the changes.

I don’t want to give away all our findings, the highlights of which are illustrated in the infographics above, but a couple of interesting things we discovered:

  • Pages have gotten bigger – from an average of 86 requests last year to 98 requests this year. This kind of growth is consistent with trends identified via the HTTP Archive.
  • Repeat views are a whopping 20% slower than they were last year, probably due to the the number of page objects.
  • Page Speed scores have gotten significantly worse, from 83% last year to 75% now. I was just talking to Pat Meenan, who created WebPagetest, and he pointed out that a probable reason for the drop is because the newest version of Page Speed checks for more optimizations, which has resulted in lower scores across the board. (Side note: It’s important to know that Page Speed doesn’t take into account advanced front-end content optimization techniques — nor could it, because there are way too many of them to track and measure. As a for-instance, we’ve had experiences here at Strangeloop where we’ve taken a page with a perfect Page Speed score, accelerated it with Site Optimizer, cut load time in half, and ended up with a new Page Speed score of 74%.)

While the findings are a mixed bag of good and bad news, to me the most exciting takeaway is the  attention this report has received from major media outlets like CBS News. It goes to confirm what became evident over Black Friday: site speed has become a mainstream issue.

Download the report: 2012 State of the Union: E-Commerce Page Speed and Website Performance

Download a high-res version of the infographics above (feel free to re-post): Poster: E-Commerce Page Speed and Website Performance (2012)

Related links:

By: Josh Bixby Category: Ecommerce, Research, Standards and benchmarks Tags: , , , , , , , , , , , ,

Interesting new findings about page views, time on site, and bounce rate across desktop and mobile browsers

20 Jan 2012

0 Comment

Last month, I talked with Mac Slocum at O’Reilly Radar about mobile performance, and he asked me a couple of interesting question:

  • Are mobile users more or less tolerant of delays than desktop users?
  • Are users of one type of system more accepting of delays than users of another?

These questions are a gateway to a fascinating area of research, because they lead into a topic that we all have pet theories about (i.e. Chrome users are more tech savvy than the average person, while Internet Explorer users are less) but have little statistical evidence to back up.

I told Mac that I planned to do more digging and report back, so here I am.

Methodology

  1. I took five e-commerce sites (full sites, not mobile versions) that Strangeloop is currently accelerating and pulled their entire transaction volume over the past month — totaling hundreds of millions of unique visits via desktop and mobile. While desktop transactions outnumbered mobile transactions, the mobile numbers were still statistically significant: the smallest set of mobile numbers comprised around 200,000 unique visits and the largest set comprised about 20 million unique visits.
  2. I extracted the following data: page views, time on site, and bounce rate.
  3. I sorted the data into the following browser/OS groups: Internet Explorer, Firefox, Chrome, iPad, iPhone, and Android (phone).
  4. I calculated the averages for each metric and browser.
  5. I graphed the numbers and looked for trends. Some interesting patterns emerged:

Desktop vs. mobile performance: average pageviewsDesktop vs mobile performance: average time on siteDesktop vs mobile browser performance: average bounce rate

Three Key Findings

Finding #1: Internet Explorer users consistently view more pages, spend more time on site, and have a lower bounce rate than Firefox and Chrome users.

The average number of page views for IE users was 6.134, as opposed to 5.17 for Firefox users and 5.14 for Chrome users. IE users spent between 30-45 seconds longer on the site than other users, and their bounce rate was lower by 5 or 6 percentage points — a pretty significant difference. Could all of this substantiate the belief (among non-IE users, at least) that IE fans are less tech savvy and therefore slower and more ponderous web users than the rest of us? Or perhaps it’s a hardware issue — are IE users more likely to be using older systems with less processing power?

But what about that lower bounce rate? At around 35%, it’s a pretty strong number, especially compared to 41% for Firefox users and 42% for Chrome. A lower bounce rate generally signifies that people who come to your site find it relevant and worth sticking around to check out. Are IE users better searchers and more likely to arrive at the right destination, or are they simply more easily satisfied than other users?

Finding #2: iPad users are more similar to desktop users than they are to smartphone users.

While iPad users view somewhat fewer pages per visit than desktop users (4.54 versus 5.14, 5.17, and 6.13), their average time on site and bounce rate were commensurate with the desktop crowd. This isn’t a huge surprise. We know that most iPad users are browsing in the comfort of their home, and they consider their iPad to be more like a small laptop than an oversized phone. What’s interesting here is that, even though iPad performance lags behind desktop (it is a mobile device, after all, and it suffers from many of the same performance constraints as a smartphone: from low processor power to touchscreen lag), iPad users seem willing to stick around for a longer desktop-like experience.

Finding #3: iPhone users consistently view fewer pages, spend less time on site, and have a higher bounce rate.

At the opposite end of the spectrum from Internet Explorer users we find iPhone users. In every sample group, iPhone users, on average, spent significantly less time on site (2:31 vs 3:20) and viewed fewer pages (2.41 vs 3.1) than Android, and had a higher bounce rate (60.76% vs 57.17%). The shorter time spent on site could be attributed to the (arguable) fact that iPhones are better-powered than other devices, but that doesn’t account for the page views and bounce rate. Do these validate all the stereotypes about iPhone users: that they — or should I say, we — are impatient, savvy web users who will bounce from a site if we can’t find what we want right away and aggressively search elsewhere? Or that we know what we want and can expedite a transaction faster and more efficiently than other users?

Takeaways

This research doesn’t directly answer Mac’s questions, but it does come at them sideways and raise some interesting — to me, anyway — questions about the types of people who use different technologies, and how and why they use them. We can’t answer these questions today, but these findings are food for thought and debate.

Your thoughts?

Related posts:

By: Josh Bixby Category: Ecommerce, Mobile, Research Tags: , , , , , , , , , ,

Colonoscopies, cold water and pain: How our memory works and how this relates to web performance

5 Jan 2012

1 Comment

Interesting fact: Back in the 1990s, colonoscopies were routinely conducted without anaesthetic or amnesiatic drugs. (Note: I’ve never undergone this procedure myself, but I can only imagine that it’s more than a bit uncomfortable.)

In the 1990s, researchers at the University of Toronto conducted a study in which patients undergoing a colonoscopy were prompted every 60 seconds to indicate the level of pain they were experiencing. Patients rated pain on a scale of zero to 10, in which zero was no pain and 10 was intolerable pain. A total of 154 patients participated in the experiment; the shortest procedure lasted 4 minutes, and the longest was 69 minutes.

In the graphs below, you can see the experience of two representative patients. As you can see, the experience of each patient varied considerably during the procedure, which lasted 8 minutes for patient A and 24 minutes for patient B.

Pain and perception: How this relates to web performance

An interesting question emerges from this: Assuming that both patients used the scale of pain similarly, who actually experienced more pain? Based on these graphs, most of us would assume that Patient B suffered significantly more than Patient A.

Surprise finding: Duration of pain doesn’t correlate with perceived intensity

After the procedure, patients were asked to rate the “total amount of pain” they had experienced during the procedure. Surprisingly, Patient A retained a much worse memory of the experience then Patient B — in fact it was twice as bad.

What emerges from this are two very interesting patterns in the human brain, which have been repeated and noted in many subsequent studies:

  • Peak-end rule: The total amount of pain was well predicted by the average of the level of pain reported at the worst moment of the experience and at its end.
  • Duration neglect: The duration of the procedure had no effect whatsoever on the rating of overall pain.

The fact that duration has no real impact on the overall experience intrigued me, so I dug deeper into this area of research.

I found another study that shed more light on this (a study whose name I really like: the Cold Pressor). In this study, participants were asked to hold their hands in painfully cold water until they are invited to remove it and are offered a warm towel. During immersion, participants were asked to provide a continuous record of the pain they experience.

Each participant endured two cold-hand episodes, with some starting with the short episode and some starting with the long one:

  • The short episode consisted of immersion for 60 seconds at 14 degrees Celsius followed by a warm cloth and a 7-minute break.
  • The long episode (conducted on the other hand) consisted of the short episode + 30 additional seconds in which the water was made 1 degree warmer without the knowledge of the participant (1 degree Celsius is barely perceptible), followed by a warm cloth and a 7-minute break.

After completing both episodes, participants were told they needed to perform a third trial and were given the choice to repeat either the first or the second experience. The participants had no explicit knowledge that the duration of each episode was different, or that the water temperature had changed.

The peak-end rule theory predicted a worse memory of the short episode, and duration neglect predicted that the difference between 60 and 90 seconds will be ignored. This is exactly what they found: 80% of the participants chose to repeat the long episode and endure 30 seconds of slightly reduced, but totally needless, pain.

If participants had been asked “Would you prefer a 60-second or 90-second immersion?” I’m certain everyone would have selected the short episode. When asked, people knew which episode was longer, but they did not use this information to make their decision.

How do the ‘peak-end rule’ and ‘duration neglect’ apply to web performance?

Interestingly, the peak-end rule would tell us that the last page in a flow (i.e. the end of the experience) would have a large impact on a user’s experience and perception. In an e-commerce scenario, the last experience is often the slowest, caused by a long credit card authentication process. For example, I just booked an airline ticket and tracked the page load time across the flow:

Country selector page 1.3 seconds
Home page 3.2 seconds
Search results page 6.4 seconds
Review flight details page 3.3 seconds
Upsell me on stuff I don’t want page 4.2 seconds
Credit card input page 2.1 seconds
Confirmation page 11.3 seconds

Conclusion: We need more studies

In the past, I’ve shared a case study showing how we tracked abandonment throughout a mobile transaction, and how slowing down different pages in the flow affected bounce rate. As a follow-up to that (which I hope to get to soon), I’m very interested in seeing, on a predetermined flow, the relationship between the performance of the last page and overall user satisfaction (measured by repeat site usage and/or cross referenced with the customer survey data that many of our customers religiously gather).

The duration neglect phenomenon is even more interesting. If human memory neglects to take duration into consideration as a key factor in determining the pain or gain of an experience, why is page performance so closely correlated to overall user satisfaction and key metrics like bounce rate, page views and conversion. I think we need to perform more controlled studies on real participants to really understand how duration neglect relates to our world.

I’ve been thinking about designing a few studies to mimic the controlled psychology studies above. If you have an experiment design you’d like to share — or better yet, if you run a psychology department and want to collaborate — let me know.

Related posts:

By: Josh Bixby Category: Human factors, Research

The 20 best web performance links of Q4

15 Dec 2011

1 Comment

Every time I write one of these posts, I’m impressed by the volume and quality of writing that happens in our industry on an ongoing basis. It’s truly an exciting time for web performance. I feel endlessly engaged by the dialogue that happens every day, and honoured to be part of it it.

This roundup (which includes links pulled from the Strangeloop WPO Hub), includes some increasingly refined thinking about mobile optimization, a handful of excellent tutorials and case studies (including some great new presentations from Velocity EU), and some revolutionary browser developments.

But my favourite link is this first one…

The best link of Q4

Retailers need for tech speed
Does it tell us anything new? No. But I’ve lost count of how many people I’ve forwarded this two-minute segment on CNBC’s “Power Lunch” — which discusses the importance of speed for e-commerce sites, particularly for mobile users, during the holiday shopping season. For me, this shows that site speed has finally jumped into the mainstream. I’m excited to see how this attention snowballs in 2012.

Mobile

Mobile UI Performance
This slide deck from Estelle Weyl’s excellent presentation at Velocity EU gives an overview of mobile performance challenges, why we need to address them differently than we deal with desktop sites, and detailed tips on how to do just that.

Performance Automation 101
This slide deck from Jeroen Tjepkema’s Velocity EU presentation explains what performance automation is, how it works, and why it’s the only viable solution for dealing with the challenges of mobile device/browser fragmentation.

HTML5 Techniques for Optimizing Mobile Performance
Great post on HTML5 Rocks: ”In this article, we will discuss the bare minimum of what it takes to create a mobile HTML5 web app. The main point is to unmask the hidden complexities which today’s mobile frameworks try to hide. You will see a minimalistic approach (using core HTML5 APIs) and basic fundamentals that will empower you to write your own framework or contribute to the one you currently use.”

Mobile Performance Manifesto
Love this post from David Calhoun itemizing and describing mobile performance best practices.

Tools

How WebPagetest works
If you’ve ever wondered how exactly WebPagetest gathers performance data from the various browsers it simulates, this is great post from Pat Meenan in which he cracks the hood of WebPagetest and explains all that.

Mobile Perf Bookmarklet
Steve Souders offers one mobile bookmarklet to rule them all: a new “master bookmarklet” that lets you install a handful of common debugger and profiler bookmarklets in your mobile broswer in one step.

Is Synthetic Monitoring Really Going to Die?
Alois Reitbauer asks: “Will User Experience Management using JavaScript agents eventually replace synthetic monitoring or will there be a coexistence of both approaches in the end?” As you might guess, the answer is not cut and dried.

Case studies, how-tos, and other research

Diagnosing Slow Web Servers with Time to First Byte
Much as it pains me to admit it, from time to time performance pains aren’t caused at the front end. Performance expert Andy King gives some good tips on how to use the time to first byte metric, as displayed on a waterfall chart, to help diagnose a slow server.

The art and craft of the async snippet
Stoyan Stefanov examines the topic of asynchronous code “from the perspective of a third party – when you’re the third party, providing a snippet for other developers to include on their pages, be it an ad, a plugin, widget, visits counter, analytics, or anything else.”

Why loading third party scripts async is not good enough
We talk about asynchronously loading third-party snippets as if that’s the sole cure for performance pains, but in this case study, Aaron Peters reminds us that sometimes it’s okay to defer their loading until after onload.

Fast Loading JavaScript
Slide deck from performance consultant Aaron Peters’ great Velocity EU presentation: “A walk-through of several JavaScript loading techniques with a characteristics table for each and at the end a decision tree to help you decide which technique to use.”

How Downtime Financially Impacts Top Ecommerce Websites
Compelling infographic showing how downtime affected the Internet Retailer 500 in 2010. Includes the estimate that downtime resulted in more than $300 million in lost revenue for the IR 500.

Testing for Frontend SPOF
Excellent post from Pat Meenan in which he simulates third-party outage with a blackhole server in order to demonstrate — via WebPagetest-generated video — how that outage slows down or disrupts page load.

Browsers and connectivity

SPDY of the Future Might Blow Your Mind Today
Great post (“definitely for protocol geeks”) by Google software engineer Mike Belshe on SPDY’s evolution and how Kindle Silk is taking it beyond other browsers.

Chrome Fast
Slides from Google software engineer Tony Gentilcore’s excellent presentation at Velocity EU, in which he gives an overview of the Chrome platform and explains what makes Chrome fast.

Report reveals drop between peak and off-peak surfing
No big surprise, but a good reminder (especially if you rely on synthetic testing) that real-world performance is a nebulous thing: UK study finds that web speed is up to 69% slower during evening peak time.

The end of an era: Internet Explorer drops below 50% of Web usage
Mark the month and year. November 2011 was the first time in more than a decade that Internet Explorer’s share of global browser usage dropped below 50%.

Opinion pieces

Your CDN is not a silver bullet for web performance
In the e-commerce and SaaS world, the two most common causes of poor web performance are third-party content and server-side processing. Neither of these bottlenecks are addressed by loading static content from a closer location via a content delivery network.

Why you have less than a second to deliver exceptional performance
dynaTrace’s Alois Reitbauer writes: “Being exceptionally fast is becoming the dogma for developing web applications. But what is exceptionally fast and how hard is it to build a top performing web site?” I like posts like this because they remind us what the fundamental questions are that our industry is trying to address.

If you have any other great links to share, let me know in the comments. And if you’re looking for more great links, we have hundreds — sorted by topic, industry, and type — over in the Strangeloop WPO Hub.

Related posts:

By: Josh Bixby Category: Browsers, Case studies, Ecommerce, Industry news, Mobile, Research, Third-party content, Tools Tags: , , , , , , , , , , , , ,

Speed tests: How fast are China’s top websites?

28 Oct 2011

2 Comments

A couple of months back, Strangeloop released a report about the load times of North American and European sites in China — arguably the fastest growing e-commerce sector in the world. We found that the average load time for these sites was just over 16 seconds.

In my post about the report, Steve Souders asked a great question: To make the case that luxury sites need to be faster, it would be great if you could compare their load time (16.2 seconds) with the load time of popular Chinese sites.

Steve’s question has been percolating on a back burner (I have a lot of back burners), and I was reminded of it when looking at Mary Meeker’s slides from her presentation on internet trends at the recent Web 2.0 Summit.

This slide in particular jumped out:

Of the top 25 websites, six are based in China. While none of these sites come even close to the top four U.S. sites in terms of profitability and valuation, it’s important to remember that this is an online marketplace that is still in its babyhood. By 2015, China’s e-commerce base is expected to increase by almost 400%. It’ll be interesting to see what this table looks like then.

But getting back to Steve’s question, I thought this would be a good time to run some tests and see how China’s top sites compare to the sites we tested in our study. (I used the same parameters: Testing via WebPagetest‘s location in Jiangsu, China, using Internet Explorer 7.)

Page load times of leading Chinese websites

Website Load time (seconds)
 CDN detected?
Sina.com 20.818 No
Ctrip.com 2.554 Yes
Alibaba.com 4.877 No
163.com 21.139 No
Tencent.com 27.468 No
Baidu.com 1.706 No
Average Chinese site
 13.102 17%
Average NA/EU site
 16.2
 29%

A few thoughts

  • This is a tiny sample size, of course, but it’s still an interesting snapshot of how China’s top websites fare when it comes to performance.
  • There’s a clear performance divide between verticals, and these correlate closely to the same performance divide I’ve observed in western-based sites. The news sites — Sina and 163.com — experienced 20+ second load times. The e-commerce sites — Ctrip (travel) and Alibaba (retail) — were much faster, with sub-5-second load times. The search site Baidu was the fastest of the bunch, while, ironically, the internet service provider Tencent was the worst. (I suspect that competition isn’t a concern for them.)
  • Only one site, Ctrip, uses a content delivery network.
  • And returning again to Steve’s question, while the overall average load time of just over 13 seconds wasn’t much lower than the 16 seconds we recorded for western sites, it should be noted again that the top e-commerce sites did load in under 5 seconds. From this, we can deduce that leading e-tailers in China are taking performance seriously — which means that western e-tailers need to do the same in order not to get left behind in the global marketplace.

Related posts:

By: Josh Bixby Category: Ecommerce, Page speed tests, Research Tags: , , , ,
1 2 3 4 5 6 7 8