You are the worst judge of your web site’s performance. Here’s why.

16 Sep 2010

7 Comments

After writing about VCs and load time benchmarks a couple of days ago, I was struck by this comment made by Fred Wilson, which I kind of glossed over the first few times I read it:

“I think that power users sometimes have a bit of sympathetic eye to the challenges of building really fast web apps, and maybe they’re willing to live with it, but when I look at my wife and kids, they’re my mainstream view of the world. If something is slow, they’re just gone.”

In our community, I think this power user mentality is our worst enemy. It’s something we have to actively fight against — both in the people we work with and, more insidiously, in ourselves. And this fight is neverending.

Everyone here at Strangeloop is a power user. Our clients are power users. Our partners are power users. You, by virtue of reading this post, are probably a power user.

On one hand, we are the people who are most likely to get, in theory, that faster is better. But in practice, I’m willing to bet that we’re more forgiving of slow sites and apps than we should be, to our peril. Here’s why.

We have the best tools.

Chances are, you’re using a late-model browser. (In fact, Google Analytics tells me that, if you’re reading this blog, there’s only a tiny chance you’re using anything older than IE8.) You’re using a souped-up computer on a snappy connection. You’re getting an elite user experience that is shared by such a statistically tiny minority that it’s pretty much irrelevant.

(Remember this video? Last year, Google did man-on-the-street interviews and asked random people what a browser is. Ninety-two percent didn’t know. These people aren’t dumb. They’re people who have better things to do than obsessively follow new browser releases. These people are your real users.)

We’re too sympathetic to the trials of development.

Building a good site or app is HARD. It needs to be attractive, interesting, engaging, useful, persuasive, entertaining, robust and, of course, fast. Those of us who have been through countless development cycles know that this hard work is being done, for the most part, by good, smart people who really do care about building great products. Because of this, and despite ourselves, we turn a blind eye to mediocre performance.

Mainstream web users don’t care about the pains of web development. Why should they? If you buy a car and it ends up being a lemon, do you take a moment to reflect on the fascinating complexity of the internal combustion engine and how hard it must be to do the repetitive assembly-line work of assembling a hundred of these amazing machines every day? Of course you don’t. You just want your car to take you from point A to point B without exploding. This is a very reasonable expectation.

We know how to cheat.

If I’m on an ad-driven site that’s taking too long to load, I hit the refresh button. I know that the slow load time was probably caused by a badly designed ad on a poorly optimized page, and that refreshing the page will probably launch a new ad that will load faster. You probably know this trick and have a bunch of other workarounds in your arsenal. Regular web users do not.

The average site visitor won’t hit refresh. He or she will wait and wait for the page to finally load… or bounce. I’m always touting the stat that says that up to 57% of online consumers will abandon a site after waiting 3 seconds for a page to load, but sometimes even I have a hard time believing it. But numbers don’t lie, which leads me to my next point…

If you can’t trust your own judgment, what can you trust?

Now that I’ve established that our gut instincts are completely useless, here’s what you can rely on:

  1. Real-user performance monitoring — Whatever tool you use, whether it’s something like Webpagetest or one of the solid RUM products on the market, you need to constantly collect data about how your users experience your site. (See this post for more information on how to do this.)
  2. Track your metrics — Try some good old-fashioned A/B testing. Accelerate your site for half your traffic, and let the other half visit the unaccelerated site. Let this test run for a few days or weeks. Track metrics like page views, bounce rate, time on site and, of course, conversions and revenue. Numbers are truth.
  3. Stay current with third-party research — Stay up-to-date with research being done by other organizations. Reading widely will give you a good overall snapshot and help you spot trends. (You can start with my stats cheat sheet.)

Related posts:

By: Joshua Bixby Category: Standards and benchmarks Tags: , , ,

Looking for VC funding? Then your website needs to be at least this fast.

14 Sep 2010

4 Comments

Last spring, Steve Souders wrote this great post explaining why web performance optimization (WPO) is poised to become the next SEO. In it, he cited top tech venture capitalist Fred Wilson, principal of Union Square Ventures. In this speech at the Future of Web Apps conference, Fred said:

First and foremost, we believe that speed is more than a feature. Speed is the most important feature. If your application is slow, people won’t use it. I see this more with mainstream users than I do with power users. I think that power users sometimes have a bit of sympathetic eye to the challenges of building really fast web apps, and maybe they’re willing to live with it, but when I look at my wife and kids, they’re my mainstream view of the world. If something is slow, they’re just gone.

We think that the application has to be fast, and if it’s not, you can see what happens. We have every single one of our portfolio company services on Pingdom, and we take a look at that every week. When we see some of our portfolio company’s applications getting bogged down, we also note that they don’t grow as quickly. There is real empirical evidence that substantiates the fact that speed is more than a feature. It’s a requirement.

You heard it. Speed is #1. Those are fighting words. I like fighting words. I took them as a challenge to see how fast Union Square Ventures’ portfolio actually performs. I picked five of the most easily recognized names in their portfolio and ran some performance tests:*

Website Page load time (U.S.) Page load time (international)
Stack Overflow 3.304 5.546
Etsy 3.982 6.394
Foursquare 4.536 16.532
Tumblr 4.907 5.630
Meetup 5.874 8.515
Average 4.5206 8.5234

Turns out, Union Square walks the walk. Their average domestic load time was about 4.5 seconds, considerably faster than the Fortune 500 benchmark of around 7 seconds. And their average international page load time came in at just over 8.5 seconds, somewhat speedier than the Fortune 500 benchmark of approximately 9.5 seconds.

After running these tests, I considered testing other leading tech VCs to see how they all compared, but after running just a handful of tests of their portfolio apps, realized that all of them were going to come in at around the same averages. My purpose today isn’t to start a “whose apps are fastest” debate. Instead, I want to highlight the fact that speed is, in fact, a real priority, one that business leaders are taking seriously. If you want your site or app to even be considered for funding, you’d better make sure it’s fast.

I would also argue that it isn’t just enough to aim for the benchmark. Your mantra should be: There’s no such thing as fast enough. Note Fred’s observation:

When we see some of our portfolio company’s applications getting bogged down, we also note that they don’t grow as quickly. There is real empirical evidence that substantiates the fact that speed is more than a feature. It’s a requirement.

As an aside, if you don’t already read Fred’s blog, you should consider adding it to your feed. Some excellent posts there.

*All tests done using Webpagetest. IE7 on a DSL line via Dulles, VA, and Sydney, Australia.

Related posts:

By: Joshua Bixby Category: Standards and benchmarks Tags:

UPDATE: Everything you wanted to know about web performance but were afraid to ask

9 Sep 2010

0 Comment

I’m cleaning out my old bookmarks and rediscovered a Forrester report from earlier this year, The Impact of Poor Web Site Performance in Financial Services. I’ve added the following key bits of data to my performance stats cheat sheet:

  • No activity on the web approaches the frequency of online account access.
  • Web site performance is second only to security in user expectations. Web site performance ranks above even functions like single sign-on or a website that is easy to use.
  • 56% of online bankers and brokers expect web pages to load in 2 seconds or less; this is far above the 47% of consumers who are just shopping online.
  • Poor website performance leads to dissatisfaction more often than any other factor. Sixty-four percent of online US bankers and brokers have had a dissatisfying experience when servicing their accounts. Web performance is far and away the biggest reason for this dissatisfaction.
  • As online tasks get more urgent or complex, online users are less likely to try later and more likely to move to more expensive channels to complete the transactions. Fifty-six percent of online bankers would move to offline channels to ask a general account question; 54% of online brokers would move to offline channels to trade investments if the website was unavailable or slow to respond.
  • The ultimate effect of poor performance is a decrease in willingness to recommend a firm, with 48% of online bankers and brokers saying that poor performance had an impact or significant impact on their likeliness to recommend a firm’s services to a friend or family member.

Also, in the time since I started creating this post, Gomez released a new commissioned report called When Seconds Count. It says that:

  • Nearly one-third (32 percent) of consumers will start abandoning slow sites between one and five seconds.
  • 39% say speed is more important than functionality for most websites, while only one in five rank greater site functionality as more important.
  • More than half of mobile users expect websites to load as quickly, almost as quickly or faster on their mobile phone, compared to the computer they use at home.
  • More than a third of mobile users (37%) said they would not return to a slow site, and 27 percent would likely jump to a competitor’s site.

Related posts:

By: Joshua Bixby Category: Ecommerce, Mobile, Research, Standards and benchmarks Tags: , , , , , , , , , ,

Google changes the game… again

8 Sep 2010

7 Comments

While today’s big Google Instant announcement doesn’t directly affect what we’re doing here at Strangeloop, it’s still incredibly exciting because it validates our core value proposition: Speed matters.

When Google (or Google!, if you want to be a stickler) first launched back in 1998, it took a radical new approach toward search engine ranking. At the time, other engines ranked results according to how many times the search keywords appeared on the page. But Google’s Larry Page and Sergey Brin theorized that search results would be more relevant if they were determined by how many other sites (and the authority of those sites) linked to a page.

While page relevance has no doubt been a huge factor in Google’s success, it’s arguable that this success is also due to the fact that — between its sleek interface and zippy performance — it has always been committed to delivering a fast, user-friendly experience. Before Google Instant was unveiled, the average search query on Google took just under one second, which is pretty great. I’ve been doing all of my searches today via Google Instant, and my slowest result has been 0.3 seconds. Super great.

But great as all that is, it’s not the real game changer. In my opinion, the real game changer is this:

Google is going to fundamentally change how people search.

Think about it:

You’re typing your search terms, and you (almost) instantaneously start getting visual feedback. While scanning this feedback, you get ideas for adding new terms. You still haven’t left the search field, but as you’re adding and refining your keywords, your search results are getting more and more relevant until you nail the exact link you need. Again, all without leaving the search field.

So, good for you. According to Google’s estimates, you’re shaving 2-5 seconds off every search query, which will ultimately save something like 350 million hours of total user time in a year. But beyond all these attention-grabbing numbers, what does this mean — in practical terms — for site owners?

Here’s what I think:

Eventually, page 2 search results are going to be irrelevant.

We’re not going to be able to pat ourselves on the back for getting our sites listed on page 2, because users are going to get everything they need from page 1. Page 1 will be the new bar. The companies that hit that bar will be the companies that are ready to squeeze every last ounce of performance from their sites.

I’ve read a few tweets and blog posts where Google is taking some heat for its so-called “myopic focus on response times,” but I think these people are missing the point. Google has found a truth that it is exploiting to the max: When it comes to the web, faster is better, and there’s no such thing as “fast enough”. Saying otherwise is what’s myopic.

Related links:

By: Joshua Bixby Category: Industry news, SEO Tags: , , , ,

What is dynamic site acceleration? And is it worth paying for?

2 Sep 2010

6 Comments

While sitting at one of those executive-takes-out-prospective-client-for-a-very-expensive-steak-dinner gigs during Velocity this year, I was struck by the observation of my dinner companion. It was his first time at Velocity and he was surprised and disappointed that CDNs were not a bigger part of the discourse.

His words:

“How can ‘use a CDN’ be one of the top three performance optimization techniques recommended by all of the tools, and yet  have no visibility at Velocity? There were no sessions about CDNs, and none of the big vendors showed up. This conference is missing a key component.”

I have been thinking about this since Velocity, and I agree that our community should try and wrap our arms around the CDNs and involve them in the debate.

Why should anyone pay a premium for dynamic site acceleration?

Most of the big-name CDNs offer their own dynamic site acceleration (aka “whole site acceleration”) products: Limelight Site, Cotendo, Akamai DSA and WA, and CDNetworks Application Acceleration. I keep getting questions about how these products help and why companies should pay a premium for these services.

I have tried in vain to find a simple description of how these technologies work and the benefits of each. Here is my best attempt.

First, let’s define the space.

In my opinion, the key defining element of dynamic site delivery is the idea that the request for the page AND all of the page objects go to the CDN. This is very different from small object CDN delivery, in which the request for the page is sent directly to your datacenter (what the CDNs call “the origin”) and the browser requests objects (CSS, JS, SWF, images) from the CDN.

After this defining element, which is shared by all products in this category, the world gets more complicated and confusing.

In my simple world, the market now splits into two major categories:

  • Those that can cache HTML at the edge.
  • Those that can cache HTML at the edge AND provide some level of acceleration for delivering the HTML from the origin to the edge, in case it can’t be cached at the edge.

Products that can cache HTML at the edge

Both categories can cache HTML at the edge. This means that when you make a request for a static page, the closet’s POP (point of presence) will respond with the HTML, without having to go back to the server.

If we look at this from a waterfall perspective, HTML caching will help make the orange bar smaller (initial connection time), make the green bar smaller (time to first byte), and perhaps help the blue bar (content download), but probably not do much for the HTML only, unless you were doing no compression before using the service. I’ve also seen the DNS lookup get better when these services are used, but that’s not because of HTML caching: it’s because these services have a pretty robust DNS infrastructure that ultimately helps DNS lookup times.

All this is assuming that your user is close to a POP and your HTML content is in the CDN cache (both of which are assumptions you can’t always make, especially for long tail content).

It is important to pause here and note the following:

  • Most of the customers I interact with have very dynamic pages and their content cannot be cached at the edge, notwithstanding the amazing marketing done by CDNs around archaic technologies such as edge site includes (ESI).
  • As I have mentioned in the past, most customers have a front-end problem and this feature only addresses the back-end problem. However, every second counts, so if you can take advantage of dynamic/whole site acceleration, then you should investigate it.
  • The benefit of static HTML caching will be enhanced by latency, packet loss and jitter. Your users in, say, rural China will get more of a benefit from this than your users in the US, if your server are in the US.

Products that can cache HTML at the edge AND provide some level of acceleration for delivering the HTML from the origin to the edge

Now imagine a world in which the HTML must come from the origin servers, i.e. your pages are dynamic and you can’t cache them at the edge. If your pages are dynamic and you can’t cache them, then the performance benefit of sending your traffic to a location very close to the user and then over a the CDN’s network to the origin server is threefold:

  • The CDN can dramatically reduce the DNS lookup time (remember you can also get this by looking at a DNS service like Ultra DNS.
  • The CDN can keep connections open between its POPs and save on the initial connection time.
  • The CDN can make tweaks at the network layer (TCP/IP) to ensure the highest rate of data transfer and, in the case of packet loss, ensure efficient recovery. This will result in faster content download time for the HTML.

In other words, the CDN can build a really big robust highway between two locations anywhere in the world and send data quickly between POPs. If we look at this from a waterfall perspective, when comparing this to going straight to the origin for the HTML, we’ll see improvements in the teal bar (DNS lookup) and orange bar (initial connection time), and we should see improvements in the blue bar (content download), since the service promises to deliver HTML over their network more quickly.

As I mentioned above, the benefits of dynamic site acceleration are most visible in a waterfall taken from a location with high latency and other network-related problems. For example, here is a page tested from San Jose versus a page tested from China.

San Jose:

China:

Obviously, content download is going to be worse in Asia, but the TCP/IP tweaks that dynamic site acceleration can provide will help much more in the second waterfall than in the first.

Other features that don’t have much to do with acceleration but help sell the product

Bundled into dynamic site acceleration are a number of other ostensible value-added services:

  • Offload: Offloading all of your traffic to a CDN means you buy less boxes and have less hassle. Ensure you understand the cost-benefit here, as this is very expensive hosting.
  • Small object caching: This is always bundled into the DSA pricing. Make sure you know how much the DSA part is going to help vs the small object part. On its own, small object can be 10 times cheaper.
  • HTML Compression: Obviously this helps with performance, but you can already get this for free on your servers. If you run a big shop, you should already have an ADC with this capability.
  • Availability features: Ability to show a pulse when the customer’s route is blocked or your servers are dead.
  • Security: Features like DDOS and web application firewall are sexy, but make sure they don’t duplicate what you already have.
  • PCI compliance: Ensure you understand what this means and if you need it.

So what to do with this information?

Understand why you need whole site delivery. Buying it for security, scalability and offload is a very different decision than if you want it for acceleration. It is very common to lump the benefits of what small object delivery and dynamic site delivery will do for you into one category. You need to separate them, because dynamic site delivery can be up to 10 times more expensive.

Your tasks

  1. Identify why you might be interested in whole site delivery. If acceleration is an important factor, proceed to the next tasks.
  2. Determine if you can cache your HTML for any meaningful period of time (3+ hours). If you can, then try it yourself and then compare the benefit of a dynamic CDN.
  3. Get waterfalls from different locations using Webpagetest and check out the HTML bar (usually the first bar) and see if reducing the DNS lookup time (teal bar: assume 50%), the initial connection time (orange: assume by 60%), the server think time (green: assume 80%, only if you can cache your HTML)) and the download time (blue: assume 5% in North America and Europe and 50% in other parts of world) actually brings you a benefit. (I’m making a broad assumption that you already have compression turned on. If you don’t, you should.)
  4. Research the different companies, or email me. (I know them all and would be happy to help.)
  5. Pick a few to try. Make sure you try both small object and dynamic site delivery.
  6. Test each vendor yourself using an open-source tool likeWebpagetest. Be wary of the tests the vendors send you as they have often gamed the system by putting test machines next to the edge caches and/or kept things in cache much longer then they would in the real world.

My opinion

I’ll be honest. I remain a skeptic.  As an acceleration play, I see the cost of dynamic site acceleration far outweighing the benefit. That’s a lot of money to pay for the small incremental performance gain you get on the HTML for most sites, especially if you’re only in North America and Europe. There are some benefits of DSA, though, in areas of high latency, and for fixing general network problems.

I do see a lot of potential for this sort of technology. I think it has some promise, but there’s been very little innovation in the last 3+ years. Nothing exciting has come out since Netli.

I’d like to be more convinced, so if anyone has real-world performance data with compelling evidence that the performance gain is significant enough to be worth the price, I’m all ears.

Related posts:

By: Joshua Bixby Category: Tools Tags: , , , , , , , , ,
1 2