Delivering Performance: How does your content management system score?

31 Aug 2010

11 Comments

Last week, I was talking with a client who is on the hunt for a ASP.NET-based CMS, and he asked me what I know about how content management systems affect site speed. It’s a really good question.

As a former executive in the CMS space, I know we always took the “we just sell guns, we don’t kill people approach.” In other words, we just built the containers and the users put in the content, so how could we be reponsible for site speed? However, I knew that the choices we made in the underlying CMS system did have a dramatic impact on site speed.

So over the weekend, I decided to try a very unscientific experiment. I singled out five ASP.NET content management systems — Sitefinity, DotNetNuke, Ektron, Kentico, and Sitecore — and tested* their page load times along with the page load times of four of their case study clients. I tried to choose a similar cross-section of sites from each, wherever possible: ecommerce, not-for-profit, corporate, and educational.

This table shows the load time for each site, with the averages for each CMS solution provider in the column on the far right, listed in order from fastest to slowest. Click through to see the test results for each site.

SITEFINITY Quaker Oats Discover BC AT&T Web Hosting The Trump Network AVERAGE LOAD TIME
3.113 7.409 6.747 3.000 4.677 4.989
DOTNETNUKE Pier 1 Imports Marriott Orlando Midas UNPAN
7.538 7.379 5.309 7.052 8.927 7.241
EKTRON NYC Ballet Special Olympics University of Virgina MGM Casino
8.528 6.464 7.246 7.412 8.136 7.557
KENTICO Habitat for Humanity Bayer Health Care CARES: Kids Fly Safe OASIS
4.088 11.008 11.472 7.405 4.942 7.783
SITECORE Boy Scouts of America Beliefnet Canadian Cancer Society WebTrends
8.465 8.141 9.629 10.346 11.827 9.681

Note that none of the sites fared particularly well. The average page load time was 7.45 seconds, which is not blazingly fast by any standard. But what was interesting to me was the variance in the averages: there was an almost 5-second gap between the company with the fastest average load time and the company with the slowest. At the outset of this little experiment, I was expecting the averages to fall within a much smaller spread. This greater variance seems like it might be an indicator of… well, something.

Some mitigating factors

Now, as I’ve already said, this was a pretty unscientific study — like most studies that take place between 10 pm and 2 am on a Saturday night. In all fairness to the CMS providers I tested, there are a few variables they have no control over, including:

  • Randomness — Maybe I randomly chose Sitefinity’s four best sites, and Sitecore’s four worst. It’s possible. Though I do think it’s interesting that top-ranked Sitefinity’s own website loaded in just over 3 seconds, while bottom-ranked Sitecore took more than 8 seconds to load.
  • How the CMS is used — If you’re committing no-nos, such as uploading massive unoptimized images or not enabling compression, the CMS can’t compensate for those things.
  • Third-party content — As I’ve mentioned in another post, third-party content is a wild card that your CMS can’t control.

But I do think there’s a kernel of truthiness even in my quick voodoo-science survey, enough to warrant asking a few questions of your CMS provider or in-house CMS developers.

If you use a CMS, or are planning to, three questions to ask

  1. How does the CMS apply Yahoo and Google‘s performance best practices for things like optimizing caching, optimizing browser rendering, and minimizing roundtrips and payload?
  2. If the CMS does apply some of these best practices, what is the provider’s process for updating your CMS software to accommodate new best practices as they emerge?
  3. How does the CMS deliver performance internally? In other words, how fast will its pages load for you when you’re the one out there actively pushing new content? This study by Aberdeen Group states that issues with application performance are affecting overall business revenues by up to 9%. That includes CMS performance. I know from painful experience that a slow CMS is an excruciating performance killer.

A giant caveat

Having said all of this, I’m very aware that it’s overly simplistic to suggest that you choose your CMS provider based solely on its ability to deliver performance. You need to take a lot of other things into consideration: core functionality, usability, support, price. All I’m proposing is that when you’re creating your CMS scorecard, add “performance” to your list of criteria.

A great big juicy business opportunity?

With site speed emerging as a critical business requirement, is this an opportunity for a CMS solution provider to step up and deliver a product that can promise to optimize pages for faster performance? It seems to me that it is.

*All tests were conducted using Webpagetest: IE7 on DSL via the server in Dulles, VA.

Related posts:

By: Josh Bixby Category: Standards and benchmarks Tags: , , , , ,

Web Performance Consulting: Five questions for Andrew King

23 Aug 2010

2 Comments

In the past month, I’ve had three major corporations ask for my opinion about independent web performance consulting: What is it? Is it worthwhile? How do you find a good consultant? I decided to take these questions directly to one of today’s leading performance consultants, Andrew King.

In addition to authoring the book Website Optimization (O’Reilly, 2008), a must-have for any performance library, Andrew is also President of Website Optimization, LLC. His company has provided performance consulting and optimization for sites like AOL.com, Time Warner, and WhitePages.com.

1. First off, what’s the value in engaging an outside performance consultant versus, say, doing a performance audit internally?

An outside consultant has seen more sites and has a more global perspective than someone from the inside. When you hire an outside consultant, I’ve found that management tends to listen and act more to the recommendations as well (even though the recommendations from an employee on the inside may be just as valid).

2. Performance consulting is a relatively new and unregulated field. If a company is looking to bring in a consultant, where would they even start to hunt for one?

Good question. The Velocity conference is a good place to start. We both attended it this year. Many consultants attend this event. I’d look for people with experience and a history of publishing quality articles and books (and tools). Google the terms you are interested in.

3. What kinds of questions should a company ask a prospective consultant to weed out the unqualified candidates?

How much experience do you have? Who are some of your clients? Can you show us case studies? Do you optimize websites as well as analyze them for performance? Can you handle ecommerce websites? Windows? What specifically do you do for a performance audit or optimization (front-end or back-end)?

4. Is there a common process that consultants follow — or should follow — when conducting a performance audit? As a corollary to this question, is there a standard toolset that should go along with this process?

Well, there is a common process that we follow. :) If it is a front-end audit (80% of the performance problems are on the front-end, according to Steve Souders’ book), we typically use YSlow, Page Speed, and WebPageTest.org for metrics and then dive into the code for before/after recommendations.

We look at the (X)HTML, CSS, JavaScript, multimedia, and server settings for areas of improvement. Many times you can replace old-style techniques (JavaScript-based menus with CSS-based, for example) and eliminate code altogether. Lately we’ve been providing optimized files for audits, which makes it a bit of a hybrid.

For back-end performance audits, we have different teams for Linux- and Windows-based sites. We look at the server setup, middleware, SQL queries, caching, old modules or code, etc. We provide recommendations in a similar format to the front-end audits. We also do a hybrid audit that includes both front-end and back-end analyses.

5. Say I’m a client who has hired a consultant. At the end of the process, what should I expect as a deliverable?

For a performance audit, you should expect a detailed report with baseline performance metrics (useful for comparison after subsequent optimizations or audits), executive summary, web page behavior (waterfalls etc.), and actionable recommendations for improvement.

Thanks, Andrew! If anyone has questions for Andrew, ask in the comments.

Related posts:

By: Josh Bixby Category: Consulting Tags: , , , , , , ,

The three biggest performance problems with third-party content (and how to fix them)

17 Aug 2010

10 Comments

[This post served as the jumping off point for a 30-minute webinar: Three Common Performance Problems Caused by Third-Party Content (And How to Fix Them).]

“Necessary evil” is too strong a phrase when it comes to describing third-party ads, analytics tools, and various other site widgets. But given how much negative talk has been centered around third-party content and web performance lately, it’s not that far off.

Traditionally we have divided the site performance problem into a front-end and back-end problem. It is becoming obvious to me as I speak to more and more customers that we need to refine this high-level taxonomy to include:

  • Front end (time that you control)
  • Back end
  • Third-party content

If your problems are at the front or back end, you have control over them. But if your problem is with third-party content, pretty much your only actionable strategy is to be smart about how you pick your vendors. (If you consider prayer an action, you can also hope and pray that your existing vendors make performance a priority. Meebo and Digg deserve special mention here for doing this, with great results.)

Most of the third-party problems I see fall into three categories:

Problem #1: Tools that promise cool new functionality, but kill your conversion gain

I’ll give you an example. At Strangeloop, we have an ecommerce client who wanted to implement a third-party recommendation engine on his site. The engine claimed to be “only one line of javascript” that would increase cart size by recommending related products. Sounds good. But that one line of code and its resulting roundtrips slowed down the site by 15%, and that slowdown leeched the conversion gains generated by the tool.

Solution: Conduct a cost-benefit analysis for all prospective tools.
This is something you should do any time you evaluate a new third-party app:

  1. Perform an A/B test of your site, with and without the tool, in a real-world environment. Generate waterfall charts for both tests, and identify how long the third-party objects take to load. Note these benchmarks.
  2. From the tool vendor, get the number for the average conversion rate bump experienced by other sites that use the tool.
  3. Using Aberdeen’s widely accepted performance stat that a 1-second page delay equals a 7% loss in conversions, calculate the potential net conversion gain or loss. For example, if a tool slows down page load by 2 seconds, that means a 14% conversion loss. But if that same tool promises a 20% conversion increase, then that’s a net gain of 6% (not including the cost of purchasing the tool).

Problem #2: Badly optimized ads

Here’s another client example. One company we helped did a performance audit and found that third-party ads comprised a whopping 50% of the calls to their pages. They were averaging 80 to 100 roundtrips per page because of the company that was serving the ads.

Solution: Think about speed when choosing your ad vendor.
When selecting a vendor, ask them how they optimize their ads to maximize performance. If they don’t have an answer for you, consider giving them a pass. (I know this isn’t always an option, and sometimes there are very good reasons to go with a company despite the performance of its content, but it’s worth keeping in mind.)

Problem #3: Badly optimized pages

I’ve talked before about how all browsers perform differently in terms of how they prioritize page elements, and about how loading elements in the wrong order can mean that your visitors’ eyeballs don’t land on the content you want them to notice. It’s worth repeating.

Solution: Prioritize page elements so that third-party content loads last, not first.
You need to look at how your key pages render across major browser types, identify which page elements are rendering first, and make sure those are the elements you most want your users to see. This might not always be possible – or desirable, if you rely on ad clicks – but it’s a strategy to bear in mind.

Further reading:

Whether you’re a developer designing third-party apps or a site owner who is choosing them, here are a few good links for further reading:

Related links:

By: Josh Bixby Category: Browsers, Third-party content Tags: , , , , , , , ,

Travelocity makes a good tool even better with new script for Webpagetest

12 Aug 2010

8 Comments

Chances are you already know that, in order to get a real sense of how your site performs, you need to test it using tools that simulate the real-world end-user experience. I talked about this a couple of weeks ago, and suggested Webpagetest as a very usable tool that delivers solid results.

After posting on this topic I got a lot of feedback, and one theme recurred. To paraphrase a customer conversation:

“It’s great to have one isolated test, but what we need is a continuous view into performance. I need hundreds of these results everyday — not just one.”

Anyone who has run dozens of tests on Webpagetest knows it is challenging/impossible to run and aggregate hundreds or thousands of web page test results. For instance, I took this picture yesterday as I was trying to put together a list of URLs for a customer that spanned multiple tests. At the height of the task, I was working in Notepad, Excel, and had dozens of WPT windows open.

Wrangling multiple=

I don’t think our community has a good answer to this problem. I know many of the metrics vendors are hard at work trying to simulate real-world scenarios or actually using real-world browsers, but these are fraught with challenges (a post that will be coming soon).

But I came across a cool innovation today that I wanted to share.

One of Strangeloop’s clients, Travelocity, has tackled this problem head on. Their Director of Development, Tony Perkins, created his own script that sits in front of his own instance of Webpagetest and automatically tests the same six landing pages every 10 minutes in perpetuity, then aggregates the results. At any given time, Tony can generate an easy-to-read chart that shows overall performance for one day or for any time span he specifies. He can even view side-by-side comparisons of how the test pages perform with and without Strangeloop’s Site Optimizer service. Pretty slick.

I say all of this not to be critical of Webpagetest, which is a fantastic tool (obviously, or else my company wouldn’t be a sponsor), and which has gotten even better with yesterday’s release of a new UI and some additional functionality. I say this to point out that front-end performance measurement tools that use real browsers and simulate the real world — particularly free, independent tools like WPT — are still in their nascency. Tony and Travelocity have proven that WPT is extensible, which is pretty exciting news for anyone who cares about fine-tuning tools so that they can get the exact data they need.

I asked Tony if he has any plans to make his script open source, and am happy to report that he does. I’ll pass along more info on this as it comes available.

Related posts

By: Josh Bixby Category: Tools Tags: ,

Never Take Your Eye Off the Ball: Four mistakes Shopzilla made so you don’t have to

10 Aug 2010

9 Comments

At Velocity 2009, Shopzilla became the poster child for web performance optimization when they reported some pretty powerful numbers. The company radically re-architected its entire online operation with an eye toward reducing infrastructure and improving performance. In the end, they sped up their average page load time from 6 seconds to 1.2 seconds and experienced a 7-12% increase in conversion rate and a 25% increase in page views.

Shopzilla’s huge quantifiable success became a rallying point for many of us, so you can imagine how excited I was to hear that the company’s lead architect, Tim Morrow, would be speaking at Velocity 2010 to update us on Shopzilla’s latest numbers. And you can imagine how surprised — and impressed — I was when Tim’s session ended up being a clear, frank discussion of how Shopzilla dropped the ball.

In the months following its enormous performance success, Shopzilla saw its page load times slowly creep from an impressive sub-2 seconds up to more than 5 seconds. Users started to complain about the site’s slowness, sending feedback like “I will not come back to this site again” and the always-cogent “It sucks.”

I’ve embedded the YouTube video of Tim’s presentation below (it’s only 11 minutes long and worth watching), but here are the key mistakes he identified:

  1. No front-end measurement. It’s not enough to fine tune your site. You need to continually test and monitor your performance.
  2. Constant feature development. Shopzilla introduces several new features every week, which adds up. Says Tim: “There’s a tension between features and performance. The added complexity and constant change tends to unravel performance.”
  3. Poorly implemented third-party content. There’s no getting away from ads and widgets, but these need to be either designed in such a way that they don’t eat up valuable bandwidth or prioritized so that they’re not the first page elements to load.
  4. Waiting too long to tackle performance problems. It’s easier and cheaper to deal with problems early in the development cycle.

I often cite the Golden Gate Bridge as an analogy for manually tuning site performance. As soon as you finish painting it, it’s time to start all over again.

But it pays off. After rallying a performance task force and tweaking performance on just a few key pages, Shopzilla saw a 0.4% increase in their conversion rate, which translated into just a two-month ROI.

Related posts:

By: Josh Bixby Category: Case studies Tags: , , , , ,