Delivering Performance: How does your content management system score?

31 Aug 2010

11 Comments

Last week, I was talking with a client who is on the hunt for a ASP.NET-based CMS, and he asked me what I know about how content management systems affect site speed. It’s a really good question.

As a former executive in the CMS space, I know we always took the “we just sell guns, we don’t kill people approach.” In other words, we just built the containers and the users put in the content, so how could we be reponsible for site speed? However, I knew that the choices we made in the underlying CMS system did have a dramatic impact on site speed.

So over the weekend, I decided to try a very unscientific experiment. I singled out five ASP.NET content management systems — Sitefinity, DotNetNuke, Ektron, Kentico, and Sitecore — and tested* their page load times along with the page load times of four of their case study clients. I tried to choose a similar cross-section of sites from each, wherever possible: ecommerce, not-for-profit, corporate, and educational.

This table shows the load time for each site, with the averages for each CMS solution provider in the column on the far right, listed in order from fastest to slowest. Click through to see the test results for each site.

SITEFINITY Quaker Oats Discover BC AT&T Web Hosting The Trump Network AVERAGE LOAD TIME
3.113 7.409 6.747 3.000 4.677 4.989
DOTNETNUKE Pier 1 Imports Marriott Orlando Midas UNPAN
7.538 7.379 5.309 7.052 8.927 7.241
EKTRON NYC Ballet Special Olympics University of Virgina MGM Casino
8.528 6.464 7.246 7.412 8.136 7.557
KENTICO Habitat for Humanity Bayer Health Care CARES: Kids Fly Safe OASIS
4.088 11.008 11.472 7.405 4.942 7.783
SITECORE Boy Scouts of America Beliefnet Canadian Cancer Society WebTrends
8.465 8.141 9.629 10.346 11.827 9.681

Note that none of the sites fared particularly well. The average page load time was 7.45 seconds, which is not blazingly fast by any standard. But what was interesting to me was the variance in the averages: there was an almost 5-second gap between the company with the fastest average load time and the company with the slowest. At the outset of this little experiment, I was expecting the averages to fall within a much smaller spread. This greater variance seems like it might be an indicator of… well, something.

Some mitigating factors

Now, as I’ve already said, this was a pretty unscientific study — like most studies that take place between 10 pm and 2 am on a Saturday night. In all fairness to the CMS providers I tested, there are a few variables they have no control over, including:

  • Randomness — Maybe I randomly chose Sitefinity’s four best sites, and Sitecore’s four worst. It’s possible. Though I do think it’s interesting that top-ranked Sitefinity’s own website loaded in just over 3 seconds, while bottom-ranked Sitecore took more than 8 seconds to load.
  • How the CMS is used — If you’re committing no-nos, such as uploading massive unoptimized images or not enabling compression, the CMS can’t compensate for those things.
  • Third-party content — As I’ve mentioned in another post, third-party content is a wild card that your CMS can’t control.

But I do think there’s a kernel of truthiness even in my quick voodoo-science survey, enough to warrant asking a few questions of your CMS provider or in-house CMS developers.

If you use a CMS, or are planning to, three questions to ask

  1. How does the CMS apply Yahoo and Google‘s performance best practices for things like optimizing caching, optimizing browser rendering, and minimizing roundtrips and payload?
  2. If the CMS does apply some of these best practices, what is the provider’s process for updating your CMS software to accommodate new best practices as they emerge?
  3. How does the CMS deliver performance internally? In other words, how fast will its pages load for you when you’re the one out there actively pushing new content? This study by Aberdeen Group states that issues with application performance are affecting overall business revenues by up to 9%. That includes CMS performance. I know from painful experience that a slow CMS is an excruciating performance killer.

A giant caveat

Having said all of this, I’m very aware that it’s overly simplistic to suggest that you choose your CMS provider based solely on its ability to deliver performance. You need to take a lot of other things into consideration: core functionality, usability, support, price. All I’m proposing is that when you’re creating your CMS scorecard, add “performance” to your list of criteria.

A great big juicy business opportunity?

With site speed emerging as a critical business requirement, is this an opportunity for a CMS solution provider to step up and deliver a product that can promise to optimize pages for faster performance? It seems to me that it is.

*All tests were conducted using Webpagetest: IE7 on DSL via the server in Dulles, VA.

Related posts:

By: Josh Bixby Category: Standards and benchmarks Tags: , , , , ,