Are we optimizing our images like cavemen?

Years from now, developers will look back on how we use images today and have a good laugh at our expense. How we handle images now is the equivalent of caveman drilling holes in skulls to release demons.

The problem: Images are getting fatter, which is hurting performance.

The web is obese. This isn’t news. Images are the biggest culprit. This isn’t news, either. Pages are, by many accounts, getting slower. Again, not news.

According to the HTTP Archive, the average web page is 1292 KB, with 801 KB of that page weight — more than 60% — being taken up by images. The proliferation of images on the web isn’t going to change and will likely continue to increase. People like high-quality pictures, and that’s not going to change, either, especially in light of the advent of retina displays.

The solution: We need to get a lot smarter about how we treat images online.

There are some obvious things you should be doing to optimize images — choose the right format, make sure they’re progressive — but in my opinion, these are primitive first-gen approaches. Today, inspired by my recent podcast chat with Ilya Grigorik, I want to explore some big-picture ideas. (No pun intended… okay, pun sort of intended.) I chatted with some of the great minds here at Strangeloop, notably Shawn Bissell, one of our senior software architects.

We might not have all the answers, but let’s start by asking the right questions.

Why not just automate how images are formatted, so that they’re always in the optimal format?

Inappropriate image formatting is a common performance culprit. We see this literally all the time. You could spend a lot of time educating every single person in your company about the best way to optimize every image type, or you could do what Ilya proposes: somehow automate how images are formatted, so that they’re always saved in the optimal format (PNG, GIF, JPEG, etc.).

At Strangeloop, we’ve built this feature into our products with a treatment called Image Compression. We recompress the image into either a JPEG and a PNG based on a set of default settings (which are user-configurable, as well as automatic), and then we use the smallest version. The smallest version is almost always a JPEG, except for small PNGs or GIFs, which are usually smaller in PNG-8 format.

There are also a bunch of tools — several of which are reviewed in this roundup on Smashing Magazine — that you can use to optimize your image formats.

The challenge, of course, is that the definition of “optimal” is, of course, highly subjective. Using a quality setting that is too low can produce artifacts (blurriness or “jaggies”). Shawn tells me that he recently discovered a cool new algorithm for determining the structural similarity (SSIM) index, which mathematically computes the difference between the original and compressed images. This is something we are currently considering adding to our products.

If we could add/popularize more image formats, what should they be?

What we’re currently using:

  • Photos – JPEG, PNG-24
  • Low complexity (few colors) – GIF, PNG-8
  • Low complexity with transparency – GIF, PNG-8
  • High complexity with transparency – PNG-24
  • Line art – SVG

What we should also be using, but aren’t:

  • JPEG 2000 – This is a great format for variable compression for different regions of interest
  • WebP – Offers better compression for high-resolution images

What we need to develop:

  • A format similar to WebP that can produce even smaller image sizes for images with a extremely high PPI (pixels per inch). This would help a lot with the huge image quality/performance challenges that we’re going to start hearing a lot more about now that the new retina displays are hitting the market.

Why is it so hard to add brand-new, better image formats?

This is a big, hairy, complicated beast. People have been working to improve image formats for years, with mixed success. One of the problems with new formats is that they’re often proprietary and require a license. Another is that they’re sometimes subject to legal issues because of unclear intellectual property rights. Browser vendors understandably don’t want to pay for the license or get caught in a legal battle.

JPEF 2000 and WebP, which I mentioned above, are classic examples of useful-yet-neglected formats:

JPEG 2000 (introduced in the year 2000) is a good example of a format that is actually superior to JPEG. Compared with the regular JPEG format, JPEG 2000 offers advantages such as support for higher bit depths, more advanced compression, and a lossless compression option.The problem: nobody uses it. This is at least partially due to the fact that, while the JPEG 2000 license is royalty free, there may be “submarine patents” on the actual IP, which are problematic.

By the somewhat opposite token, WebP (introduced in 2010) is an open standard that gets great results (according to Google, WebP can reduce filesizes by up to 45%, which is phenomenal), yet it’s still only supported by Chrome and Opera — no doubt due to the fact that it was developed by Google.

The lesson to be learned here is that if you want to create the next great image format, you need to make it free, open, and totally vendor neutral.

Why is the HTTP Vary header a problem for CDNs?

Ilya and I touched on this very briefly in our chat, and a few people have since asked me to explain this further, so I want to take a minute to elaborate here. The Vary header is a problem for CDNs or caching proxies because they are required to keep a different version of the resource for each different value of the header specified in the Vary header.

The classic usage of the Vary header is to vary by Accept-Encoding so that the CDN would keep different copies of the webpage in its cache for different compression encodings (gzip, deflate, plain). This case isn’t a problem, but in order to serve a different image to different browsers, the Vary header needs to vary based on something browser specific. “Browser specific” usually means varying by the User-Agent header, but there are so many different versions of User-Agent headers (even for the same browser!) that CDNs are afraid that varying by User-Agent will cause severe cache fragmentation. So they only like varying by Accept-Encoding, and not by anything else. And some CDNs have even done away with that by implementing their own compression engines.

(At Strangeloop, we avoid this problem by having our product use the User-Agent header on the HTML request — typically not cached in the CDN — to rewrite all the URLs to the images to point at different versions of the images. This way the CDN cache only has one copy per version (which is optimal), and the correct version is controlled by the URL in the HTML, so the browser always downloads this correct version.)

Takeaways

This post just grazes the surface of this topic. How we handle images on the web is a massive, complex issue, but our main challenges can be pared down to this:

  1. We need to get a lot smarter about how we handle images on the web, because they’re only going to keep getting fatter.
  2. We need to embrace existing formats, such as WebP, that will make our images leaner.
  3. We need to develop a new format that can handle the emerging demands of retina displays.

Related posts:

14 thoughts on “Are we optimizing our images like cavemen?

  1. Automatic format selection is cool. Currently I’m using pngquant with max qualityqualityoption – it skips a file if it’s not good as png and then i can save it as jpeg.

  2. Josh, great post. A few quick comments, and questions…

    “The lesson to be learned here is that if you want to create the next great image format, you need to make it free, open, and totally vendor neutral.” – WebP is both royalty-free open source-source. The fact that its being developed at Google is beside the point — it’s one of our projects on the the Make the Web Fast initiative.

    On the UA + Vary front: agreed, that’s a dead end. However, Vary: UA is the wrong tool for the job. Really, we need UA: Accept. See my post here: http://www.igvita.com/2012/12/18/deploying-new-image-formats-on-the-web/

    Finally, curious: “we need to develop a new format that can handle the emerging demands of retina displays”. There is nothing inherently different about retina images, short of just having many more pixels. Why wouldn’t WebP fit the bill? I am reading your comment as implying that current image formats are not sufficient.

  3. With the invention of retina displays, vector based formats are becoming ever more important. Graphics should be done in SVG. It is small to download, and will look gorgeous on retina, and also when zoomed in.

  4. It would be a bit easier to optimize images if browsers implemented the full specs first (try to open an arithmetic encoded, a hierarchical or a 12-bits per component JPEG), IE6 had huge problems handling PNG transparency.
    Even today Adobe Photoshop does not produce true grayscale pictures and is unable to create Color type 3 (paletted) PNGs with more than one tRNS (transparency) entry.
    Mozilla killed JNG (an MNG spin-off that hold a JPEG image and a full alpha channel) to foster their own APNG (Animated PNG) that was not even closely related. https://bugzilla.mozilla.org/show_bug.cgi?id=88020
    I cannot believe they are not implementing WebP strictly for technical reasons… I’m really disappointed by Mozilla, WebP is definitely a good step forward.

    Way too many software is based on the same old libs (IJG libjpeg or libpng) but optimisation products that really stand out from the pack (PNGOUT, PNGWolf, JPEGmini) use their own compression engine or bring new ideas, the same goes for Deflate (of course zlib works, but compare its results against those of KZIP+kzip2gz).

    For text files its the same again, UTF-8 is the sole encoding recommended for HTML5, if you read a few technical papers about Unicode compression (BOCU-1, SCSU) you’ll notice that they state that bzip2 is the most suited algorithm to compress Unicode texts, is the Web moving toward broad bzip2 adoption or something especially tailored to handle UTF-8? No, it sticks with Deflate… whereas even ZIP archives can hold Deflate64, Bzip2, PPMd or LZMA compressed data nowadays.

  5. On the JPEG-2000 front, I recently wrote about this at work (http://blogs.loc.gov/digitalpreservation/2013/01/is-jpeg-2000-a-preservation-risk/) with more of an emphasis on long-term access and agree with the points in your conclusion but I think a far greater problem is the fact that there has not been a solid, fast open-source implementation. For years it was difficult to even get a copy of the specification or an official test suite, dramatically lowering the number of potential contributors.

    https://bugzilla.mozilla.org/show_bug.cgi?id=36351#c120 discusses what it would take to get a new format into Firefox: a high-quality open-source library, some level of security vetting and a solid rights statement. That’s daunting for a new format, particularly since many people have been willing to simply pay the marginal costs for larger PNG, JPEG, etc. or the development costs of something like http://www.jpegmini.com/ rather than trying to push the adoption boulder up hill until you reach critical mass.

  6. Pingback: Pingdom's weekend must-read articles #41

  7. Pingback: Tweet parade (no.08 Feb 2013) | gonzoblog

  8. > Mozilla killed JNG

    Who knew Mozilla holds such a great the power over Internet that it can just take a format and kiil it? Can google or microsoft kill a format?

  9. Joshua,

    I wrote a related article in Perfplanet calendar (http://calendar.perfplanet.com/2011/lossy-image-compression/), it looks like determining “optimal” is doable (and really worth it), my former collegue actually wrote an open source tool to do that and described his solution in detail: https://github.com/rflynn/imgmin

    And there is also JPEGmini (http://jpegmini.com/server) commercial vendor, they sell a server solution and recently release Amazon AWS-based solution that is available on pay as you go basis.

    This is a great breakthrough and brings hope to JPEG compression automation.

    - Sergey

  10. Pingback: Joshua Marantz (Google): Why erring on the conservative side is a good thing [PODCAST]

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>