In defence of Blaze

I follow a lot of blogs, but I rarely read comments, especially on contentious posts, because people can quickly lose all sense of decency and get nasty. A recent example of this: a couple of weeks ago, Blaze released its study of mobile browser performance — announcing that the iPhone browser is 52% slower than Android — and came under heavy fire from many people.

As it turned out, some errors were made in this study. I won’t get into the particulars here (Blaze has been forthright about it in their own post), but these errors triggered a flood of vitriol, which I think the folks at Blaze have handled exceptionally well.

Measuring real-world web performance is a very, very difficult task. There are countless variables involved, and these variables are in a state of constant flux. I’m not saying this to issue a “get out of jail free” card for any company that takes on an ambitious performance study.  However, I am saying this to point out that this is why there are so few ambitious web performance studies. But here’s the thing: our industry needs these ambitious studies. We need to take studies like Blaze’s for what they are: a first step in our strategy for gathering data.

As a community, I think we need to stand up and support the fact that companies like Blaze make the effort to put studies like this together, providing the opportunity for the scientific process to take hold. In their original post, they spelled out their methodology quite clearly, thus making it easy for people like you and me to replicate their study – and if we choose, to start with a different hypothesis and prove Blaze’s hypothesis wrong.

I don’t believe that the people at Blaze had any intention of misrepresenting the truth, as they’ve been accused of by some. They’ve admitted their mistakes, corrected their post, and broadcast their corrections through the same social media channels they used to broadcast their original study. This deserves respect.

Reading the negative comments and press directed at Blaze makes me a bit skittish about releasing data myself. I’m sure I’m not alone in feeling this way. Our industry has come a long way in just a few years. We need an open and collaborative environment in order for us all to continue to make forward progress.

Related posts:

8 thoughts on “In defence of Blaze

  1. I think anyone who has anything to do with web performance appreciates the study…

    Haters gonna hate… nothing can be done abt it…

  2. It’s easier to offer grace to others when you have struggled yourself. I agree this stuff is hard and Blaze probably deserve a break. Thanks for keeping it classy. We’re lucky to have people like you in our industry.

  3. Joshua,

    Here’s the mistake Blaze made… transparency. I’ve been in the performance business for a long time. We learned with Mod_Gzip that the only way for people to believe it was to release the application.

    People want to “replicate and verify” the results that are being posted. Blaze doesn’t allow that. Sure they explained their methodology but you still cannot see behind the curtain.

    And as the bard says, therein lies the problem. Interestingly enough we haven’t had one negative comment on our Mobile Performance solution. You can download the software, install it on your device and run the tests for yourself.

    Performance is a brutal world. It’s ruled by the slide rule geeks. And they’re merciless. If they cannot replicate your results “under their conditions” then you’re in for a world of grief.

    Blaze will get better, but unfortunately they’ve got an up hill battle.

  4. This is exactly what I felt when I saw all the negative talk about Blaze. I wanted to write a post on the same lines as yours, Thank you Joshua for writing this.

    I second Sajal’s comment. Only people who work on web performance will appreciate the work done by Blaze(and lot of other such companies).

    The only part I would have added to the post is about dealing with Big names. It’s a known fact that anything to do with Apple(more than Google) get’s so much attention. So this I think is a lesson for everyone out there to make sure we get it right & triple check stats before we go public.


  5. Joshua,

    Thanks for the post, it’s much appreciated.

    As you point out, we had no bias or agenda here, and were fully open about what we did and how. I guess once you reach enough people there are some in that group who prefer bashing to discussion. Luckily most of those are not in the performance industry…

    I’d like to think it won’t keep us from releasing future studies, but I definitely need a slight breather before the next one :)


  6. +1 for this post Joshua.
    Blaze’s study is valuable in many ways, one being the discussion it triggered.

    I hope Blaze conducts more studies and we see more discussion, hopefully in a respectful fashion

  7. Thanks for the positive feedback, all.

    Slightly unrelated but still interesting: this morning, I was reading this post, written by the Red Cross’s director of social media, Wendy Harmon, about a minor Twitter gaffe that they handled quite well before it could blow up:

    My favourite quote:

    “Every time I see a nonprofit or company using social tools, my brain reminds me that there’s no such being as nonprofits and companies – there’s only a network of people doing work under the same name with the same goals. Social media belongs to real humans doing a very human activity – connecting with one another over shared interests.”

    This really struck me as something that we all need to be reminded of regularly.

  8. Pingback: » MeasureMatters #2

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>