I follow a lot of blogs, but I rarely read comments, especially on contentious posts, because people can quickly lose all sense of decency and get nasty. A recent example of this: a couple of weeks ago, Blaze released its study of mobile browser performance — announcing that the iPhone browser is 52% slower than Android — and came under heavy fire from many people.
As it turned out, some errors were made in this study. I won’t get into the particulars here (Blaze has been forthright about it in their own post), but these errors triggered a flood of vitriol, which I think the folks at Blaze have handled exceptionally well.
Measuring real-world web performance is a very, very difficult task. There are countless variables involved, and these variables are in a state of constant flux. I’m not saying this to issue a “get out of jail free” card for any company that takes on an ambitious performance study. However, I am saying this to point out that this is why there are so few ambitious web performance studies. But here’s the thing: our industry needs these ambitious studies. We need to take studies like Blaze’s for what they are: a first step in our strategy for gathering data.
As a community, I think we need to stand up and support the fact that companies like Blaze make the effort to put studies like this together, providing the opportunity for the scientific process to take hold. In their original post, they spelled out their methodology quite clearly, thus making it easy for people like you and me to replicate their study – and if we choose, to start with a different hypothesis and prove Blaze’s hypothesis wrong.
I don’t believe that the people at Blaze had any intention of misrepresenting the truth, as they’ve been accused of by some. They’ve admitted their mistakes, corrected their post, and broadcast their corrections through the same social media channels they used to broadcast their original study. This deserves respect.
Reading the negative comments and press directed at Blaze makes me a bit skittish about releasing data myself. I’m sure I’m not alone in feeling this way. Our industry has come a long way in just a few years. We need an open and collaborative environment in order for us all to continue to make forward progress.