Speed Perception

TL;DR: Please take the speedPerception challenge to help us confirm results about which web performance metrics best match human perception of speed.

Last summer, I was involved in a study called “SpeedPerception”, a large-scale web performance crowdsourced study focused on the perceived loading performance of above-the-fold content aimed at understanding what “slow” and “fast” mean to users. I am now involved in the second part of this study which aims to confirm (or refute) our findings.

SpeedPerception: the general idea

Traditional web performance metrics, like those defined in W3C Navigating Timing draft specification focus on timing each process along the content delivery pipeline, such as Time to First Byte (TTFB) and Page Load Time. SpeedPerception’s goal is to tackle the web performance measurement challenge by looking at it from a different angle: one which puts user experience into focus by focusing on the visual perception of the page load process. We show the user  sample video pairs of websites loading generated with http://www.webpagetest.org/ (WPT), and ask them which of the pair they perceive as having loaded faster.

In the first phase, we measured only Internet Retailer top-500 (IR500) sites in desktop size. Now we are testing whether the results we measured are true: in other words, do they only work for our IR500 sites on desktop? Will we get consistent results when testing  Alexa top-1000 (Alex1000) homepages? Will we see the same results if we test on mobile size screens with mobile lie-fi performance?

In this second phase, we’re testing both mobile and desktop versions of both IR500  and Alexa1000 website home pages. We’ve also added a way of measuring the user’s time to click so we can compare apples to apples.

The goal is to create a free, open-source, benchmark dataset to advance the systematic study of how human end-users perceive the webpage loading process: the above-the-fold rendering in particular. Our belief (and hope) is that such a benchmark can provide a quantitative basis to compare different algorithms and spur computer scientists to make progress on helping quantify perceived webpage performance.

Take the Speed Perception challenge !

How was SpeedPerception created?

Videos were created using  Patrick Meenan’s open-source WebPagetest (a.k.a WPT). We made 600+ videos of  2016 IR500 and Alexa-1000 home pages loading. The runs were done in February 2017. Videos were turned into gifs. Video pairs  were grouped using a specific set of rules to help limit bias and randomness. Everything is available on GitHub at https://github.com/pdey/SpeedPerception.

Open Source

Just like we did with the results of phase 1 of SpeedPerception, once the crowd-sourcing component generates a sufficient amount of user data, we will open source the dataset, making it available to the web performance community, along with the analysis of what we discover.

Please help us by taking the SpeedPerception Challenge now. Thanks.

Results from Phase 1

In phase 1, we discovered a combination of three values: an abbreviated SpeedIndex up to time to click (TTC) and an abbreviated Perceptual Speed Index up to TTC, in conjunction with startRender (or Render), can achieve upwards of 85%+ accuracy in explaining majority human A/B choices. Does the power of this new combination “model” hold true for all sites, or just our original data set? This is what we’re working on finding out.

If you’re interested in phase 1, here’s some more light reading:

 

Web Performance Stats are Overrated

Note: A more polite version of this post originally appeared on Instart Logic’s blog

Who cares about performance?  Your customers, that’s who!  Speeding up your site by several seconds, or even by hundreds of milliseconds, will make your customers happy. Well, at least they’ll be less irritated.  That’s pure logic. I am not citing statistics. I am using common sense.

For example, a 50% decrease in file size — cutting bloat in half — has a greater impact on someone 2G mobile than on a T1 line. Common sense dictates the impact is greater on a 5Mb site than a 5Kb one, no matter the network speed as performance gains achieved by halving a 5Kb download may not even have a noticeable effect. Similarly, improving a 12s page load to 11s doesn’t have the same impact as improving a site from 3s down to 2s. The impact of a one second improvement depends on the original experience. This is obvious. It’s common sense.

Conclusions based on common sense are often accurate, but aren’t considered factual like conclusions based on statistics.  Statistics are often used to demonstrate something as fact, but a statistic is really only a fact about a piece of data. Mark Twain is credited with saying “Facts are stubborn things, but statistics are pliable.”

We discuss web performance statistics as if the web were a monolith, as if all web applications were uniform, like performance improvement effects are linear. They may not be represented by a simple exponential equation, but performance isn’t a simple science, and metric interpretation shouldn’t be assumed to be linear, or even accurate.

The statistic many web speeders quote “a one second delay in web page responsiveness leads to a 7% decrease in conversions, an 11% drop in pageviews, and a 16% decrease in customer satisfaction” is kind of bullshit today. This quote comes from a 2008 study: a study that came out a few months after the iPhone SDK was released and Android Market came into being, before either proliferated. When that study was conducted, the iPad and Android tablet were a few years off. As web speeders, we can’t quote a performance study that basically predates mobile. Yet, that’s what we’re doing.

While the customer satisfaction statistics might be BS, the conclusions are not. The longer the site takes to load, the less happy your customers will be, the more users will abandon your site, the lower your conversion rates, and the less money you’ll make. It’s not even that your customers will be unhappy — it’s that they may not be your customers anymore. That’s common sense, no matter what the actual percentages are.

Definitely give yourself a performance budget and test your site. Test your application throughout the development process. Optimize and right size your images, setting dimensions. Reduce DNS lookups. Reduce bloat, minimizing request size, GZipping all requests. Make fewer HTTP requests or serve content over HTTP/2, caching what you can. Basically, follow as many performance recommendations as you can.

Improve the performance of all of your content, not just your home page. Focus your energies on the critical path of your site. It’s obviously less important to optimize some pages, say the “libraries we use” page linked to from an “about us” page linked to from a political candidate’s website. But do realize users often enter your site thru a side door: for example, a product page, with the shopping cart experience being a necessity for all sales for an online store. In the case of eCommerce, optimizing your homepage might create the most “savings”, but will do little to improve user experience for your actual potential customers.

And never forget: improving the download speed is not enough: improving your site by one second delay won’t improve customer satisfaction if, once loaded, they’re met with an unresponsive UI.

Or maybe it will.

Let me go back to 2008 to test that out.