Measuring Page Performance
Posted on: March 24, 2011
- In: Performance | web2.0
- 12 Comments
Web pages today are becoming increasingly complex and it is now well recognized that simply measuring page load time does not represent the response time of a page.
But what exactly do we mean by response time? Terms such as time to interactivity, perceived performance, above the fold performance etc. have come into vogue. Let’s examine these in turn.
Time To Interactivity
In last year’s Velocity Conference, Nicholas Zakas of Yahoo! gave an excellent presentation on how the Yahoo! Front Page responsiveness was improved in which he focused on time to interactivity. In other words, when can a user actually interact with a page (say click on a link) is more important than ensuring that the entire page gets rendered. With pages increasingly containing animated multi-media content which the user may not care about, this definition may make sense. However, imagine a page whose primary purpose is to serve up links to other pages (e.g. news) and loads lots of images after onload. All the links appear first which means the user can interact with the site, yet the page has lots of white space where the images go – can we truly just measure the time to interactivity and claim this to be the response time of a page?
Perceived Performance
Another popular term, perceived performance is defined loosely as the time that the user perceives it takes for the page. This means that all the major elements of the page that the user can easily see must be rendered. By definition, this measurement is highly subjective. Perceptions differ – for e.g. one user may not miss the Chat pane in gmail while for another this is a very important feature. Further, again by definition, this metric is application dependent. Nevertheless, for a particular web application, it is possible for developers and performance engineers to agree on a definition of what is perceived performance and work towards measuring/improving it.
Above The Fold Performance
In the Velocity Online conference last week, Jake Brutlag of google proposed “Above the Fold Time” (AFT) as a method of measuring a more meaningful response time. This was defined as the time taken to render all of the static elements in the visible portion of the browser window. Jake and others have put in some serious thought and defined the algorithm to distinguish between the static and dynamic (e.g. ads) content on a page.
Clearly, one part of the AFT proposal is valid – one doesn’t really care about the content on the page that is not visible initially and which the user can only get to by scrolling. But measuring everything above the fold has its issues as well. Take for instance the new Yahoo! Mail Beta. Y! Mail now has extensive social features to enable one to connect to Facebook, Twitter, Messenger and endless third-party applications. It is arguable whether the user will expect all of these third-party links to be on the page before he “perceives” that his request is complete. The page still looks finished without those links.
In my opinion, we need to distinguish between the essential parts of the page vs the optional ones (A caveat here – although no one would argue that an ad is essential, it is essential for the page to look complete). Looking at just static vs dynamic pixels misses this point. The difficulty of course is that there is no uniform way to define what is essential – it once again becomes application/page specific.
But for now, that’s the way I am going – defining “perceived performance” on a case by case basis.
12 Responses to "Measuring Page Performance"
March 24, 2011 at 12:34 pm
H Shanti:
(I work at Yottaa, a web performance company). Great post. How to quantify “user experience” is not easy. We have looked into the various options you mentioned in the post. In the end, we realize that one timing metric alone is not sufficient to judge webpage performance. In the end, we think one must take into consideration of not only the key moments of page loading, but also the “progression” of these moments, such as “Time to Title”, “Time to First Paint”, “Time to Display” and “Time to Interact” in order to quantify user experience. For example, a webpage with a longer “Time to Display” but quicker “Time to First Paint” can deliver a better user experience than a webpage with a quicker “Time to Display” but slower “Time to First Paint”. Another example, a site with a shorter ATF would arguably be better than a site with a longer ATF. However, if the former shows a white screen until ATF while the latter shows progressive rendering, the latter is definitely better. More details please see http://blog.yottaa.com/2011/03/monitor-website-speed-for-better-ux-and-seo-part-i/.
Let me know how you think and feel to give me a shout at my email address if that is more appropriate for more discussions. Thanks.
March 25, 2011 at 9:32 am
Coach – I read that Yottaa blog entry and I don’t agree with it completely. Time to interaction doesn’t necessarily come last. In a properly designed page, a user can and should be able to interact with the page before it finishes displaying. And like I said in this post, time to display ALL of the elements may be irrelevant to perceived performance.
March 25, 2011 at 5:34 am
Coach,
Great idea, but I’m afraid it fails when it comes to Mobile. Here’s an example – what if the page is “scaled” to fit inside the mobile screen? Then there is essentially no above the fold time – it’s just load time.
Secondly check out these results. http://www.5o9mm.com/har/viewer/v.pl?path=accounts/5o9/android-03-25-2011-13-27-04-GMT-www-yottaa-com.har
This is http://www.yottaa.com on an AT&T Samsung Galaxy in real time. You can see the carrier, device and geo-location where i ran the test. The page is 663.7k and takes 16.6 seconds to download.
If you click on the HAR tab at the top you’ll see the webkit events for how the browser processed the page.
Now my question for you is simple (well maybe not). You now have all the data about how the page loaded – where would you say the first byte showed up? Is it when the Title shows up at the 2 second mark or is it at the 41% done mark which is less than 2 seconds later, or is it when we see the first Yottaa header at the 5.3 second mark?
Personally I think the only way to really measure this is to put in the ability to support “custom timing events” so engineers can measure exactly when something loaded.
Finally everyone is focusing on AFT on the Desktop where it (IMO) has little relevance. The users don’t notice it anymore. Where the “puck is now heading” is the Mobile web where performance and optimization become much more critical when it comes to the user experience.
Cheers,
Peter
March 29, 2011 at 6:31 pm
I absolutely agree that website readiness is a complex topic and there is probably no way to automate the information gathering other then measuring real user experience through some complex and yet unknown method.
Some of those methods might include some tools like eye tracking, brain scanning and plain old usability sessions, it’s probably not possible to automate the process without skewing the results or making it just very expensive to do.
I recently wrote in Show Slow blog about one possible solution – using Human Computation to solve this issue: http://www.showslow.com/blog/using-human-computation-to-answer-web-performance-questions/
Since the problem is similar in complexity to concept recognition, it is possible that there is no other way to approach it then through humans.
Would love to hear your thoughts on this.
Sergey
March 31, 2011 at 4:37 pm
Sergey – you may be right. But for internal performance work we need some standard definition of “completion” to help track performance. We do this for now with the age old manual human method aided by videos and screen-shots.
March 30, 2011 at 10:49 pm
Hi Shanti, just discovered your blog through Twitter (we’re both listed on the list “webperformance”). Would measuring perceived performance be done using a visual tool like the “Film Strip” feature in WebPageTest.org?
March 31, 2011 at 4:31 pm
@Sergey,
I think you have to define what you want to see. Take this example – this is Yahoo’s Mobile home page: http://www.5o9mm.com/har/viewer/accounts/5o9/showpics.pl?filename=android-03-31-2011-09-59-15-GMT-www-yahoo-com.har
We took a snapshot in real time as it was loading in Android’s browser. You can see that the page is enormous – 320*2743 pixels – you can see that the base time for the entire page load was 1301563626751 ms but the screen shot actually took place at the 127903 ms time point.
Now where’s the above the fold point?
It looks like it was at the 320 pixel point – but Android never loaded the page until everything was in and assembled. So in effect you have no ability to determine anything.
Eye tracking fails here, so does everything else. No one would expect the mobile browser to wait until everything is in before it displays the page. That’s not the way it works on the desktop (welcome to Mobile)
That said we believe there is a way to solve the Above the Fold timing point, but that will have to wait for another post.
Peter
[…] Measuring Page Performance Yahoo! performance engineer Shanti Subramanyam’s thoughts on perceived and above the fold timing (AFT) for web page response time. […]
April 1, 2011 at 10:24 am
@Shanti,
We have an update for you. You can now see exactly where the fold is now on Mobile. We’ve added a grid/pixel ruler. Here’s Yahoo’s Mobile page http://www.5o9mm.com/har/viewer/accounts/5o9/showpics.pl?filename=android-03-31-2011-09-59-15-GMT-www-yahoo-com.har You can now clearly see the page dimensions alongside the browser dimensions.
It’s clear where the fold is – the issue that remains is getting a time on exactly when that occurred. (Note – Android’s browser waits until the page is fully assembled before loading it)
Performance Blog 
March 24, 2011 at 12:25 pm
Hello Shanti,
Great blog post. I thought you might be interested in the following. I’m one of the inventors of Mod_Gzip and with my partner (the other one) we’ve been working on Measuring Mobile performance. We’ve taken a different approach to the problem. We have a browser that allows you to measure “in real time” not only the device capabilities, but also the carrier network, the geo-location and how the browser (in this case Android’s) is processing the incoming page.
Let me share two links with you. Link 1 is this page http://www.5o9mm.com/har/viewer/v.pl?path=accounts/5o9/android-03-24-2011-20-09-36-GMT-perfwork-wordpress-com-2011-03-24-measuring-page-performance.har
You can see the performance waterfall (sorry about the pie charts, they’re on the next one) with all the associated timings. If you want to dig “even” deeper you can click on the HAR tab at the top of the page for some more Magic. Look for the _webkit_event messages. And you will see how the browser is processing the web page.
So knowing all of the above – what would you predict the “AFT” time really as? It’s harder than you think to come up with a really accurate time. You’ll need to dig into the webkit events to learn more.
Finally if this is not enough – we’ve come up with a way to “inject your own timing events”. These are custom timing events so that you can know in real time how the browser interacted with the HTML, CSS & JavaScript. To see how they work – click on this link and do a right click and view source in your browser. http://www.5o9mm.com/Cuzillion/index.htm
If you want to see what those measurements look like from inside the browser then click on this link http://www.5o9mm.com/har/viewer/v.pl?path=accounts/5o9/android-03-17-2011-19-25-28-GMT-www-5o9mm-com-Cuzillion-index-htm.har and click on the HAR tab at the top. It shows you exactly how the page loaded.
Hopefully with these kind of timing events we can get a handle on “perceived performance”.
Cheers,
Peter.
NOTE: Edited by Shanti to reduce text somewhat
March 25, 2011 at 9:49 am
Peter – interesting instrumentation. The trouble with this of course is that it is time consuming to add and I can’t see small-to-medium sites doing this. It is a lot of work to even come to an agreement on what elements need to be included in the definition of “perceived performance” (trust me – I deal with this daily) and although the likes of google, facebook and yahoo can do these things, I just don’t see this as an easy solution for the rest of the world.
March 25, 2011 at 10:17 am
There’s nothing to add “unless” you want to learn more. As for the definition – absolutely, there’s no consensus, but for those who want to experiment the tools are now available. However perhaps what is more interesting is the BI that comes from the tests. We’re already seeing big differences on Carrier networks and geo-location. For example did you you know that Bing loads faster in Israel than Google? The user experience is far more than “a custom timing event”. It’s the network, the device and real time geo-location.
Cheers,
Peter