April 11, 2018 at 12:05How fast is AMP really?
- Transfer
The Accelerated Mobile Pages (AMP) project from Google caused some commotion for ideological reasons, but the technology itself was not dismantled in detail. A few weeks ago, Ferdy Kristant wrote about the unfair advantage that AMP receives through preloading . It made me wonder: how well does the AMP really work ? I saw tests like Ferdy’s when one or two pages were compared, but I haven’t yet met comprehensive objective tests.
Measuring AMP performance is actually not as easy as it sounds. There are at least four contexts to consider:
As Ferdy noted, when you click on an AMP article from a Google search, it loads instantly. Here, as never before, the small AMP lightning icon is appropriate. But you don’t notice that Google provides such an instant transition due to the active preload of AMP documents in the background.
In the search carousel, this is literally a frame where the entire AMP document is uploaded. If you click on this page AMP, then it is already loaded in the background - and will immediately appear on the screen.
AMP works great in the context of a Google search. Like any page preloaded in the background. AMP's only advantage in this context is the initial launch provided by Google .
In other words, evaluating AMP performance based on how these pages load in search results says nothing about the effectiveness of AMP itself, but rather about the effectiveness of preloading content.
In his post, Ferdy analyzed a page from Scientas. He found that without preloading it does not open instantly. On a 3G emulation, a Scientas AMP article shows you a white screen for 3.3 seconds.
You might think that this happens with only one page. Like, there is a wide variation in speed and, perhaps, Scientas is just a single example. Fair remark, so let's figure it out.
The first thing I watched was the news. I do not recommend this to anyone, but I had to.
When an AMP article came across, I added its URL to the table. It doesn’t matter what topic or site: if AMP is specified, then the link was placed in the table. The only filtering was to not include more than two URLs from the same domain.
In the end, after this filtering, I got a list of 50 different AMP articles. I ran them through WebPageTest to emulate a 3G connection on Nexus 5. Each page was built through AMP and also loaded from the source server .
AMP consists of three main parts:
When we talk about the AMP library, we mean AMP JS and AMP HTML together. AMP HTML is both a subset of HTML (there are limitations that you can and cannot be used), and an addition to it (AMP HTML includes a number of custom AMP components and properties). AMP JS is a library that gives you these custom elements, and also makes various optimizations for documents based on AMP. Since it is based on HTML, CSS and JS, you can completely build a document using the AMP library without using the Google AMP cache.
The AMP library must provide a certain level of consistency in performance. For the most part, she copes.
Most test pages loaded within a reasonable range of one another. But at both ends of the spectrum, some deviations were observed: the minimum values turned out to be quite low, and the maximum values were frighteningly large.
In general, AMP performance is relatively predictable. However, the numbers suggest that AMP support does not give a 100% guarantee that the page will become quick or easy. As with any other technology, it is possible to create an AMP document that is slow and heavy.
Any claim that AMP provides a certain level of performance depends on your attitude to extremes and on what you consider “performance”. If you are trying to build an entire site based on AMP, then you should understand: although it is unlikely to turn out to be too bloated, it will not immediately load at lightning speed. This will require some work.
At least this applies to the library itself. Perhaps the AMP cache will show a significant increase in speed?
The AMP library itself helps, but not as much as you might think. Maybe the Google cache will add speed?
The Google AMP Cache is a CDN for delivering AMP documents. Like most CDNs, it caches documents and applies a number of optimizations to them. The cache also includes a verification system that the document is a valid AMP document. When you see the AMP icon, for example, in a Google search, the document is downloaded from Google AMP Cache.
I again drove the same 50 pages through WebPagetest. This time I loaded every page with a Google AMP CDN. Pete Minen did me a favor and shared a script for WebPagetest that preheats connections to the Google CDN, so that the result is more like what you see in reality.
When loading from the cache, AMP pages showed a noticeable performance improvement in all respects.
In general, the benefits of the cache are quite substantial. Although the maximum values are still depressing (a slightly higher maximum is explained mainly by the difference in banners in different tests). But the median average, which corresponds to most AMP documents, has become better in all respects.
The improvement is not surprising given the various performance optimizations that CDN automatically performs, including:
Once again, it is worth noting that none of these optimizations require the use of AMP. Most large CDN providers do the same. You yourself can automate all these optimizations during the build process.
I'm not saying that you need to drop the Google cache, just keep in mind that you can and should use these optimization methods regardless of whether you use AMP or not. There is nothing unique about AMP or even AMP cache.
We made sure that the AMP library itself provides only a small performance boost, but the cache with all optimizations takes download speed to a new level.
One of the arguments in favor of AMP is that it does not need to be an “expert” to make a productive site. Although I would argue that many of the sites we saw can be called "productive", but it makes sense to compare AMP documents with their canonical equivalents.
For the next round of testing, I took the canonical version of each page and checked it under the same conditions. It turned out that most often AMP documents outperform their equivalents without AMP, which indicates insufficient optimization of the latter.
Forget the Google cache for a second and compare the AMP library with canonical pages again.
The metrics “Start loading” and “Speed index” do not noticeably benefit the library. In fact, the start of rendering in AMP documents is even slower .
This is not too surprising. As mentioned above, AMP documents use the AMP JS library for many optimizations and resource loading. When you rely on this JavaScript to render the page, you’ll immediately get a hit on visualization metrics. This is observed until the AMP cache enters the game, which returns normal values for the start of rendering and the speed index.
However, in other terms, the AMP is clearly superior to the canonical version.
The verdict on AMP effectiveness is a bit mixed. On the one hand, ceteris paribus, the use of AMP does not necessarily result in faster page loading. There is no guarantee that AMP will not slow down or jam your data.
On the other hand, AMP documents are usually faster than the original versions. More efficient load balancing in AMP significantly reduces overhead. Suddenly, publishers who like to include third-party scripts on canonical pages drastically reduce (at least they are forced to do this) the number of third-party scripts for AMP pages.
The biggest advantage of AMP is not a library, because its functions can be implemented independently. This is not a cache - many of its optimizations are implemented through a good build script, and any worthy CDN provider has all of them. This does not mean that the AMP JS library or cache does not really have the right things - they are. They just do not give the biggest difference in terms of performance.
The main advantage of AMP is the limitation on how much content can fit into one page.
For example, here is a graph showing all the requests for the same page, written under the requirements of AMP (right), compared with the canonical version (left). I apologize for having to scroll.
Comparison of loading graphs for the canonical version of the article (left) and the AMP version (right). AMP restrictions minimize the number of requests.
The page size in the 90th percentile for the canonical version is 5,229 KB. The size in the 90th percentile for AMP documents from the same source is 1553 KB, the difference is about 70%. The number of requests in the 90th percentile for the canonical versions of pages is 647, for AMP documents - 151. A reduction of almost 77%.
AMP restrictions actually reduce page size and remove extraneous junk from them. Publishers are willing to make this concession for Google’s content distribution services, but are hesitant to do so for canonical versions of pages.
If we evaluate AMP in terms of Web acceleration, then the evidence is not particularly convincing. AMP works on all sitesin addition to the version without AMP.
Absolutely. Everyone has. Sites.
And usually these versions without AMP are heavy and slow. If you read the news on these sites and did not specifically click on the AMP button, then AMP will not improve page loading in any way. AMP does not solve the underlying problem; he just hides her a little.
Time will tell if anything will change. Perhaps, as at the beginning of the transition from mobile sites
Measuring AMP performance is actually not as easy as it sounds. There are at least four contexts to consider:
- How well does AMP work in the context of Google search?
- How well does the AMP library work as a standalone framework?
- How well does AMP work when booting from the cache?
- How good is the AMP compared to the canonical document?
How well does AMP work in the context of Google search?
As Ferdy noted, when you click on an AMP article from a Google search, it loads instantly. Here, as never before, the small AMP lightning icon is appropriate. But you don’t notice that Google provides such an instant transition due to the active preload of AMP documents in the background.
In the search carousel, this is literally a frame where the entire AMP document is uploaded. If you click on this page AMP, then it is already loaded in the background - and will immediately appear on the screen.
AMP works great in the context of a Google search. Like any page preloaded in the background. AMP's only advantage in this context is the initial launch provided by Google .
In other words, evaluating AMP performance based on how these pages load in search results says nothing about the effectiveness of AMP itself, but rather about the effectiveness of preloading content.
How well does the AMP library work as a standalone framework?
In his post, Ferdy analyzed a page from Scientas. He found that without preloading it does not open instantly. On a 3G emulation, a Scientas AMP article shows you a white screen for 3.3 seconds.
You might think that this happens with only one page. Like, there is a wide variation in speed and, perhaps, Scientas is just a single example. Fair remark, so let's figure it out.
The first thing I watched was the news. I do not recommend this to anyone, but I had to.
When an AMP article came across, I added its URL to the table. It doesn’t matter what topic or site: if AMP is specified, then the link was placed in the table. The only filtering was to not include more than two URLs from the same domain.
In the end, after this filtering, I got a list of 50 different AMP articles. I ran them through WebPageTest to emulate a 3G connection on Nexus 5. Each page was built through AMP and also loaded from the source server .
AMP consists of three main parts:
- AMP HTML
- AMP JS
- AMP cache
When we talk about the AMP library, we mean AMP JS and AMP HTML together. AMP HTML is both a subset of HTML (there are limitations that you can and cannot be used), and an addition to it (AMP HTML includes a number of custom AMP components and properties). AMP JS is a library that gives you these custom elements, and also makes various optimizations for documents based on AMP. Since it is based on HTML, CSS and JS, you can completely build a document using the AMP library without using the Google AMP cache.
The AMP library must provide a certain level of consistency in performance. For the most part, she copes.
Most test pages loaded within a reasonable range of one another. But at both ends of the spectrum, some deviations were observed: the minimum values turned out to be quite low, and the maximum values were frighteningly large.
| Metrics | Min | Max. | Median | 90th percentile |
|---|---|---|---|---|
| Start rendering | 1765 ms | 8130 ms | 4617 ms | 5788 ms |
| Visually complete view | 4604 ms | 35 096 ms | 7475 ms | 21 432 ms |
| Speed index | 3729 | 16,230 | 6171 | 10 144 |
| The size | 273 KB | 10 385 KB | 905 KB | 1,553 KB |
| Inquiries | 14 | 308 | 61 | 151 |
In general, AMP performance is relatively predictable. However, the numbers suggest that AMP support does not give a 100% guarantee that the page will become quick or easy. As with any other technology, it is possible to create an AMP document that is slow and heavy.
Any claim that AMP provides a certain level of performance depends on your attitude to extremes and on what you consider “performance”. If you are trying to build an entire site based on AMP, then you should understand: although it is unlikely to turn out to be too bloated, it will not immediately load at lightning speed. This will require some work.
At least this applies to the library itself. Perhaps the AMP cache will show a significant increase in speed?
How well does AMP work when booting from the cache?
The AMP library itself helps, but not as much as you might think. Maybe the Google cache will add speed?
The Google AMP Cache is a CDN for delivering AMP documents. Like most CDNs, it caches documents and applies a number of optimizations to them. The cache also includes a verification system that the document is a valid AMP document. When you see the AMP icon, for example, in a Google search, the document is downloaded from Google AMP Cache.
I again drove the same 50 pages through WebPagetest. This time I loaded every page with a Google AMP CDN. Pete Minen did me a favor and shared a script for WebPagetest that preheats connections to the Google CDN, so that the result is more like what you see in reality.
logdata 0
navigate https://cdn.ampproject.org/c/www.webpagetest.org/amp.html
logdata 1
navigate %URL%When loading from the cache, AMP pages showed a noticeable performance improvement in all respects.
| Metrics | Min | Max. | Median | 90th percentile |
|---|---|---|---|---|
| Start rendering | 1427 ms | 4828 ms | 1933 ms | 2291 ms |
| Visually complete view | 2036 ms | 36,001 ms | 4924 ms | 19 626 ms |
| Speed index | 1966 | 18 677 | 3277 | 9004 |
| The size | 177 KB | 10.749 KB | 775 KB | 2.079 KB |
| Inquiries | thirteen | 305 | 53 | 218 |
In general, the benefits of the cache are quite substantial. Although the maximum values are still depressing (a slightly higher maximum is explained mainly by the difference in banners in different tests). But the median average, which corresponds to most AMP documents, has become better in all respects.
The improvement is not surprising given the various performance optimizations that CDN automatically performs, including:
- Caching Images and Fonts
- Maximum Image Size Limit
- Compress images on the fly, as well as generate variants of different sizes with the addition of srcset for them
- Using HTTP / 2 and HTTPS
- Clear HTML comments
- Automatically enable tooltips for resources such as
dns-prefetchandpreconnect
Once again, it is worth noting that none of these optimizations require the use of AMP. Most large CDN providers do the same. You yourself can automate all these optimizations during the build process.
I'm not saying that you need to drop the Google cache, just keep in mind that you can and should use these optimization methods regardless of whether you use AMP or not. There is nothing unique about AMP or even AMP cache.
How good is the AMP compared to the canonical document?
We made sure that the AMP library itself provides only a small performance boost, but the cache with all optimizations takes download speed to a new level.
One of the arguments in favor of AMP is that it does not need to be an “expert” to make a productive site. Although I would argue that many of the sites we saw can be called "productive", but it makes sense to compare AMP documents with their canonical equivalents.
For the next round of testing, I took the canonical version of each page and checked it under the same conditions. It turned out that most often AMP documents outperform their equivalents without AMP, which indicates insufficient optimization of the latter.
| Metrics | Min | Max. | Median | 90th percentile |
|---|---|---|---|---|
| Start rendering | 1763 ms | 7469 ms | 4227 ms | 6298 ms |
| Visually complete view | 4231 ms | 108 006 ms | 20 418 ms | 54 546 ms |
| Speed index | 3332 | 45 362 | 8152 | 21,495 |
| The size | 251 KB | 11 013 KB | 2762 KB | 5229 KB |
| Inquiries | 24 | 1743 | 318 | 647 |
Forget the Google cache for a second and compare the AMP library with canonical pages again.
The metrics “Start loading” and “Speed index” do not noticeably benefit the library. In fact, the start of rendering in AMP documents is even slower .
This is not too surprising. As mentioned above, AMP documents use the AMP JS library for many optimizations and resource loading. When you rely on this JavaScript to render the page, you’ll immediately get a hit on visualization metrics. This is observed until the AMP cache enters the game, which returns normal values for the start of rendering and the speed index.
However, in other terms, the AMP is clearly superior to the canonical version.
Increased productivity. ... but for whom?
The verdict on AMP effectiveness is a bit mixed. On the one hand, ceteris paribus, the use of AMP does not necessarily result in faster page loading. There is no guarantee that AMP will not slow down or jam your data.
On the other hand, AMP documents are usually faster than the original versions. More efficient load balancing in AMP significantly reduces overhead. Suddenly, publishers who like to include third-party scripts on canonical pages drastically reduce (at least they are forced to do this) the number of third-party scripts for AMP pages.
The biggest advantage of AMP is not a library, because its functions can be implemented independently. This is not a cache - many of its optimizations are implemented through a good build script, and any worthy CDN provider has all of them. This does not mean that the AMP JS library or cache does not really have the right things - they are. They just do not give the biggest difference in terms of performance.
The main advantage of AMP is the limitation on how much content can fit into one page.
For example, here is a graph showing all the requests for the same page, written under the requirements of AMP (right), compared with the canonical version (left). I apologize for having to scroll.
Comparison of loading graphs for the canonical version of the article (left) and the AMP version (right). AMP restrictions minimize the number of requests.
The page size in the 90th percentile for the canonical version is 5,229 KB. The size in the 90th percentile for AMP documents from the same source is 1553 KB, the difference is about 70%. The number of requests in the 90th percentile for the canonical versions of pages is 647, for AMP documents - 151. A reduction of almost 77%.
AMP restrictions actually reduce page size and remove extraneous junk from them. Publishers are willing to make this concession for Google’s content distribution services, but are hesitant to do so for canonical versions of pages.
If we evaluate AMP in terms of Web acceleration, then the evidence is not particularly convincing. AMP works on all sitesin addition to the version without AMP.
Absolutely. Everyone has. Sites.
And usually these versions without AMP are heavy and slow. If you read the news on these sites and did not specifically click on the AMP button, then AMP will not improve page loading in any way. AMP does not solve the underlying problem; he just hides her a little.
Time will tell if anything will change. Perhaps, as at the beginning of the transition from mobile sites
http://m.to responsive sites, publishers are slowly harnessing. But right now, it seems that calls for switching to AMP are doing exactly what you think: they are promoting AMP, not performance.