Beginner's Guide to HTTP Caching Headers

Original author: Kyle Young
  • Transfer
This article provides information on caching headers (ZK) for HTTP and the corresponding behavior of content delivery networks (CDNs). If you want to figure out how caching headers fit into the modern web, or you just wonder what your colleagues are talking about, this article is for you.

If you already understand the benefits of ZK and want to expand your knowledge, I recommend that you refer to the documentation from W3.

What can ZK do for you?


Simply put, caching allows you to store web resources at remote points along the way from your server to a user browser. The browser also stores a cache so that clients do not constantly ask you for the same resources.

Web traffic caching settings are crucial for the sites you visit. If you pay for traffic, earn revenue from e-commerce, or just want to maintain your reputation as a good web developer, you need to understand how caching works.

In the case of resources like your company logo, favicon site, or basic CSS files that do not change from request to request, you can allow the requestor to keep copies of these files for a while. If your visitors were children in the back seat of the car, who would always ask you, “Have we arrived yet?”, This permission would be akin to the answer “No, and we need another 20 minutes, so remember my answer.”

By reducing the number of requests to the server, you increase the number of requests that it can handle. Pictures, scripts and style sheets can usually be cached in tail and mane, and dynamically created pages (forums, web applications) are usually not worth it. If you are primarily concerned about performance, all your dynamic content should be minimized with AJAX resources, and the rest should be cached to the maximum.

For customers and CDN


Historically, the cache settings were only related to the client’s browser, so do not forget about them. But today, with the proliferation of content delivery networks, CDNs, it’s more important to understand how caching works at intermediate points on the web.

What is a CDN?

In order not to read the entire Wikipedia article : these are the servers (in the plural) sitting between your server and your client. Each of them caches your content in accordance with the rules that you specify in various HTTP headers.


When properly configured, CDN transfers your content to clients through the fastest and closest server. In addition, CDN works as a buffer between you and users. We are interested in the percentage of cached requests — those requests that the CDN processed without pulling our server. Depending on the traffic and architecture, this number can reach 90%, although you will notice the effect with smaller numbers. It should be noted that with a small number of requests, most of them will be sent to your server - therefore this percentage makes sense only together with the caching time and the overall load of the site. But if you configure only one cache, and the caching headers will not work correctly, the totals may even become worse than they were.

image
Your servers deliver content to intermediate servers that are present in different regions.

In addition to caching, CDN has a nice side effect: if your servers suddenly crash, CDN in some cases buffer requests so that users may not notice it.

Main headlines


1. cache-control

The most important of all. Usually you specify its parameters in a line, something like:

cache-control: private, max-age = 0, no-cache

These settings are called cache response directives, and they are as follows:

private | public

Indicates whether the content is intended for a specific user. If so, you do not need to cache it.

no-cache

The directive itself says that this request must be done anew each time. The etag header, which is described below, is commonly used. The fun begins when you give the field a name after this directive. Then the caching servers understand that the answer can be cached, but you must delete the specified fields. This, for example, is useful for the proper operation of cookies. However, some old programs do not know how to work with this trick.

no-store

Reports that this response does not need to be stored. Surprisingly, a fact. If the cache works by the rules, it will make sure that none of the request parts will be stored. This is necessary in order to secure any sensitive information.

max-age

Usually, the lifetime of a resource is set via expires, but if you need to be more specific, you can specify max-age in seconds. And this directive has an advantage over expires.

s-maxage

A bit like the previous one, but s here means shared cache, and is needed for CDN. This directive takes precedence over max-age and expires when it comes to CDN servers.

must-revalidate

He says that each request needs to be done anew, and under no circumstances should the user provide cached content. It has an advantage over all other directives that enable caching. It is mainly used in some special protocols (for example, money transfers).

no-transform

Some proxies can compress and convert content to speed up work. This directive prohibits this behavior.

proxy-revalidate

Approximately the same as must-revalidate, but for intermediate CDN servers. Why wasn't she called s-mustrevalidate? Who knows. The point is that to check if the content has not been updated, it is necessary for each new user only once.

2. expires

Initially, this was the standard method for determining when a resource expires. Today, max-age and s-maxage have an advantage over it, but it is always useful to set this header for backward compatibility.

By setting a date more than a year away, you will violate the header specification.

3. etag

Acronym for entity-tag. This is the unique identifier of the requested resource — usually a hash of its contents, or a hash of the time it was updated. In general, the client’s method of requesting from the CDN “give me resource X if it has etag different from mine”.

4. vary

Very powerful thing. IE in the past handled it incorrectly, and even now it’s not doing it right. At some point, even Chrome with him was buggy. Essentially, the header tells the caching system which of the headers can be used to determine if the content that they have in the cache is valid. If we consider the cache as a data storage of the key-value type, then using vary adds these values ​​to the keys.

Often you can see a header like Accept-Encoding, which makes sure that your gzip-compressed resources will be accepted by the client. It saves a lot of traffic. In addition, setting

vary: User-Agent

will make your site more SEO friendly if you render different HTML / CSS depending on the User-Agent. Google will notice this little thing and Googlebot will process your mobile content too.

5. pragma

A rather old directive that can do a lot of things, which, however, is already being processed using more modern ones. We are most interested in the

pragma: no-cache form,

which in modern clients turns into

cache-control: no-cache

Warnings


Not all CDNs or client programs work according to specifications. If you are a web developer, you are familiar with this problem. Therefore, before starting the service, you should always test it to make sure that everything works as it should.

In addition, you may notice that some headings overlap or overlap. This is due to small differences between different methods, or due to the transition of the web from HTTP / 1.0 to HTTP / 1.1, in which caching is described in more detail.

1. Compression

CDN providers receiving a request to gzip as acceptable compression should also request compressed content from the server, or provide a compressed version of the resource to the client. Modern CDNs can perform resource compression independently.

When testing the performance of CDN servers, you need to consider that some are configured to check for both compressed and uncompressed versions of resources. This check slightly increases the response time.

2. SSL

CDN is a man-in-the-middle. You need to think about organizing HTTPS traffic and how it goes to your server. Many CDNs will redirect a request to somesite.com/asset to somesite.com/asset , so if your server’s logic depends on the protocol, either change the logic or ask the CDN to redirect to HTTPS.

What about dynamic content?


Usually in this case you need to set cache-control: no-cache so that the CDNs do not cache it. If you are interested in how to speed up his work a little, continue reading.

Typical Dynamic Content

HTTP/1.1 200 OK
Server: Apache
X-Rack-Cache: miss
ETag: "e6811cdbcedf972c5e8105a89f637d39-gzip"
Status: 200
Content-Type: text/html; charset=utf-8
Expires: Mon, 29 Apr 2013 21:44:55 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Mon, 29 Apr 2013 21:44:55 GMT


Most dynamic content is not so inconsistent. The list of active users, for example, has a lifetime of 10-20 seconds. Chart pages can probably live for a few minutes. A feed can be cached for a while, especially with etag. If your site’s load is high, it’s worth trying to cache some of its resources for a short while.

Cache Time Analysis


So what is the best time to set resources? It depends on the amount of traffic, the size of resources, the size of the cache.

In addition, you need to think about the main drawback of caching - reducing control over resources. If you need to update the resource instantly, you will have problems if you asked him some time ago the life time per year. Especially if you set this time not only for CDN (s-maxage), but also for users (max-age).

The longest period for the cache is a year, or 31536000. But this is a bad idea. It's like getting a tattoo on your face. If your servers do not withstand at least daily requests from the CDN about whether the resource has changed, then it's time to change the server.

Static Content Headers

HTTP/1.1 200 OK
Cache-Control: no-transform,public,max-age=300,s-maxage=900
Content-Type: text/html; charset=UTF-8
Date: Mon, 29 Apr 2013 16:38:15 GMT
ETag: "bbea5db7e1785119a7f94fdd504c546e"
Last-Modified: Sat, 27 Apr 2013 00:44:54 GMT
Server: AmazonS3
Vary: Accept-Encoding
X-Cache: HIT


Example cache settings for static content with S3. It is recommended that the cache store the resource for 900 seconds, for user programs - 300 seconds. The final heading indicates which of the CDNs processed the request.

One exception to the commandment is "Do not cache resources for a year", or rather, one hack that allows you to bypass the problems of a long cache. You can rename a resource every time you release a new version. An increasing version number, a temporary label, a hash can be added to his name.

The following table shows cache loss. Assuming that the resource receives 500 requests per minute, the following percentage of cache requests are obtained for different resource storage time options:

Cache Time (min)Percentage of cache requestsOriginal requests per hour
199.8%60
599.96%12
2099.99%3
6099.997%1
8640099.9998%<1


What percentage of cache requests will suit you? Usually from 60 seconds to 60 minutes is the normal lifetime for a resource. For pseudo-dynamic content, you can set the caching time in the range up to 60 seconds.

CDN check


Verify that the headers pass through the CDN as you expect. Typically, the CDN inserts some kind of X-header, which indicates the details of this resource. Here are some tools that seemed pretty useful to me.

1. Web Inspector

The easiest method is to right-click on a page in Chrome, select Inspect Element, go to the Network tab and click on the HTML resource. If this is not selected by default, select the Headers tab to see all the headers. Chrome also has the ability to set the user agent and not use the local cache.

2. Charles Proxy

Through such a proxy, you can redirect traffic and manipulate DNS queries, record headers, watch performance statistics - all this in an understandable and simple GUI.

3. the cURL

Make requests from the command line and view response headers. Useful keys: -A view user anget, -b - cookies, -F - form data, -H - assign headers, -I - request only headers.

4. hurl.it

Roughly speaking, this is a cURL with a clear interface. You can set headers and view response headers and the response body.

5. Python and Requests

Requests is a Python library for web requests. With its help, you can write automated tests for your site.

Marginal notes


Now that you have read the entire guide, a few notes.

Most web servers, such as Apache and Nginx, do most of the work for you. You only have to work with the cache-control header. Browsers are usually configured for strong caching of content, so you often have to deal with excessive caching than the other way around. Usually you specify a path like “/ static” that caches for some reasonable time, for example, 300 seconds. Then you need to make sure that the root path “/” is given with the heading “cache-control: no-cache”, but it is better to redirect the user for dynamic content directly to your servers, and leave for CDN only what lies in “/ static”.

A CDN usually has the ability to break caching rules and cache as much as you need, regardless of the headers. They also often deal with protocols quite freely, which is why it is so important to test what the CDN servers give and compare them with the headers of your servers.

For additional materials to speed up the site, we recommend that you refer to the documentation of Mobile Web Performance .

Also popular now: