Dummy Origin: Testing CDN Performance

Before deciding to work with a particular CDN, I would like to make sure that she will do exactly what we expect from her. Of course, you read the documentation, and discussed everything in detail with the consultants, but this is not enough? And if you approach the business as eagerly as we do, this is not enough. You would like to test the CDN, get the objective characteristics of its work, compare them with your requirements, etc.

Dummy Origin is the tool that allows you to do all this in the best way, and most importantly - easily.

Dummy Origin is a specialized server written in Go. It was specifically designed to test pull mode CDNs. It is lightweight, fast, easy to install, and most importantly, it is open source, i.e. free!

Why use Dummy Origin and not test the CDN directly using your production server? Because it’s easier to check the various proxy modes.

• You do not need to change anything in the configuration, server and site settings.
• You can freely change the headers in the responses and analyze how the CDN responds to this.
• It is very easy to test various critical scenarios such as “my server gives a 502 error, does the CDN support the distribution of outdated copies of pages while offline on the main site?

Features of Dummy Origin

General: Dummy Origin injects arbitrary headers into a request. This is done using the URL parameter string, which is then inserted as the request header. For example, send to the URL ?ETag="41a0544696e19971383984fdaaeb5aed", Dummy Origin will display ETag:"41a0544696e19971383984fdaaeb5aed"in the request header.

This mechanism allows you to test the behavior of the CDN without changing the server configuration and rebooting it, which would be required to change the headers.

Other features of Dummy Origin:

Opportunity	Description	Benefits
Gzip	Compresses the response using standard gzip compression. Based on file extensions, not Content-Type. Currently supports .html, .css .js and .json extensions, but the rest can be easily added	Tests the work of CDN with compressed files
Conditional Queries	Usually the server gives a Last-Modifiedheader along with a response of 200 and gives 304 or 200 if it receives a If-Modified-Sincerequest	Allows you to check whether the CDN sends requests with a condition after the expiration date of the file in the cache
Range Requests	Sends the requested range of bytes in a 206 partial response and the corresponding header Content-Length	Important for transferring large amounts of data via CDN, such as video files
404 and 301	Gives a 404 error if the resource does not exist, makes a 301 redirect if /index.html is requested	Does the CDN 404 cache responses? Does CDN 301 forward or does it handle the redirect itself?
Special error generator	When the server receives the request as GET / err /, он возвращает http статус . Величина должна быть в диапазоне от 400 до 599	Allows you to test the behavior of the CDN in case of various errors by the server. In particular, you can check whether the CDN response caches 404.
X-Tb-Time	The server always displays the current time in the X-Tb-Time header.	Helps determine if a CDN service serves data from the cache or from the server, and for how long the content has been in the cache.
Loggly Support	Request and response details are processed through Loggly if the environment variable LOGGLY_TOKEN is defined.	Loggly - a cloud-based log processing service, very convenient for parsing logs of CDN requests to the server

At Dummy Origin, there is currently no:

• Https
• Brotli (compression algorithm used in particular in the Chrome browser)
• Cannot display different content / headers based on requests with User-Agent or Accept-Language headers.

Installation and configuration

Server installation is described in README.md on GitHub.

Place several files on the server. They should be similar to those files that will represent your real content. Several images of different sizes, text files - scripts or just text. Some kind of video.

Do not place index.html on the server to see which files are in it while viewing the source domain in the browser.

Configure CDN. Set on the CDN the option to distinguish addresses with different query strings as unique (that is, to distinguish between file.jpg?123and file.jpg?456, in Air.rf, this is the option “Cache with query string”). Usually this mode is set to CDN by default, but still check. Also ensure that the CDN forwards all the headers from the source server to the client, including the X-Tb-Time header.

Configure DNS. Also set a low TTL value on the server of about 300 seconds, this will come in handy later when testing the response of the CDN to the lack of response from the server.

CDN Behavior Testing: Examples

Here the address dummy.mydomain.com is used as the address of the CDN and dummy.origin.com is the address of the source server. Test using cURL. Run cURL with the following options: curl -vo / dev / null and send a GET request, and in response see only the headers, sending the main body of the page to / dev / null.

Do not use the –I flag to send a HEAD request. CDN can handle HEAD requests in a completely different way than GET. In addition, your users are unlikely to send to the HEAD server, their requests will be mainly GET.

Do not cache and receive all data from the source server

This is not the most typical example of using CDN, but if you need to leave some of the content uncached, you need to make sure that everything works as it should. The question here is which Cache-Control directive will use more correctly. Based on RFC 7234, it seems logical to use Cache-Control: no-store. RFC writes:

The no-store directive indicates that data should not be cached for both request and response. Affects both the public cache and the local cache.

Let's try:

curl -vo /dev/null 'http://dummy.mydomain.com/15kb.jpg?Cache-Control=no-store'

Below you see the response headers for our CDN Fastly tests. Surprisingly, Fastly does not respect the no-store and caches data after it does not detect it during the first request in its cache.

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Cache-Control: no-store
Content-Type: image/jpeg
Last-Modified: Tue, 21 Mar 2017 10:13:25 GMT
X-Tb-Time: 2017-03-30 14:39:11.358951299 +0000 UTC
Content-Length: 15887
Accept-Ranges: bytes
Date: Thu, 30 Mar 2017 14:39:23 GMT
Via: 1.1 varnish
Age: 12
X-Served-By: cache-ams4136-AMS
X-Cache: HIT
X-Cache-Hits: 1
X-Timer: S1490884763.605627,VS0,VE0
Vary: Accept-Encoding

Maybe no-cache or max-age = 0 will work? We need the CDN not to cache responses and not send conditional requests to the original server.

Testing showed that no-cache also does not work - CDN continues to cache responses, but max-age = 0 worked.

Cache for 300 seconds, then check with the original

Writing a CDN task to cache a file for 300 seconds is quite simple: Cache-Control = max-age = 300. The question is this: when the cache file expires, what type of request will the CDN server send - conditional or unconditional? Someone may say that a conditional query will be optimal here, because if the file is not out of date, the server will say 304 - “no, the file is not out of date, you can take it from the cache”, and the CDN will quickly give the client the copy it has. This is faster than if the server started sending the entire file itself.

RFC 7234 does not state that the CDN should send a conditional request to verify that the cache is up to date. Therefore, it will not be surprising if the CDN sends an unconditional request in this situation (which is not very good for performance).

curl -vo /dev/null 'http://dummy.mydomain.com/15kb.jpg?Cache-Control=max-age=300'

Fastly and CDNetworks send conditional requests. Air.rf and StackPath send unconditional requests to the server, updating the cache with a new version of the file. This can be seen from the logs of the source server and from the value of the X-Tb-Time field in the response header of the CDN to the client. Cache-Control = max-age = 300, must-revalidate was also tested, but to no avail.

HTTP/1.1 200 OK
Date: Fri, 31 Mar 2017 11:20:16 GMT
Content-Type: image/jpeg
Content-Length: 33956
Access-Control-Allow-Origin: *
Cache-Control: max-age=20,must-revalidate
Last-Modified: Fri, 31 Mar 2017 09:49:42 GMT
X-Tb-Time: 2017-03-31 11:20:16.091275242 +0000 UTC
Server: NetDNA-cache/2.2
Accept-Ranges: bytes
X-Cache: EXPIRED
Connection: keep-alive

Cache for 5 minutes, allow to issue an expired version for another 15 minutes

curl -vo /dev/null 'http://dummy.mydomain.com/15kb.jpg?Cache-Control=max-age=300&stale-while-revalidate=900&stale-if-error=900'

It is important to test three situations here:

1. Source server not found DNS (NXDOMAIN)

For a long time after the DNS TTL expired in the cache for our server, the CDN still returned the content stored in it. It's good.

More interestingly, the CDN continued to contact the original server for fresh versions of the content. It seems that, together with the content, it also stores the IP addresses from which it was received, and thus the DNS error was "covered", which made it impossible to calculate IP. Or maybe Fastly supports a long-lived connection to the server? Here you need to look deeper.

2. The source server issues a response of 400

Something went wrong on the server, and it started throwing a 400 Bad request error. It was expected that the CDN would handle stale-if-error in this case, but it just broadcasted the 400 response to the client. As it turned out, the expectations turned out to be incorrect, since RFC 5861 “HTTP Cache-Control extensions for outdated content” states:

An error is a situation in which a 500, 502, 503, or 504 HTTP response is returned.

A 400 is not an error, by default.

3. The server discards all incoming traffic

In the third scenario, the server’s DNS records were replaced by binding the domain to address 72.66.115.13. This address is associated with blackhole.webpagetest.org. This is a special address for testing situations when the server does not respond to any request.

Has the CDN issued content stored in it? No, she began to give a 503 response after about 1 second of waiting, and this 503 response contained the Retry-After: 0 header. Air.rf in the described situation responds from the Retry-After: 500 headers and has customizable connection and response times from the site hosting .

This is what the default settings for the proxy are. The documentation says: “If Varnish cannot contact the source server for any reason, a 503 response is generated anyway.” This property is inherited from the Varnish code, on the basis of which the majority of CDN proxies are created.

StackPath in this situation may produce archived content, but this situation must be configured by the user for errors 404, 502, 503, etc. However, StackPath does not handle the NXDOMAIN or blackhole situation.

So, use Dummy Origin to make sure your CDN provider is working properly. This is a free open source tool.