Why (today) return 444 is not always helpful

    The Nginx web server has a wonderful response code 444, which “closes” the connection without sending data. This functionality is very useful when filtering parasitic traffic - if we are sure that the client is not valid by some criteria, then there is no need to notify it, for example, with the 403rd response. It is more efficient to simply stop data transmission, which, often, can significantly reduce the load on the server.

    Recommendations to use such answers can be found everywhere in the instructions for blocking referrals from popular sites and referral spam, protection against DDoS, etc.

    And, in general, for many years, these tips could be used almost without looking, but ... modern browsers do not stand still and periodically give us new surprises.

    Mandatory program
    Server:

    $ uname -orm
FreeBSD 11.1-STABLE amd64


    $ nginx -v
nginx version: nginx/1.15.0

    Customer:

    >ver
Microsoft Windows [Version10.0.15063]

    Google Chrome 67.0.3396.99 (Official build), (64 bit)
    Firefox Quantum 61.0 (64-bit)

    I took for experience only browsers Chrome and Firefox. Not because the rest are sinless, just in these two, the behavior described below was fairly stable.

    So, we have a website and we want to restrict access to a location without any conditions and without transmitting any data:

    server {
...
  location = /code/444 {
    return 444;
  }
...
}

    Expectation:

    • the browser will send one request;
    • the server will receive one request, close the connection and write one line with the code 444 to the log ;
    • without receiving data, the browser will display an error and stop communicating with the server.

    For greater clarity, let's display our expectation with the Nginx log line:

    18.12.12.29 - - [28/Jun/2018:11:50:10 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"

    Reality Chrome:

    18.12.12.29 - - [28/Jun/2018:11:50:10 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"18.12.12.29 - - [28/Jun/2018:11:50:10 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"18.12.12.29 - - [28/Jun/2018:11:50:15 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"18.12.12.29 - - [28/Jun/2018:11:52:04 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"

    Chrome sent a request, did not receive a response, immediately sent another request, and then began to periodically check the site for accessibility.

    The reality of Firefox:

    18.12.12.29 - - [28/Jun/2018:12:34:38 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:34:38 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:34:38 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:34:38 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:34:38 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:34:39 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:34:39 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:34:39 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:34:39 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:34:39 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:34:39 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:34:40 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:34:40 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:34:40 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:34:40 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:34:40 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:34:41 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:34:41 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:34:41 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:34:41 +0000] "GET /code/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"

    Firefox decided not to waste time on trifles and immediately sent 20 requests (sometimes making 10). But then no longer makes "checks."

    At the same time in the debugger, it reports as for one request:

    Spoiler header
    image

    Thus, expecting to receive one request from the client, in fact, we receive 3-20 requests. In addition to requests that are completely unnecessary to us, there is a risk to feed such a log, for example, to an anti-DDoS script and block this ip. And it would seem half the trouble, because we already “gave” to him 444, that is, we don’t feel sorry for him, but vice versa can happen - we will show the client something we didn’t even plan.

    Modify config:

    map $http_referer     $code_if {
    "~*https://habr.com/post/415565/" 1;
    "http://tison.ru/ref" 1;
    default 0;
}
...
server {
...
  location = /code/444 {
    return 444;
  }
...
  location = /codeif/444 {
    if ( $code_if = 1 ) { return 444; }
    add_header "Content-Type" "text/html; charset=UTF-8" always;
    return 200 "Expected code 444";
}
...
}

    Here, we have banned the return of content to all clients have a referrer "
    https://habr.com/post/415565/" when moving to the page tison.ru/codeif/444

    Expectation:

    Knowing is already above features, we assume that the browser will make more than one request, but the 200th for a response from the server, they will not receive .

    In the form of a log, again, we expect one or more lines of the form:

    18.12.12.29 - - [28/Jun/2018:12:52:02 +0000] "GET /codeif/444 HTTP/1.1"4440"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"

    When debugging in the console, everything works as expected. The server closed the connection and we did not receive any content:

    $ curl --referer "https://habr.com/post/415565/" tison.ru/codeif/444
curl: (52) Empty reply fromserver

    Well, the browser reality.

    Chrome:

    18.12.12.29 - - [28/Jun/2018:12:58:12 +0000] "GET /codeif/444 HTTP/1.1"4440"http://tison.ru/ref""Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"18.12.12.29 - - [28/Jun/2018:12:58:12 +0000] "GET /codeif/444 HTTP/1.1"4440"http://tison.ru/ref""Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"18.12.12.29 - - [28/Jun/2018:12:58:13 +0000] "GET /codeif/444 HTTP/1.1"20017"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"

    Firefox:

    18.12.12.29 - - [28/Jun/2018:12:56:29 +0000] "GET /codeif/444 HTTP/1.1"4440"http://tison.ru/ref""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:56:29 +0000] "GET /codeif/444 HTTP/1.1"4440"http://tison.ru/ref""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:56:29 +0000] "GET /codeif/444 HTTP/1.1"4440"http://tison.ru/ref""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:56:29 +0000] "GET /codeif/444 HTTP/1.1"4440"http://tison.ru/ref""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:56:30 +0000] "GET /codeif/444 HTTP/1.1"4440"http://tison.ru/ref""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:56:30 +0000] "GET /codeif/444 HTTP/1.1"4440"http://tison.ru/ref""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:56:30 +0000] "GET /codeif/444 HTTP/1.1"4440"http://tison.ru/ref""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:56:30 +0000] "GET /codeif/444 HTTP/1.1"4440"http://tison.ru/ref""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:56:30 +0000] "GET /codeif/444 HTTP/1.1"4440"http://tison.ru/ref""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:56:30 +0000] "GET /codeif/444 HTTP/1.1"4440"http://tison.ru/ref""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:56:31 +0000] "GET /codeif/444 HTTP/1.1"20017"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"18.12.12.29 - - [28/Jun/2018:12:56:31 +0000] "GET /favicon.ico HTTP/1.1"2006782"-""Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"

    Having made several “honest” requests for requests, both browsers dropped the referrer and calmly displayed the blocked page. Firefox, however, does not always do this. On my tests ~ in 15% of requests, which even more "helps" in the analysis.

    This behavior has been observed for quite some time, but when exactly Chrome and Firefox switched to this mode of operation, I can’t say. As well, and how long this mode will act.

    Therefore, today the use of return 444 does not always lead to the expected results. Well, tomorrow, we are looking forward to see what other browser developers will appreciate.
    Tags:

    Also popular now: