Bagofich .RU or how to get problems where they should not be for many years
UPD . Due to the fact that many commentators do not read in full, I will write here a brief summary of the problem: for .RU Glue Records are not cleared when changing domain delegation. At a minimum, for domains managed through Ru-Center and Reg.ru. The article itself is a “life story” about what problems such a “feature” of the .RU zone can cause, how they were diagnosed and resolved.
“One my friend” (c) told a story about his adventures with DNS.
Imagine that 2016 ends, on your DNS servers there are several hundred domains, a long time ago you started running 5 powerful DNS servers in different data centers, you want to finally get rid of old DNS servers that recently just take money to rent and take up space in the rack. However, immediately after trying to disconnect them, the support phones become red-hot, so the old servers have to be turned on quickly and thoughtfully understood.
Once again we check the DNS zones, make sure that there are no old addresses anywhere and for a very long time. We check from different regions with requests for each of our NS. Everywhere everything is given correctly. We take tcpdump and look at requests for port 53 on the old DNS. It turns out that there are surprisingly many requests for them. And this despite the fact that the IP addresses of these servers have not been featured for many months!
We look at tcpdump on the new DNS, and there are noticeably fewer requests than on the old ones. Just miracles!
After analyzing the queries to the old DNS, we take the top10 client addresses and find out that it is BIND9 (responds to dig + short version.bind txt chaos @ server) of large domestic providers (Rostelecom, TTK, etc.).
We tearfully ask several colleagues to resolve client domains from the home Internet via the provider's DNS server (
Suddenly, the idea comes up to request the root servers of the .RU zone (currently it is a.dns.ripn.net., B.dns.ripn.net., Etc., the list can be obtained with the command
We recall the theory of DNS, just in case we read the Wikipedia article on gluing zones (Glue Records) . Since Glue Records have not been used for a very long time (5-6 years), we come to the wrong conclusion: one of the clients managed to specify the old DNS IP addresses when delegating the domain. I had to write scripts that requested whois from the list of client domains for analysis.
Unfortunately, due to this incorrect assumption, we lost time, but found other errors. For example, clients love domain mail from Yandex and manage to delegate the domain to our NS and Yandex NS at the same time.
The next morning, after analyzing once again the collected whois data, we find nothing. It remains only to contact the RU-Center with diagnostics and our assumptions. Despite the long-term cooperation with this company, we had practically no calls for technical support, mainly questions arose on the accounting part. Therefore, we send an email to support @ and wait for an answer. After a couple of hours of waiting for an answer to Email, our employees can no longer endure and try to call. After listening to a sufficient amount of music, we get to a living person who finds our letter and reports the ticket number. Hurrah! However, we do not receive a response to the application. A few more calls with a frequency of 1-2 hours do not bring the result closer. Fortunately, at the end of the day we get something like this:
Very surprised. The last time glue records were many years ago. Is this really possible? We raise the mail archives, we find letters there for 2011, when the domain was delegated to other ns and are surprised again. It turns out that for so many years it worked incorrectly and no one noticed anything!
In the meantime, the last hours of renting old pieces of iron are going on, and we understand that with TTL of 4 days in provider caches, you will need to pay another month of rent, and this is not quite what was planned.
Please support Ru-Center to remove glue records. We don’t need them at all, why do we need to bind to new IP addresses ?! Moreover, in our DNS zone for each NS several IP addresses + IPv6 are indicated. And we get an answer that baffles us. Something like this:
Those. we conclude that the registrar does not have a ready-made mechanism for this operation (it seemed to us like a button in the support employee’s interface), which means that either no one faced a similar problem (which is unlikely), or they simply changed the IP addresses in glue records and lived further. But this is wrong, because it is logical that if the IP addresses for NS are deleted from the registrar, then glue records should also be deleted. We assume that these are old glitches, the beginning of this decade, have long been resolved, and probably we were just out of luck.
By noon the next day, we are carefully interested in email, is there an answer to the application? After we go through the quest several times listening to music on the phone, we call live people, but apart from “your application has been submitted to the system administration department” we can’t achieve anything. Almost a day has passed, we have no choice but to follow the advice indicating the new glue records, which we are doing. After a few hours, the changes take effect (as you remember, .RU is not updated too often) and we delete these glue records (we don’t need them and even interfere). We are waiting for the next .RU update, however, .RU root servers continue to respond with these entries, but with new IP addresses. Is there a glitch left? !!!
Since there is no answer on the application, the next day we again go through quests with telephone music. I will especially like the level when they asked me to connect with the support manager and the girl warned that “she would have to wait a while” and delivered us music, which stopped after 30 minutes and the call broke. Fortunately, towards the end of the working day, the answer finally came that the glue record was deleted. Hooray, comrades!
Remembering TTL at 4 days in provider caches, we look at tcpdump. Indeed, there are less and less requests for old DNS servers, i.e. everything goes as planned. It remains only to wait. Just in case, we take a couple of unnecessary domains and try to reproduce the situation. And she repeats!
For history, I’ll leave a recipe here for how to reproduce the situation:
Take the first ingavto.ru domain, create A records for ns10.ingavto.ru and ns11.ingavto.ru in it, pointing to the IP addresses of two DNS servers and delegate this domain to them with an indication IP addresses at the registrar (glue records). We are waiting for the .RU zone to be updated.
We take the second domain body-m-auto.ru, delegate it to ns10.ingavto.ru and ns11.ingavto.ru, we are waiting for the update.
The first domain is delegated to any other NS, in the zone we change the addresses for ns10.ingavto.ru and ns11.ingavto.ru in A records, we are also waiting for the update.
As a result, we have in the root zone .RU incorrect glue records and glitches, which are described above.
We will ask the IP addresses of NS servers from Google:
The answer matches what is written in the DNS zone. An example of a request to the root server, it can be seen that completely different IP addresses are given:
Unfortunately, we could not find out, this problem is only associated with Ru-Center or is relevant for other registrars . If you have a couple of domains for experiments purchased through another registrar, please test and write in the comments.
With a high probability, such a problem may exist not only for the .RU domain, but also for the domain.RF , but we also do not have such domains for the test now.
It also begs the obvious conclusion that at the moment it is not necessary to use glue record for the .RU domain and in some cases it makes sense to contact the registrars with an application for cleaning such records.
PS It would be nice if someone from the specialists working at Ru-Center or another registrar commented on the situation.
“One my friend” (c) told a story about his adventures with DNS.
Background
Imagine that 2016 ends, on your DNS servers there are several hundred domains, a long time ago you started running 5 powerful DNS servers in different data centers, you want to finally get rid of old DNS servers that recently just take money to rent and take up space in the rack. However, immediately after trying to disconnect them, the support phones become red-hot, so the old servers have to be turned on quickly and thoughtfully understood.
Chapter 1
Once again we check the DNS zones, make sure that there are no old addresses anywhere and for a very long time. We check from different regions with requests for each of our NS. Everywhere everything is given correctly. We take tcpdump and look at requests for port 53 on the old DNS. It turns out that there are surprisingly many requests for them. And this despite the fact that the IP addresses of these servers have not been featured for many months!
We look at tcpdump on the new DNS, and there are noticeably fewer requests than on the old ones. Just miracles!
After analyzing the queries to the old DNS, we take the top10 client addresses and find out that it is BIND9 (responds to dig + short version.bind txt chaos @ server) of large domestic providers (Rostelecom, TTK, etc.).
Chapter 2
We tearfully ask several colleagues to resolve client domains from the home Internet via the provider's DNS server (
dig any clientsite.ru. @ns1.provider.ru ) and get an answer with incorrect IP addresses and a huge TTL 345600 (4 days !!!) in the ADDITIONAL SECTION. Moreover, if a domain name is delegated, the provider’s DNS server starts responding with the correct IP and the correct TTL, but the “happiness” ends together with the expiration of this TTL. The situation is reproduced in 6 cases out of 10, where the provider uses BIND9 as a recursor. We are assembling a test site with the latest version of BIND9 on debian and ubuntu, the situation is reproducing, restarting, etc. don't help.Chapter 3
Suddenly, the idea comes up to request the root servers of the .RU zone (currently it is a.dns.ripn.net., B.dns.ripn.net., Etc., the list can be obtained with the command
dig ns ru. ). And we get the same answer with invalid IP addresses, TTL 345600 in the ADDITIONAL SECTION. We managed to localize the problem, now we need to understand how it arose.Chapter 4
We recall the theory of DNS, just in case we read the Wikipedia article on gluing zones (Glue Records) . Since Glue Records have not been used for a very long time (5-6 years), we come to the wrong conclusion: one of the clients managed to specify the old DNS IP addresses when delegating the domain. I had to write scripts that requested whois from the list of client domains for analysis.
Unfortunately, due to this incorrect assumption, we lost time, but found other errors. For example, clients love domain mail from Yandex and manage to delegate the domain to our NS and Yandex NS at the same time.
Chapter 5
The next morning, after analyzing once again the collected whois data, we find nothing. It remains only to contact the RU-Center with diagnostics and our assumptions. Despite the long-term cooperation with this company, we had practically no calls for technical support, mainly questions arose on the accounting part. Therefore, we send an email to support @ and wait for an answer. After a couple of hours of waiting for an answer to Email, our employees can no longer endure and try to call. After listening to a sufficient amount of music, we get to a living person who finds our letter and reports the ticket number. Hurrah! However, we do not receive a response to the application. A few more calls with a frequency of 1-2 hours do not bring the result closer. Fortunately, at the end of the day we get something like this:
The existence of glue records means that the child DNS servers with these IP addresses were previously specified for the domain. To change these records, you must delegate the domain to the same child DNS servers with the new IP addresses.
Very surprised. The last time glue records were many years ago. Is this really possible? We raise the mail archives, we find letters there for 2011, when the domain was delegated to other ns and are surprised again. It turns out that for so many years it worked incorrectly and no one noticed anything!
Chapter 6
In the meantime, the last hours of renting old pieces of iron are going on, and we understand that with TTL of 4 days in provider caches, you will need to pay another month of rent, and this is not quite what was planned.
Please support Ru-Center to remove glue records. We don’t need them at all, why do we need to bind to new IP addresses ?! Moreover, in our DNS zone for each NS several IP addresses + IPv6 are indicated. And we get an answer that baffles us. Something like this:
the application has been submitted for execution to the system administration department, we will inform you of the result
Those. we conclude that the registrar does not have a ready-made mechanism for this operation (it seemed to us like a button in the support employee’s interface), which means that either no one faced a similar problem (which is unlikely), or they simply changed the IP addresses in glue records and lived further. But this is wrong, because it is logical that if the IP addresses for NS are deleted from the registrar, then glue records should also be deleted. We assume that these are old glitches, the beginning of this decade, have long been resolved, and probably we were just out of luck.
Chapter 7
By noon the next day, we are carefully interested in email, is there an answer to the application? After we go through the quest several times listening to music on the phone, we call live people, but apart from “your application has been submitted to the system administration department” we can’t achieve anything. Almost a day has passed, we have no choice but to follow the advice indicating the new glue records, which we are doing. After a few hours, the changes take effect (as you remember, .RU is not updated too often) and we delete these glue records (we don’t need them and even interfere). We are waiting for the next .RU update, however, .RU root servers continue to respond with these entries, but with new IP addresses. Is there a glitch left? !!!
Chapter 8
Since there is no answer on the application, the next day we again go through quests with telephone music. I will especially like the level when they asked me to connect with the support manager and the girl warned that “she would have to wait a while” and delivered us music, which stopped after 30 minutes and the call broke. Fortunately, towards the end of the working day, the answer finally came that the glue record was deleted. Hooray, comrades!
Chapter 9
Remembering TTL at 4 days in provider caches, we look at tcpdump. Indeed, there are less and less requests for old DNS servers, i.e. everything goes as planned. It remains only to wait. Just in case, we take a couple of unnecessary domains and try to reproduce the situation. And she repeats!
For history, I’ll leave a recipe here for how to reproduce the situation:
Take the first ingavto.ru domain, create A records for ns10.ingavto.ru and ns11.ingavto.ru in it, pointing to the IP addresses of two DNS servers and delegate this domain to them with an indication IP addresses at the registrar (glue records). We are waiting for the .RU zone to be updated.
We take the second domain body-m-auto.ru, delegate it to ns10.ingavto.ru and ns11.ingavto.ru, we are waiting for the update.
The first domain is delegated to any other NS, in the zone we change the addresses for ns10.ingavto.ru and ns11.ingavto.ru in A records, we are also waiting for the update.
As a result, we have in the root zone .RU incorrect glue records and glitches, which are described above.
We will ask the IP addresses of NS servers from Google:
$ dig +short A ns10.ingavto.ru. @8.8.8.8
136.243.55.209
$ dig +short A ns11.ingavto.ru. @8.8.8.8
136.243.55.194The answer matches what is written in the DNS zone. An example of a request to the root server, it can be seen that completely different IP addresses are given:
$ dig any body-m-auto.ru @a.dns.ripn.net.
; <<>> DiG 9.10.3-P4-Ubuntu <<>> any body-m-auto.ru @a.dns.ripn.net.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 269
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 3
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;body-m-auto.ru. IN ANY
;; AUTHORITY SECTION:
BODY-M-AUTO.RU. 345600 IN NS ns11.ingavto.ru.
BODY-M-AUTO.RU. 345600 IN NS ns10.ingavto.ru.
;; ADDITIONAL SECTION:
ns10.INGAVTO.RU. 345600 IN A 89.249.20.188
ns11.INGAVTO.RU. 345600 IN A 89.249.24.177
;; Query time: 42 msec
;; SERVER: 2001:678:17:0:193:232:128:6#53(2001:678:17:0:193:232:128:6)
;; WHEN: Sat Dec 10 00:00:30 MSK 2016
;; MSG SIZE rcvd: 153Epilogue
Unfortunately, we could not find out, this problem is only associated with Ru-Center or is relevant for other registrars . If you have a couple of domains for experiments purchased through another registrar, please test and write in the comments.
With a high probability, such a problem may exist not only for the .RU domain, but also for the domain.RF , but we also do not have such domains for the test now.
It also begs the obvious conclusion that at the moment it is not necessary to use glue record for the .RU domain and in some cases it makes sense to contact the registrars with an application for cleaning such records.
PS It would be nice if someone from the specialists working at Ru-Center or another registrar commented on the situation.
