Resolve IP Addresses on Linux: An Intuitive and Detailed Description
Setting up network interaction between services is not an easy task and is often done without a deep understanding of how to configure the system and what settings affect what. After migrating services in docker containers from centos 6 to centos 7, I encountered a weird web server behavior: he tried to join the service via IPv6, and the service only listened to the IPv4 address. The standard advice in this situation is to disable IPv6 support. But this will not help in some cases. Which ones? In this article, I set out to collect and explain in detail how applications resolve' addresses are.
The publication will be useful to novice administrators and developers.
After reading this article, you will learn:
- What is the Linux algorithm for resolving hostnames?
- how to redefine the logic of determining hostnames;
- what features and libraries does the OS use;
- what traps exist during configuration and how to prevent them;
The Linux operating system has several sources for determining the address by hostname. All the necessary functionality for the definition is in the GNU C Library (glibc) . glibc is essentially a framework and implements many useful functions for the developer, providing its own API to simplify development. Among other things, glibc implements POSIX . Functions such as open, read, write, malloc, printf, getaddrinfo, dlopen, pthread_create, crypt, login, exitfor Linux provides exactly glibc.
Utilities known to many host, digand nslookupuse glibc, but are delivered separately.
And I also conduct a telegram channel About IT without ties and a blog . On the channel I talk about management problems and how to solve them, I write about the principles of thinking in solving business problems, about how to become an effective and highly paid specialist.
Now that the developer has the opportunity to call the family function getaddrinfofrom glibc to determine the address, there is a need to configure the return values. For example, whether to use first /etc/hostsor a query to the DNS server. In glibc, this configuration is done using a circuit called Name Service Switch (NSS).
If to explain on the fingers, then NSS allows you to set the database and the search order in these databases to provide the service. In our case, the service is a search by hostname, and the database can be /etc/hostseither a DNS server. This is not the only service configured through NSS, mail alias services, a search service for users and groups are provided. See the list in the manual .
Thanks to NSS, you can configure the mentioned databases without rebuilding applications in runtime. The configuration in the file is performed /etc/nsswitch.conf. Below is an example of a config from the standard one /etc/nsswitch.confin Centos 7.
$ grep ^hosts /etc/nsswitch.conf
hosts: files dns myhostnamefiles , dns and myhostname are the database aliases for the search. files on most systems implies use /etc/hosts, the dns base is the DNS server to which the hostname search query will be performed, and myhostname is the most unusual base, the existence of which few people know and it is not part of the standard delivery in glibc. In some distributions, the mdns4_minimal database is also present . An analysis of these databases is provided below.
The databases are used in the order in which they are declared in /etc/nsswitch.confand if a record is found in the current database, then the chain is exited and the result is returned. If there is no result, it goes to the next base in the list. If no result is found in any database, then such an answer is given to the glibc function request getaddrinfo. The behavior of the transition to the next database and the conditions for such a transition can be additionally configured, for example, if the DNS is unavailable (not to be confused with the lack of a record), complete the chain. A clear and simple explanation of the principle of setting conditions for is /etc/nsswitch.confgiven in this article .
The files base , and in particular /etc/hostsout of the box in Centos 7, looks like this:
$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6It can be noted that there are two entries for localhost : IPv4 and IPv6 address. This can play a trick and at the end of the article I will tell you why.
The base dns when determining the address uses the name server specified in the config /etc/resolv.conf. Here is an example of mine /etc/resolv.confon a host system:
$ cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 127.0.0.1
nameserver 192.168.100.1Name server are also used in a chain and in the order of their declaration. In my case, the first is the local DNS server (I use dnsmasq ) to set the local .privzone addresses . If a match is found, then the address from the local network is returned. All other queries are sent to the main DNS server with the address 192.168.100.1.
The myhostname base is present in the Centos and Ubuntu distribution, but is not part glibc. Unaware of this fact, I spent a lot of time trying to find out why IPv6 addresses are returned to me to determine the host. It works as follows:
- When requesting a local hostname (what the command returns
hostname), the plugin returns all the IP addresses of public interfaces (i.e., all except loopback), in the absence of such interfaces, the IPv4 address127.0.0.2and IPv6 address are returned::1; - When a hostname is requested, localhost or localhost.localdomain returns an IPv4 address
127.0.0.1and an IPv6 address::1; - When a hostname request ends in .localhost or .localhost.localdomain, it returns an IPv4 address
127.0.0.1and an IPv6 address::1;
In the manual they still write about special logic with processing the _gateway hostname , but apparently this is some kind of patch, since I did not get it with Centos 7.
The mdns4_minimal base or mdns_minimal base is required for Avahi to work correctly. If necessary, you can refer to the Arch documentation for Avahi , where briefly and clearly given information
on use.
Now that the information on the bases and principles of their work is given, it is worth noting the differences in the definition of addresses in different tools, which leads to problems in runtime.
Typically, administrators verify the hostname using the host command . This is incorrect, since host, like dig, use only DNS resolution, but do not use NSS. Nginx, for example, uses the getaddrinfo function , and it uses NSS. This leads to the fact that driven into the /etc/hostshostname can work with nginx, but will not resolve in other ways. It is /etc/hostsmuch worse when an IPv6 address for a hostname is driven in, and only the IPv4 address is returned in the DNS settings. In this case, the administrator can verify that the host command returns only the IPv4 address and calm down, and then the application using getaddrinfoglibc will start and find the IPv4 and IPv6 address for the same hostname. The source of errors ...
To check the results returned by each of the databases, the documentation recommends using the getent utility .
Below are some examples of working with getentIPv6 enabled.
The /etc/nsswitch.conffollowing chain of bases is contained in:
hosts: files dns myhostnameThe /etc/hostsfollowing information is contained in
$ cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopbackThe command getent ahosts <hostname>will show a list of all addresses that could be found. With these settings, it will output the following:
$ getent ahosts localhost
::1 STREAM localhost
::1 DGRAM
::1 RAW
127.0.0.1 STREAM
127.0.0.1 DGRAM
127.0.0.1 RAWThe command allows you to pinpoint a specific base and find out what the base resolves. Consider the return values for each database:
$ getent -s hosts:files ahosts localhost
::1 STREAM localhost
::1 DGRAM
::1 RAW
127.0.0.1 STREAM
127.0.0.1 DGRAM
127.0.0.1 RAW
$ getent -s hosts:dns ahosts localhost
::1 STREAM localhost
::1 DGRAM
::1 RAW
127.0.0.1 STREAM
127.0.0.1 DGRAM
127.0.0.1 RAW
$ getent -s hosts:myhostname ahosts localhost
::1 STREAM localhost
::1 DGRAM
::1 RAW
127.0.0.1 STREAM
127.0.0.1 DGRAM
127.0.0.1 RAWIf you remove /etc/hostslocalhost from the line, the output will change:
$ getent -s hosts:files ahosts localhost
$ getent -s hosts:dns ahosts localhost
::1 STREAM localhost
::1 DGRAM
::1 RAW
127.0.0.1 STREAM
127.0.0.1 DGRAM
127.0.0.1 RAW
$ getent -s hosts:myhostname ahosts localhost
::1 STREAM localhost
::1 DGRAM
::1 RAW
127.0.0.1 STREAM
127.0.0.1 DGRAM
127.0.0.1 RAWNow the dns database and myhostname return answers, and the files database does not contain data. For DNS queries, the name server is configured /etc/resolv.confin my container, for example
$ cat /etc/resolv.conf
nameserver 127.0.0.11
options ndots:0Dnsmasq is installed on the host machine, which proxies and caches DNS server responses. The response from DNS will depend on the settings of the DNS server to which the request was received. RFC 1912 recommends that in clause 4.1, configure the DNS server so that localhost points to 127.0.0.1.
Certain zones should always be present in nameserver configurations:primary localhost localhost primary 0.0.127.in-addr.arpa 127.0 primary 255.in-addr.arpa 255 primary 0.in-addr.arpa 0
These are set up to either provide nameservice for "special"
addresses, or to help eliminate accidental queries for broadcast or
local address to be sent off to the root nameservers. All of these
files will contain NS and SOA records just like the other zone files
you maintain, the exception being that you can probably make the SOA
timers very long, since this data will never change.
The "localhost" address is a "special" address which always refers to
the local host. It should contain the following line:localhost. IN A 127.0.0.1
In my case, dnsmasq out of the box contains entries for localhost, as RFC recommends.
$ dig +noall +answer localhost ANY @127.0.0.1
localhost. 0 IN A 127.0.0.1
localhost. 0 IN AAAA ::1This is disabled either by deleting records from /etc/hoststhe DNS server itself, or by including the option no-hostsin /etc/dnsmasq.conf.
After enabling the getent option for the myhostname database, it will return a non-empty result, but as noted above, with myhostname enabled, IPv4 and IPv6 address will be returned. On systems with static IP addresses, you can safely turn off the myhostname plugin and configure local hosts using /etc/hosts. An alternative is to disable IPv6.
The IPv6 status on the server can be obtained from the kernel parameters. The value 0 is returned when IPv6 is on, and 1 when it is off.
$ sysctl net.ipv6.conf.all.disable_ipv6 net.ipv6.conf.default.disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.default.disable_ipv6 = 0In ifconfig output, IPv6 listening interfaces contain the string inet6 . Below is an example of output with IPv6 turned off and on, respectively:
# diabled
$ ifconfig -a
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.101.5 netmask 255.255.255.0 broadcast 0.0.0.0
ether 02:42:c0:a8:65:05 txqueuelen 0 (Ethernet)
RX packets 15789549 bytes 2553533549 (2.3 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 9783999 bytes 1318627420 (1.2 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1 (Local Loopback)
RX packets 606047 bytes 67810892 (64.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 606047 bytes 67810892 (64.6 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
# enabled
$ ifconfig -a
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.101.5 netmask 255.255.255.0 broadcast 0.0.0.0
inet6 fe80::42:c0ff:fea8:6505 prefixlen 64 scopeid 0x20<link>
ether 02:42:c0:a8:65:05 txqueuelen 0 (Ethernet)
RX packets 15787641 bytes 2553216408 (2.3 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 9782965 bytes 1318487919 (1.2 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1 (Local Loopback)
RX packets 605949 bytes 67799887 (64.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 605949 bytes 67799887 (64.6 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0You can turn off IPv6 by calling
$ sysctl -w net.ipv6.conf.all.disable_ipv6=1
$ sysctl -w net.ipv6.conf.default.disable_ipv6=1What will change after shutdown? I rolled back all the configs to the standard ones: /etc/hostslocalhost with IPv4 and IPv6 addresses is present, the option is disabled in dnsmasq no-hosts. Disable IPv6 with the commands above and getent output is as follows:
$ getent -s hosts:files ahosts localhost
127.0.0.1 STREAM localhost
127.0.0.1 DGRAM
127.0.0.1 RAW
127.0.0.1 STREAM
127.0.0.1 DGRAM
127.0.0.1 RAW
$ getent -s hosts:dns ahosts localhost
127.0.0.1 STREAM localhost
127.0.0.1 DGRAM
127.0.0.1 RAW
$ getent -s hosts:myhostname ahosts localhost
127.0.0.1 STREAM localhost
127.0.0.1 DGRAM
127.0.0.1 RAWWow, in the first output we have duplicated address 127.0.0.1. To understand why this happens, you should refer to the source code of glibc and the code of the getent utility. Below is a piece of getent utility code.
/* This is for hosts, but using getaddrinfo */
static int
ahosts_keys_int (int af, int xflags, int number, char *key[])
{
// ...
hint.ai_flags = (AI_V4MAPPED | AI_ADDRCONFIG | AI_CANONNAME
| idn_flags | xflags);
hint.ai_family = af;
for (i = 0; i < number; ++i)
{
struct addrinfo *res;
if (getaddrinfo (key[i], NULL, &hint, &res) != 0)
result = 2;
else
{
struct addrinfo *runp = res;
while (runp != NULL)
{
// printf goes here
}
freeaddrinfo (res);
}
}
return result;The AI_V4MAPPED flag of the getaddrinfo function maps IPv6 addresses to IPv4 if IPv6 addresses were not found as a result of polling the database. The AI_ADDRCONFIG flag will force getaddrinfo to check for the availability of IPv6 / IPv4 addresses configured in the system and if at least one IPv6 / IPv4 address is missing, IPv6 / IPv4 will not be returned regardless of what the specific database will respond.
Since getent has both flags enabled, and in the /etc/hostspresence of localhost addresses 127.0.0.1and ::1, getaddrinfo will get hosts from the NSS database (in the example above we discussed this database), addresses 127.0.0.1and ::1then not finding any IPv6 addresses in the system (disabled by kernel parameters) and will perform mapping ::1-> 127.0.0.1.
To better understand this concept, I will give examples with the output of getaddrinfo on the same system, with different settings ai_flags and ai_family . In /etc/hostsare included for localhost IPv4 and IPv6 addresses.
Source code can be found on my github .
# ./getaddrinfo localhost
IP addresses for localhost:
[AF_INET] AI_V4MAPPED
IPv4: 127.0.0.1
IPv4: 127.0.0.1
[AF_INET] AI_V4MAPPED AI_ALL
IPv4: 127.0.0.1
IPv4: 127.0.0.1
[AF_INET] AI_ADDRCONFIG
IPv4: 127.0.0.1
IPv4: 127.0.0.1
[AF_INET] AI_V4MAPPED AI_ADDRCONFIG
IPv4: 127.0.0.1
IPv4: 127.0.0.1
[AF_INET] AI_V4MAPPED AI_ALL AI_ADDRCONFIG
IPv4: 127.0.0.1
IPv4: 127.0.0.1
--------------
[AF_INET6] AI_V4MAPPED
IPv6: ::1
[AF_INET6] AI_V4MAPPED AI_ALL
IPv6: ::ffff:127.0.0.1
IPv6: ::ffff:127.0.0.1
IPv6: ::1
[AF_INET6] AI_ADDRCONFIG
getaddrinfo: Name or service not known
[AF_INET6] AI_V4MAPPED AI_ADDRCONFIG
getaddrinfo: Name or service not known
[AF_INET6] AI_V4MAPPED AI_ALL AI_ADDRCONFIG
getaddrinfo: Name or service not known
--------------
[AF_UNSPEC] AI_V4MAPPED
IPv4: 127.0.0.1
IPv6: ::1
[AF_UNSPEC] AI_V4MAPPED AI_ALL
IPv4: 127.0.0.1
IPv6: ::1
[AF_UNSPEC] AI_ADDRCONFIG
IPv4: 127.0.0.1
IPv4: 127.0.0.1
[AF_UNSPEC] AI_V4MAPPED AI_ADDRCONFIG
IPv4: 127.0.0.1
IPv4: 127.0.0.1
[AF_UNSPEC] AI_V4MAPPED AI_ALL AI_ADDRCONFIG
IPv4: 127.0.0.1
IPv4: 127.0.0.1
--------------The output shows that with _ai family equal to _AI UNSPEC (return both IPv4 and IPv6) and without the AI_ADDRCONFIG flag getaddrinforeturns two addresses, IPv4 and IPv6, which many administrators do not expect to see. This happens regardless of whether IPv6 is turned off in the kernel settings. If you /etc/hostsremove the address ::1, then IPv6 addresses will completely disappear from the output getaddrinfo(with the flag AF_UNSPEC).
With IPv6 enabled and having ::1in /etc/hosts, IPv4 and IPv6 will be returned. To prevent the return of the IPv6 address, comment out the IPv6 address in /etc/hosts. If the addresses are found in /etc/hosts, then in dns and myhostname the glibc database will not get into.
It remains to check how it behaves getaddrinfofor the dns database. To do this, I’ll leave only the dns database /etc/nsswitch.conffor hosts and resolve google.com. The output below is with IPv6 enabled.
$ sysctl -w net.ipv6.conf.default.disable_ipv6=0
$ sysctl -w net.ipv6.conf.all.disable_ipv6=0
$ ./getaddrinfo google.com
IP addresses for google.com:
[AF_INET] AI_V4MAPPED
IPv4: 216.58.215.78
[AF_INET] AI_V4MAPPED AI_ALL
IPv4: 216.58.215.78
[AF_INET] AI_ADDRCONFIG
IPv4: 216.58.215.78
[AF_INET] AI_V4MAPPED AI_ADDRCONFIG
IPv4: 216.58.215.78
[AF_INET] AI_V4MAPPED AI_ALL AI_ADDRCONFIG
IPv4: 216.58.215.78
--------------
[AF_INET6] AI_V4MAPPED
IPv6: 2a00:1450:401b:806::200e
[AF_INET6] AI_V4MAPPED AI_ALL
IPv6: ::ffff:216.58.215.78
IPv6: 2a00:1450:401b:806::200e
[AF_INET6] AI_ADDRCONFIG
IPv6: 2a00:1450:401b:806::200e
[AF_INET6] AI_V4MAPPED AI_ADDRCONFIG
IPv6: 2a00:1450:401b:806::200e
[AF_INET6] AI_V4MAPPED AI_ALL AI_ADDRCONFIG
IPv6: ::ffff:216.58.215.78
IPv6: 2a00:1450:401b:806::200e
--------------
[AF_UNSPEC] AI_V4MAPPED
IPv4: 216.58.215.78
IPv6: 2a00:1450:401b:806::200e
[AF_UNSPEC] AI_V4MAPPED AI_ALL
IPv4: 216.58.215.78
IPv6: 2a00:1450:401b:806::200e
[AF_UNSPEC] AI_ADDRCONFIG
IPv4: 216.58.215.78
IPv6: 2a00:1450:401b:806::200e
[AF_UNSPEC] AI_V4MAPPED AI_ADDRCONFIG
IPv4: 216.58.215.78
IPv6: 2a00:1450:401b:806::200e
[AF_UNSPEC] AI_V4MAPPED AI_ALL AI_ADDRCONFIG
IPv4: 216.58.215.78
IPv6: 2a00:1450:401b:806::200e
--------------And here is the output with IPv6 turned off:
$ sysctl -w net.ipv6.conf.all.disable_ipv6=1
$ sysctl -w net.ipv6.conf.default.disable_ipv6=1
$ ./getaddrinfo google.com
IP addresses for google.com:
[AF_INET] AI_V4MAPPED
IPv4: 216.58.215.78
[AF_INET] AI_V4MAPPED AI_ALL
IPv4: 216.58.215.78
[AF_INET] AI_ADDRCONFIG
IPv4: 216.58.215.78
[AF_INET] AI_V4MAPPED AI_ADDRCONFIG
IPv4: 216.58.215.78
[AF_INET] AI_V4MAPPED AI_ALL AI_ADDRCONFIG
IPv4: 216.58.215.78
--------------
[AF_INET6] AI_V4MAPPED
IPv6: 2a00:1450:401b:806::200e
[AF_INET6] AI_V4MAPPED AI_ALL
IPv6: ::ffff:216.58.215.78
IPv6: 2a00:1450:401b:806::200e
[AF_INET6] AI_ADDRCONFIG
getaddrinfo: Name or service not known
[AF_INET6] AI_V4MAPPED AI_ADDRCONFIG
getaddrinfo: Name or service not known
[AF_INET6] AI_V4MAPPED AI_ALL AI_ADDRCONFIG
getaddrinfo: Name or service not known
--------------
[AF_UNSPEC] AI_V4MAPPED
IPv4: 216.58.215.78
IPv6: 2a00:1450:401b:806::200e
[AF_UNSPEC] AI_V4MAPPED AI_ALL
IPv4: 216.58.215.78
IPv6: 2a00:1450:401b:806::200e
[AF_UNSPEC] AI_ADDRCONFIG
IPv4: 216.58.215.78
[AF_UNSPEC] AI_V4MAPPED AI_ADDRCONFIG
IPv4: 216.58.215.78
[AF_UNSPEC] AI_V4MAPPED AI_ALL AI_ADDRCONFIG
IPv4: 216.58.215.78
--------------As you can see, the situation with AI_ADDRCONFIG is very similar.
In conclusion, I will give an example of how not taking into account all of the above, to plunge into problems. IPv6 enabled, /etc/nsswitch.confstandard.
$ cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
$ grep hosts /etc/nsswitch.conf
hosts: files dns myhostname
$ cat /etc/resolv.conf
nameserver 127.0.0.11
options ndots:0
$ sysctl net.ipv6.conf.all.disable_ipv6 net.ipv6.conf.default.disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.default.disable_ipv6 = 0What will return host localhostor dig ANY localhost? What will return getaddrinfo, for example, with flags like nginx?
$ dig +noall +answer ANY localhost @127.0.0.11
localhost. 0 IN A 127.0.0.1
$ host localhost
localhost has address 127.0.0.1
$ ./getaddrinfo localhost
IP addresses for localhost:
[AF_UNSPEC] AI_ADDRCONFIG
IPv6: ::1
IPv4: 127.0.0.1nginx will try to connect to two addresses: 127.0.0.1and ::1, and the application may not expect this. Source for errors.
What conclusion can be drawn from everything written?
- If you are developing an application that works with a network, keep in mind that IPv6 can be disabled;
- When setting up the server, consider the presence of NSS and the sequence of checks, use getent or my code to check;
- If you don’t know how the application works, then if you have the source codes, you can see with which flags the
addresses resolve and check with the system settings; - If there are no source codes, then we can make an assumption about the flags used by the behavior in the strace output.
PS Subscribe to my channel About IT without ties . I talk about the fundamental problems in the IT business and how to solve them, about the request for promotions and about building a career as a manager or your own business.