July 2, 2012 at 18:37UDP and response delivery problem
- Transfer
Below is a translation of an article about the problem of working with udp in network applications. The translator allowed himself to change examples: in the source text other network addresses and ruby code. The translation used a simple script on the pearl. The essence of the problem and the solution to this do not change.
In addition, my comments were added in places (in brackets, in italics).
The picture for attracting attention is taken from the text of the wonderful book “ learnyousomeerlang.com ”
Hard work of lightweight protocols
Sometimes it starts to seem that protocols without establishing a connection do not justify the whole mess that they cause.
As an example, let’s analyze the situation with receiving a response when a UDP datagram with an initial request is sent to an additional IP address on the interface (alias or secondary IP).
There is an eth1 interface:
$ ip a add 192.168.1.235/24 dev eth1 && ip a ls dev eth1
2: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:30:84:9e:95:60 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.47/24 brd 192.168.1.255 scope global eth1
inet 192.168.1.235/24 scope global secondary eth1
inet6 fe80::230:84ff:fe9e:9560/64 scope link
valid_lft forever preferred_lft forever
What does the code usually look like for receiving a package by udp? Well, an echo server might look very much like the one under the cat:
echo_server.pl
#!/usr/bin/perl
use IO::Socket::INET;
# flush after every write
$| = 1;
my ($socket,$received_data);
my ($peeraddress,$peerport);
$socket = new IO::Socket::INET (
MultiHomed => '1',
LocalAddr => $ARGV[0],
LocalPort => defined ($ARGV[1])?$ARGV[1]:'5000',
Proto => 'udp'
) or die "ERROR in Socket Creation : $! \n";
print "Waiting for data...";
while(1)
{
$socket->recv($recieved_data,1024);
$peer_address = $socket->peerhost();
$peer_port = $socket->peerport();
chomp($recieved_data);
print "\n($peer_address , $peer_port) said : $recieved_data";
#send the data to the client at which the read/write operations done recently.
$data = "echo: $recieved_data\n";
$socket->send("$data");
}
$socket->close();
This is a fairly simple pearl script that will show who the udp packet came from, the contents of the packet and send the packet back to the sender. Nowhere is easier. Now run our server:
$ ./echo_server.pl
Waiting for data...
Let's see what he listens to:
$ netstat -unpl | grep perl
udp 0 0 0.0.0.0:5000 0.0.0.0:* 9509/perl
And after that, connect from the remote machine to our server using the primary IP:
-bash-3.2$ nc -u 192.168.1.47 5000
test1
echo: test1
test2
echo: test2
How it looks in tcpdump on our machine (well, or it should look):
-bash-3.2$ tcpdump -i eth1 -nn port 5000
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
17:41:00.517186 IP 192.168.3.11.44199 > 192.168.1.47.5000: UDP, length 6
17:41:00.517351 IP 192.168.1.47.5000 > 192.168.3.11.44199: UDP, length 12
17:41:02.307634 IP 192.168.3.11.44199 > 192.168.1.47.5000: UDP, length 6
17:41:02.307773 IP 192.168.1.47.5000 > 192.168.3.11.44199: UDP, length 12
Just fantastic - I send the package and get the package back. In netcat, we get back no matter what we print (a ridiculous effect if you print "arrows").
And now the same thing to the secondary address on the same interface:
-bash-3.2$ nc -u 192.168.1.235 5000
test1
test2
How this craziness looks in tcpdump this time:
-bash-3.2$ tcpdump -i eth1 -nn port 5000
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
17:48:32.467167 IP 192.168.3.11.34509 > 192.168.1.235.5000: UDP, length 6
17:48:32.467292 IP 192.168.1.47.5000 > 192.168.3.11.34509: UDP, length 12
17:48:33.667182 IP 192.168.3.11.34509 > 192.168.1.235.5000: UDP, length 6
17:48:33.667332 IP 192.168.1.47.5000 > 192.168.3.11.34509: UDP, length 12
Well, of course, no self-respecting network stack is going to receive packets from a completely unfamiliar address, even if the ports are correct. Thus, the client will never receive return packets and will think that the server is simply dropping its requests.
What happens at first glance seems like complete nonsense. But in fact, this is a common defect for a protocol without setting up a session, such as UDP. You see, our socket listens to any address (the empty LocalAddr parameter is passed to the system as an address of the form “0.0.0.0”, any available one, which makes the socket listen on all available addresses. And no, I don’t know why this is so This is not a particularly intuitive action). When we receive a packet in our application using socket-> recv (), we do not get information about what specific address the packet was sent to. We only know that the operating system decided that the package was for us (here you have the encapsulation). All we know is where the package came from. And due to the fact that the kernel does not store any information about connections for the socket (the kernel is a logical thing, asked without connections - it will be without connections) when it comes time to send the packet back, all we can tell is where to send the packet. (In pearl, this is done implicitly. The address and port of the sender of the datagram are associated with the $ socket object, so you do not need to specify it in the send call).
But the real puzzle begins when we try to put down the sender address in the response datagram. Once again: the kernel does not store any information about the sender or receiver, since we work without connections. And since we listen to “any” interface, the operating system thinks that it has a carte blanche to send a packet from the address that it “likes”. In Linux, it seems, the main address of the interface from which the packet will be sent is selected. (Actually, the address is determined in accordance with RFC1122, 3.3.4.2 “Multihoming Requirements” , according to the routing table - the translator’s note) . So for the common case of “one address - one interface” - everything works. But as soon as it comes to less common situations, nuances begin to appear.
The solution is to create sockets that listen to specific addresses. And send a packet from these sockets: the kernel will know which address you want to send packets from and everything will be fine. Looks pretty simple, huh? Well, of course, any sane network application already does that, huh? So it is obvious that the realization of UDP in Ruby just crap (in the original source code to Ruby - translator's note;). That’s what I thought at the beginning, and I don’t blame you if you thought the same. But while the Rubikon of war with the authors of the Ruby's UDPSocket has not been crossed, let's do a little experiment with other commonly used applications. For example, SNMPd. The daemon from the net-snmpd package in ubunt is subject to the same problem as our test application above. It does not seem that this is some kind of new rake that they just stepped on and scattered with a bunch of patches for fixes.
So in general, everyone suffers from the same "disease." By "everyone" is meant "some UDP servers." There is a certain amount of software that is not subject to a similar problem with aliases on interfaces. Bind immediately comes to mind and NTPd works fine if it is started after you have configured all the interfaces. What is the difference? The difference is that these services are somewhat “smarter” and bind to all addresses in the system separately. On the example of bind:
$ netstat -lun |grep :53
udp 0 0 192.168.1.47:53 0.0.0.0:*
udp 0 0 192.168.1.47:53 0.0.0.0:*
udp 0 0 127.0.0.1:53 0.0.0.0:*
This is very cool and solves the problem. The exception is the situation when you add an extra alias after the daemon starts. Bind will not pick up the new address and you will have to restart the server. In addition, this complicates the code a bit, since you have to get along with a bunch of sockets inside the program (for example, use select () instead of just blocking it on an attempt to receive.) In general, no one likes unnecessary difficulties, but you can handle this . However, the real problem is the “do not add addresses after the daemon starts” rule. The need to check if ip addresses have been added to the system and restart the service after adding the address will become a real problem.
However, there is some workaround for this problem as well. Here we recall ntpd. The ports he listens to are as follows:
$ netstat -nlup | grep 123
udp 0 0 192.168.1.235:123 0.0.0.0:*
udp 0 0 192.168.1.47:123 0.0.0.0:*
udp 0 0 127.0.0.3:123 0.0.0.0:*
udp 0 0 127.0.0.1:123 0.0.0.0:*
udp 0 0 0.0.0.0:123 0.0.0.0:*
udp6 0 0 fe80::230:84ff:fe9e:123 :::*
udp6 0 0 ::1:123
NTPd listens for each address individually and additionally listens on any address available to the system. I do not know exactly why this is necessary. If you just listen to each address separately, then everything will be fine, as is the case with the bind. But if you add another address to the interface after starting ntpd, then the same problem starts to appear, as in the case of the udp-echo server. So I don’t think that listening on any interface gives any plus. However, this makes ntpd behave somewhat differently from Bind: when you send a packet to the interface added after the start of Bind, it simply ignores you (it does not have a socket that listens to your requests). Ntpd tries to send a response and suffers from the problem of an incorrect address in the responses.(But you can change the primary address on the interfaces and create new interfaces, note translator).
At the moment, the best solution seems to follow the path of Bind and ntpd and listen to all addresses separately with the "focus" from ntpd: listen additionally to 0.0.0.0. At the same time, if I received a packet at 0.0.0.0, then I need to start scanning the addresses available in the system and bind additionally to them. This should solve the problem.
It remains only to make it work (and solve a lot of problems that will probably come out on the way). Wish me good luck. The cries of pain and torment that you hear (no matter where you are) are probably mine.
UPD: an interesting explanation from Quasar_ru appeared in the comments. All the same, the implementation of UDP in scripting languages is ambiguous: in pure C, you can write a client application that can receive a response from the server from another address. The benefits of such an implementation are controversial, but implementation is possible.
