ARP: The nuances of Cisco equipment and interesting cases. Part 2
Hi Habr! In the previous part of the article, we examined the features of ARP on Cisco routers related to NAT rules and the Proxy ARP function. In this post I’ll try to make out the differences in ARP between Cisco routers and Cisco ASA firewalls. At the end of the article, I will share some interesting cases from practice related to the work of ARP.
NAT and Proxy ARP on Cisco ASA Firewalls
The Cisco ASA firewall behavior for ARP for NAT rules and Proxy ARP settings is different from Cisco routers. By default, when you configure NAT rules on the Cisco ASA, the device will respond to ARP requests that match the internal global addresses of the NAT rules. At the same time, this behavior does not depend on the ownership of the internal global IP address of the ASA interface IP subnet. This behavior ensures that the sysopt noproxyarp <interface name> option is configured .
Consider the examples based on the following simple topology:
Cisco ASA Interface Settings:
interface Vlan1 nameif inside security-level 100 ip address 192.168.20.1 255.255.255.0 ! interface Vlan2 nameif outside security-level 0 ip address 198.18.0.1 255.255.255.0
NAT rule and access list settings on the outside interface:
object network TEST-PC-tcp3389 host 192.168.20.5 nat (inside,outside) static 198.18.99.2 service tcp 3389 3389 access-list acl-outside-in extended permit tcp any object TEST-PC-tcp3389 eq 3389 access-group acl-outside-in in interface outside
As you can see from the presented settings, we publish the tcp 3389 (RDP) port under the address 198.18.99.2, which does not belong to the IP subnet of the Cisco ASA interface.
Check the sysopt noproxyarp option:
asa# sh run all sysopt . . . no sysopt noproxyarp inside no sysopt noproxyarp outside
It can be seen that the option is not set (noproxyarp is not enabled). Let's try to open the tcp port 3389 of address 198.18.99.2 and at the same time see the sniffer:
Success: Cisco ASA answers the ARP request and the port opens.
Let's try to set the sysopt noproxyarp outside option . We clear the arc-cache on the computer connected to the outside interface and try to open the port again: the
ASA no longer responds to ARP requests to the target IP address. However, it does not matter if the destination IP address is on the IP subnet of the ASA interface. With the sysopt option set, noproxyarp ASA will respond to ARP requests addressed exclusively to the interface IP address.
Do not compare the configuration of the ip proxy-arp interface of the router with the configuration of the optionsysopt noproxyarp Cisco ASA. The Cisco ASA option does not include proxying any ARP requests through the firewall. This option is solely responsible for serving internal global IP addresses of NAT rules and static entries in the ASA ARP table configured with the alias keyword .
Secondary IP Address on ASA
In some cases, ARP response functionality needs to be disabled for specific NAT rules. To do this, use the no-proxy-arp keyword in the NAT settings. The most common example is setting up NAT exceptions when using a VPN on the Cisco ASA. Suppose the local area network behind the Cisco ASA is 192.168.20.0/24, the local area network on the remote side of the Site-to-Site VPN is 192.168.30.0/24. The NAT exception for this VPN can be configured as follows:
object network local-LAN subnet 192.168.20.0 255.255.255.0 object network remote-LAN subnet 192.168.30.0 255.255.255.0 nat (any,any) source static local-LAN local-LAN destination static remote-LAN remote-LAN
The presented configuration indicates for the Cisco ASA that the local-LAN 192.168.20.0/24 IP subnet is fully published on each Cisco ASA (any, any in the NAT rule) under the internal global addresses of the same local-LAN IP subnet. Therefore, with this configuration, the Cisco ASA will respond to any ARP request to the destination IP address from the IP subnet 192.168.20.0/24 that arrives at any interface, including the inside interface.
Imagine a situation where a host with the address 192.168.20.5 wants to access the host from the same IP subnet with the address 192.168.20.6. An ARP request is generated to the target address 192.168.20.6. ARP request is broadcast and reaches both the target host and insideCisco ASA interface. The configured NAT rule causes the Cisco ASA to respond with its MAC address to the ARP request. If the ARP response from the Cisco ASA arrives before the response from the target host, all traffic directed to the target host will go to the Cisco ASA, where it will be safely dropped.
In the presented example, the Cisco ASA will work as a “black hole” for the local IP subnet, trying to “suck” all traffic onto itself. At the same time, the policy NAT element (setting NAT after the destinations static keywords ) does not save the situation. Compared with the router, this setting on the Cisco ASA is similar in effect to the joint configuration of ip proxy-arp and ip local-proxy-arpon the router interface. To avoid this effect, add the no-proxy-arp keyword in the NAT rule on the Cisco ASA :
nat (any,any) source static local-LAN local-LAN destination static remote-LAN remote-LAN no-proxy-arp
Note : the described effect can be avoided by specifying exact interfaces in the NAT rule settings, instead of any keywords. For example, nat (inside, outside) ...
Before proceeding to the description of cases from practice, I will highlight the main points of the second part of the article:
- By default, the Cisco ASA will respond to ARP requests for the internal global IP address of the NAT rule, regardless of the interface’s IP subnet. This behavior is controlled by setting the sysopt noproxyarp <interface name> option .
- Cisco ASA allows you to disable ARP responses for specific NAT rules using the no-proxy-arp keyword . In particular, for NAT exception rules, it is necessary to disable ARP responses in order to avoid communication problems on the local network.
- Proxy ARP functionality is not explicitly configured on the Cisco ASA, but the necessary effect can be achieved using NAT rules.
So, cases from practice. I must say right away that all the described problems occurred when I or my colleagues connected the new Cisco network equipment to Internet providers. In our experience, it is this scenario that is most often accompanied by problems with ARP.
Case No. 1. Secondary Provider IP Address
We connected the Cisco ASA firewall to the provider. Connection was successful and all services worked correctly. However, after some time, the connection disappeared. A detailed analysis showed that if you initiate an outgoing connection, then it works stably (traffic goes in both directions). The problem appears only with incoming connections from the Internet (for example, a remote connection to a router). In this case, a direct dependence of incoming connections on outgoing is traced: if there are outgoing connections, then incoming connections begin to work correctly. However, after some time, it is again impossible to connect remotely or “ping” the device from the Internet.
Since the physical and link layer are supposedly all right, we began to check the operation of ARP. We launched debug arp on the ASA and tried to clear arp-cache. From the debug messages we saw that the ASA correctly sends the ARP request and receives the ARP response from the provider without any problems. In this example, the IP of our ASA is 80.XX4, its MAC address is a0ec. ****. ****, the IP address of the provider is 80.XX1, the MAC address of the provider is aa43. ****. ****:
arp-send: arp request built from 80.X.X.4 a0ec.****.**** for 80.X.X.1 at 978772020 arp-refresh: Trying to refresh ARP for outside 80.X.X.1 arp-in: response at outside from 80.X.X.1 aa43.****.**** for 80.X.X.4 a0ec.****.**** having smac aa43.****.**** dmac a0ec.****.****\narp-set: added arp outside 80.X.X.1 aa43.****.**** and updating NPs at 978772020 arp-in: resp from 80.X.X.1 for 80.X.X.4 on outside at 978772020
However, after some time, they noticed a message in debug arp on the ASA:
arp-in: request at outside from 195.Y.Y.1 aa43.****.**** for 80.X.X.4 0000.0000.0000 having smac aa43.****.**** dmac ffff.ffff.ffff\narp-in: Arp packet received from 195.Y.Y.1 which is in different subnet than the connected interface 80.X.X.4/255.255.255.0
Judging by this message, the ASA receives an ARP request with a valid Sender MAC Address aa43. ****. **** but an invalid Sender IP Address - 195.YY1. This invalid ASP ARP request discards and does not send an ARP response.
Thus, when outgoing traffic exists, the ASA sends an ARP request (if necessary, when an appropriate entry in the ASA ARP table needs updating) to the provider and receives an ARP response. Thanks to the ARP request from the ASA, the provider’s equipment also receives an ASA record in its ARP table. However, when there is no outgoing traffic from the ASA for a long time and the ASA does not send an ARP request, the record about the ASA expires on the provider's equipment in the ARP table. The provider equipment tries to update the record by sending an ARP request, but does not receive a response. From here "floating" problems with communication appear.
It remains to understand why the provider’s equipment sends an ARP request with the wrong Sender IP Address. Later, the provider checked this situation on its part and even showed a dump of ARP traffic, which confirmed the ASA debug messages:
324: 22:03:41.056546 802.1Q vlan#2 P0 arp who-has 80.X.X.4 tell 195.Y.Y.1 325: 22:03:41.937329 802.1Q vlan#2 P0 arp who-has 80.X.X.4 tell 195.Y.Y.1 326: 22:03:42.822909 802.1Q vlan#2 P0 arp who-has 80.X.X.4 tell 195.Y.Y.1
It turned out that on the provider’s interface the address 195.YY1 was configured as primary, and the IP address 80.XX4, which was the default gateway for us, was configured as secondary. This setting fully explained the situation. In this case, the problem was solved by adding a static ARP record on the provider's equipment.
We had a completely similar situation on another site when we used a Cisco router to connect to the provider. On the equipment of the provider, our gateway was also configured as seconday IP address. In this case, to speed up the process of solving the problem, we added a secondary IP address on the router from the same subnet as the primary IP address on the provider's interface. After that, our router began to successfully respond to ARP requests from the provider, since an IP address from the same subnet as the address in the ARP request appeared on our interface.
Case No. 2. ARP incompatibility
We were tasked with replacing the Cisco ISR with a Cisco ASA firewall in one of our offices. The ASA firewall was preconfigured and sent to the installation point. After connecting to the provider, the Cisco ASA was not available for remote connection. At first glance, everything worked correctly on the device. The ASA firewall correctly determined the MAC address of the provider's router using the standard ARP request / ARP response procedure. Packets left the device’s external interface on the Internet. At the same time, nothing came to the ASA in the opposite direction. This fact was recorded by the built-in packet capture tools.
After contacting the provider, it was found that the packets from the ASA were successfully delivered to the provider's equipment, and the provider also saw response packets coming from the upstream equipment. After the router was reconnected, the traffic again started going correctly in both directions. This meant that the problem was somewhere at the junction between the provider and the ASA firewall. After re-contacting the provider with a detailed description of the problem, it was determined that the provider's equipment did not see the ASA firewall MAC address. The assembled demo stand proved the correct operation of the ASA (the role of the provider was played by the Cisco router). For some reason, the provider’s device did not record the ASA in the ARP table after receiving the ARP response from the ASA. Most interestingly, the ARP request coming from the Cisco ASA was not discarded.
As a result, the provider was asked to make a static binding in the ARP table. This case showed the ARP incompatibility of the provider's equipment with the Cisco ASA firewall. Unfortunately, the provider did not announce the manufacturer of its equipment.
Case No. 3. Gratuitous ARP
And again we connect to the Cisco ASA provider. This time instead of the MS TMG server. The peculiarity of this case is that MS TMG was connected to the provider's device through an L2 switch. It was supposed to connect the ASA also through the L2 switch:
So, we disconnect MS TMG and instead of it we connect Cisco ASA in the same port of the L2 switch. We see a standard situation: the traffic leaves the external Cisco ASA port, but there are no response packets. After contacting the provider, it turns out that the equipment of the provider still sees the MAC address of the MS TMG server behind the IP address that has passed to the Cisco ASA.
Further investigation revealed that the Cisco ASA firewall does not send a Gratuitous ARP message when the interface transitions from DOWN to UP. And since the equipment of the provider is connected through the L2 switch, when changing the device on our side, the channel to the provider does not drop, and the interface of the provider’s router always remains on. The only way to notify the provider in a timely manner that the device and MAC address has changed on our side is Gratuitous ARP. Otherwise, you will have to wait for the timeout of the ARP record from the provider, and this is usually 4 hours.
In this case, we made the no ip address, ip address xxxx yyyy commands on the Cisco ASA interface. After that, the ASA sent Gratuitous ARP and everything took off.
In this two-part article, I tried to consider the intricacies of ARP on Cisco equipment in cases of using network address translation (NAT), as well as the Proxy ARP functionality. Understood the differences in ARP between Cisco routers and Cisco ASA firewalls. At the end of the article, I examined the problems that arise when connecting to an Internet provider, due to the work of ARP.
The described cases show how important it is to remember the need to check ARP in the process of solving and eliminating network problems. I hope this article will be useful for a deeper understanding of ARP.
Waiting for comments. Maybe someone will be able to tell their own interesting cases related to the work of ARP.