VXLAN MacSec: infrastructure connectivity between DC

Connecting Distributed Infrastructure with VXLAN and MacSec

To link virtual machines across data centers without public internet, we use a VXLAN tunnel secured with MacSec encryption. This delivers L2 connectivity with minimal overhead. The setup involves two providers: DC1 (IP 192.0.2.135 on eth0) hosting databases and backups, and DC2 (IP 198.51.100.75 on ens1) running the website and backups.

On each node, we create a VXLAN interface with VNI 1 and UDP port 4789. MAC addresses are set manually for predictability.

DC1:

sudo ip link add vxlan0 type vxlan id 1 dstport 4789 dev eth0
sudo ip link set vxlan0 up
sudo ip link set vxlan0 address de:ad:be:ef:00:01
sudo bridge fdb append to 00:00:00:00:00:00 dst 198.51.100.75 dev vxlan0

DC2:

sudo ip link add vxlan0 type vxlan id 1 dstport 4789 dev ens1
sudo ip link set vxlan0 up
sudo ip link set vxlan0 address de:ad:be:ef:00:02
sudo bridge fdb append to 00:00:00:00:00:00 dst 192.0.2.135 dev vxlan0

After assigning IPs (e.g., 10.255.255.1/2), test connectivity with ping and SSH. Tcpdump shows VXLAN traffic encapsulating ICMP/TCP packets.

Encrypting Traffic with MacSec

Plain VXLAN offers no confidentiality. MacSec (802.1AE) layers on top for L2 encryption tied to MAC addresses. Overhead is low, and traffic is unreadable even in tcpdump.

DC1:

sudo ip link add link vxlan0 macsec0 type macsec encrypt on
sudo ip macsec add macsec0 tx sa 0 pn 1 on key 01 0123456789abcdef0123456789abcdef
sudo ip link set macsec0 up
sudo ip macsec add macsec0 rx address de:ad:be:ef:00:02 port 1
sudo ip macsec add macsec0 rx address de:ad:be:ef:00:02 port 1 sa 0 pn 1 on key 02 fedcba9876543210fedcba9876543210
sudo ip address add 192.168.255.1/24 dev macsec0

DC2:

sudo ip link add link vxlan0 macsec0 type macsec encrypt on
sudo ip macsec add macsec0 tx sa 0 pn 1 on key 02 fedcba9876543210fedcba9876543210
sudo ip link set macsec0 up
sudo ip macsec add macsec0 rx address de:ad:be:ef:00:01 port 1
sudo ip macsec add macsec0 rx address de:ad:be:ef:00:01 port 1 sa 0 pn 1 on key 01 0123456789abcdef0123456789abcdef
sudo ip address add 192.168.255.2/24 dev macsec0

In tcpdump, packets appear as encrypted 802.1AE frames with PN and SCI. Inner traffic (ICMP, TCP) stays hidden.

MacSec advantages over TLS:

• Operates at L2 level.
• Minimal overhead.
• MAC-bound without extra protocols.

Scaling to a Third Node

To add a home lab (DC3, IP 203.0.113.52 on enp3s0 with monitoring and backups), reuse the same VNI. Add FDB entries and MacSec RX/TX keys.

DC3:

sudo ip link add vxlan0 type vxlan id 1 dstport 4789 dev enp3s0
sudo ip link set vxlan0 up
sudo bridge fdb append to 00:00:00:00:00:00 dst 198.51.100.75 dev vxlan0
sudo bridge fdb append to 00:00:00:00:00:00 dst 192.0.2.135 dev vxlan0
sudo ip link add link vxlan0 macsec0 type macsec encrypt on
sudo ip macsec add macsec0 tx sa 0 pn 1 on key 03 abcdef0123456789abcdef0123456789
sudo ip link set macsec0 up
sudo ip macsec add macsec0 rx address de:ad:be:ef:00:01 port 1 sa 0 pn 1 on key 01 0123456789abcdef0123456789abcdef
sudo ip macsec add macsec0 rx address de:ad:be:ef:00:02 port 1 sa 0 pn 1 on key 02 fedcba9876543210fedcba9876543210
sudo ip address add 192.168.255.3/24 dev macsec0

On DC1 and DC2, add:

• FDB for 203.0.113.52.
• RX address de:ad:be:ef:00:03 with key 03.

Key Takeaways

• L2 Connectivity: VXLAN provides seamless networking across providers without NAT.
• Encryption: MacSec secures traffic at the link layer, keeping payloads hidden from tcpdump.
• Scalability: New nodes join by adding FDB entries and keys—no tunnel rebuilds needed.
• Efficiency: Low CPU/RAM overhead compared to IPsec or WireGuard.
• Compatibility: Works on Linux kernels 4.7+ (VXLAN) and 4.6+ (MacSec).

— Editorial Team