Configuring a FullMesh network on Mikrotik via EoIP tunnels
The initial situation is this: there are 8 offices in different parts of the country, it is necessary to bring them into a single network so that the availability of each office is maximum in any disaster. As routers in all offices are Mikrotik. On the main site - CCR CCR1036-12G, on the rest - 1100 AHx2
In order to avoid problems with the Internet, 2 channels were stretched from different providers, we also reserved power and came to the question “what network should I build?”. As the name of the article shows, in the end we decided to build FullMesh.
This scheme fully meets the requirements of the management - if any Internet channel or even any office fails, the network remains connected. There was only a question with routing. Of the options was a universal bridge with RSTP, OSPF and static routes. Naturally, in the end, I chose OSPF - fewer problems than on static and less load on routers than on RSTP.
The setting itself and the finished config under the cut.
I decided to connect routers using EoIP tunnels, 2 for each pair - from the primary provider to the primary and from the backup to the backup. I will describe the configuration for one pair, since the rest are configured identically.
On the first router, create 2 tunnels:
We lift the tunnels from the second side:
We configure OSPF, we will exchange routes through the backbone zone.
On the first router:
On the second router:
Finally, add the addresses for the created tunnels:
On the first router:
On the second router:
We get 2 routers that exchange routes to their zones using OSPF. We repeat this procedure for all pairs of routers.
As a result, we get such a FullMesh network (I apologize in advance for the quality of the scheme - I did not find how to adequately draw a network diagram on Linux, because I used the online Gliffy drawing):
All routers are included in the common backbone area with id 0.0.0.0 + each of them is borderline for its own zone with an ID equal to the local IP of the router.
Immediately after the routers discover each other, they exchange the routes they know and select the best ones based on the cost of the link. In this configuration, the best route will always be the most direct (more precisely 2 direct routes - the main and the backup), in case of failure of which the traffic going through the failed route will be delivered through all the routers that remain available.
Thus, the technical work of the provider stopped scaring us, as well as periodic problems with the passage of GRE packets in certain directions - for a complete network connectivity, less than half of the existing tunnels are sufficient. Well, as a bonus, we got traffic balancing between tunnels - since the costs of the main and backup tunnels are the same, OSPF automatically sends traffic to both tunnels, balancing the load between them approximately equally.
If you have any questions or suggestions on optimizing this configuration - welcome to comment.
As promised, the ready-made config of one of the routers (all names and IPs are changed by agreement with the security service):
UPD: this config is merged from the test lab, and not from the sell. Since the writing of this article in the prod, they managed to abandon this scheme in favor of L2VPN from providers with OSPF for routing.
In order to avoid problems with the Internet, 2 channels were stretched from different providers, we also reserved power and came to the question “what network should I build?”. As the name of the article shows, in the end we decided to build FullMesh.
This scheme fully meets the requirements of the management - if any Internet channel or even any office fails, the network remains connected. There was only a question with routing. Of the options was a universal bridge with RSTP, OSPF and static routes. Naturally, in the end, I chose OSPF - fewer problems than on static and less load on routers than on RSTP.
The setting itself and the finished config under the cut.
I decided to connect routers using EoIP tunnels, 2 for each pair - from the primary provider to the primary and from the backup to the backup. I will describe the configuration for one pair, since the rest are configured identically.
On the first router, create 2 tunnels:
/interface eoip
ladd keepalive=1s,3 local-address=xxx.xxx.xxx.xxx name=FIRST remote-address=yyy.yyy.yyy.yyy tunnel-id=1
add keepalive=1s,3 local-address=zzz.zzz.zzz.zzz name=FIRST_BAK remote-address=www.www.www.www tunnel-id=2
We lift the tunnels from the second side:
/interface eoip
ladd keepalive=1s,3 local-address=yyy.yyy.yyy.yyy name=FIRST remote-address=xxx.xxx.xxx.xxx tunnel-id=1
add keepalive=1s,3 local-address=www.www.www.www name=FIRST_BAK remote-address=zzz.zzz.zzz.zzz tunnel-id=2
We configure OSPF, we will exchange routes through the backbone zone.
On the first router:
/routing ospf area
add area-id=192.168.0.1 name=FIRST
/routing ospf interface
add cost=10 dead-interval=5s hello-interval=1s interface=FIRST \
network-type=point-to-point use-bfd=yes
add cost=10 dead-interval=5s hello-interval=1s interface=FIRST_BAK \
network-type=point-to-point use-bfd=yes
/routing ospf network
add area=FIRST network=192.168.0.0/24
add area=backbone network=10.0.0.0/22
On the second router:
/routing ospf area
add area-id=192.168.1.1 name=SECOND
/routing ospf interface
add cost=10 dead-interval=5s hello-interval=1s interface=FIRST \
network-type=point-to-point use-bfd=yes
add cost=10 dead-interval=5s hello-interval=1s interface=FIRST_BAK \
network-type=point-to-point use-bfd=yes
/routing ospf network
add area=SECOND network=192.168.1.0/24
add area=backbone network=10.0.0.0/22
Finally, add the addresses for the created tunnels:
On the first router:
ip address add address=10.0.1.1/30 interface=FIRST network=10.0.1.0
ip address add address=10.0.1.5/30 interface=FIRST_BAK network=10.0.1.4
On the second router:
ip address add address=10.0.1.2/30 interface=FIRST network=10.0.1.0
ip address add address=10.0.1.6/30 interface=FIRST_BAK network=10.0.1.4
We get 2 routers that exchange routes to their zones using OSPF. We repeat this procedure for all pairs of routers.
As a result, we get such a FullMesh network (I apologize in advance for the quality of the scheme - I did not find how to adequately draw a network diagram on Linux, because I used the online Gliffy drawing):
All routers are included in the common backbone area with id 0.0.0.0 + each of them is borderline for its own zone with an ID equal to the local IP of the router.
Immediately after the routers discover each other, they exchange the routes they know and select the best ones based on the cost of the link. In this configuration, the best route will always be the most direct (more precisely 2 direct routes - the main and the backup), in case of failure of which the traffic going through the failed route will be delivered through all the routers that remain available.
Thus, the technical work of the provider stopped scaring us, as well as periodic problems with the passage of GRE packets in certain directions - for a complete network connectivity, less than half of the existing tunnels are sufficient. Well, as a bonus, we got traffic balancing between tunnels - since the costs of the main and backup tunnels are the same, OSPF automatically sends traffic to both tunnels, balancing the load between them approximately equally.
If you have any questions or suggestions on optimizing this configuration - welcome to comment.
As promised, the ready-made config of one of the routers (all names and IPs are changed by agreement with the security service):
Config
# aug/23/2015 19:15:28 by RouterOS 6.30.2
# software id = 4RCZ-RTPX
#
/interface ethernet
set [ find default-name=ether10 ] mac-address=4C:5E:0C:5A:64:22 name=\
ISP2
set [ find default-name=ether1 ] mac-address=4C:5E:0C:5A:64:19
set [ find default-name=ether2 ] mac-address=4C:5E:0C:5A:64:1A master-port=\
ether1
set [ find default-name=ether3 ] mac-address=4C:5E:0C:5A:64:1B master-port=\
ether1
set [ find default-name=ether4 ] mac-address=4C:5E:0C:5A:64:1C master-port=\
ether1
set [ find default-name=ether5 ] mac-address=4C:5E:0C:5A:64:1D master-port=\
ether1
set [ find default-name=ether6 ] mac-address=4C:5E:0C:5A:64:1E
set [ find default-name=ether7 ] mac-address=4C:5E:0C:5A:64:1F
set [ find default-name=ether8 ] mac-address=4C:5E:0C:5A:64:20
set [ find default-name=ISP1 ] mac-address=4C:5E:0C:5A:64:21 name=ISP1
set [ find default-name=ether11 ] mac-address=4C:5E:0C:5A:64:23
set [ find default-name=ether12 ] mac-address=4C:5E:0C:5A:64:24
set [ find default-name=ether13 ] mac-address=4C:5E:0C:5A:64:25
/interface eoip
add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:74:DC:6B:70:C1 \
name=FIRST remote-address=xxx.xxx.xxx.xxx tunnel-id=1
add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:74:DC:6B:70:C1 \
name=FIRST_BAK remote-address=xxx.xxx.xxx.xxx tunnel-id=2
add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:B8:B3:AB:DB:17 \
name=SECOND remote-address=xxx.xxx.xxx.xxx tunnel-id=3
add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:3B:12:E5:7E:BC \
name=SECOND_BAK remote-address=xxx.xxx.xxx.xxx tunnel-id=4
add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:3B:12:E5:7E:BC \
name=THIRD remote-address=xxx.xxx.xxx.xxx tunnel-id=5
add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:B8:B3:AB:DB:17 \
name=THIRD_BAK remote-address=xxx.xxx.xxx.xxx tunnel-id=6
add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:B8:B3:AB:DB:17 \
name=FOURTH remote-address=xxx.xxx.xxx.xxx tunnel-id=7
add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:3B:12:E5:7E:BC \
name=FOURTH_BAK remote-address=xxx.xxx.xxx.xxx tunnel-id=8
add keepalive=1s,3 local-address=xxx.xx.xxx.xxx mac-address=02:3B:12:E5:7E:BC \
name=FIFTH remote-address=xxx.xxx.xxx.xxx tunnel-id=9
add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:B8:B3:AB:DB:17 \
name=FIFTH_BAK remote-address=xxx.xxx.xxx.xxx tunnel-id=10
add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:B8:B3:AB:DB:17 \
name=SIX remote-address=xxx.xxx.xxx.xxx tunnel-id=11
add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:3B:12:E5:7E:BC \
name=SIX_BAK remote-address=xxx.xxx.xxx.xxx tunnel-id=12
add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:B8:B3:AB:DB:17 \
name=SEVENTH remote-address=xxx.xxx.xxx.xxx tunnel-id=13
add keepalive=1s,3 local-address=xxx.xxx.xxx.xxx mac-address=02:3B:12:E5:7E:BC \
name=SEVENTH_BAK remote-address=xxx.xxx.xxx.xxx tunnel-id=14
/routing ospf area
add area-id=192.168.0.1 name=LOCAL
/snmp community
set [ find default=yes ] addresses=192.168.0.0/16
/system logging action
set 0 memory-lines=100
set 1 disk-lines-per-file=100
/tool user-manager customer
set admin access=\
own-routers,own-users,own-profiles,own-limits,config-payment-gw
/user group
set read policy="read,test,winbox,sniff,sensitive,!local,!telnet,!ssh,!ftp,!re\
boot,!write,!policy,!password,!web,!api"
/ip firewall connection tracking
set generic-timeout=1m tcp-close-timeout=5s tcp-close-wait-timeout=5s \
tcp-established-timeout=1m tcp-fin-wait-timeout=5s tcp-last-ack-timeout=\
5s tcp-time-wait-timeout=5s udp-stream-timeout=1m
/ip address
add address=192.168.0.1/24 interface=ether1 network=192.168.0.0
add address=xxx.xxx.xxx.xxx/xx interface=ISP2 network=xxx.xxx.xxx.xxx
add address=xxx.xxx.xxx.xxx/xx interface=ISP1 network=xxx.xxx.xxx.xxx
add address=10.0.1.18/30 interface=FIRST_BAK network=10.0.1.16
add address=10.0.1.46/30 interface=SECOND network=10.0.1.44
add address=10.0.1.50/30 interface=SECOND_BAK network=10.0.1.48
add address=10.0.1.73/30 interface=THIRD network=10.0.1.72
add address=10.0.1.77/30 interface=THIRD_BAK network=10.0.1.76
add address=10.0.1.86/30 interface=FOURTH network=10.0.1.84
add address=10.0.1.90/30 interface=FOURTH_BAK network=10.0.1.88
add address=10.0.1.94/30 interface=FIFTH network=10.0.1.92
add address=10.0.1.98/30 interface=FIFTH_BAK network=10.0.1.96
add address=10.0.1.102/30 interface=FIRST network=10.0.1.100
add address=10.0.1.217/30 interface=SIX network=10.0.1.216
add address=10.0.1.221/30 interface=SIX_BAK network=10.0.1.220
add address=10.0.1.225/30 interface=SEVENTH network=10.0.1.224
add address=10.0.1.229/30 interface=SEVENTH_BAK network=10.0.1.228
/ip dns
set servers=8.8.4.4
/ip firewall filter
add action=drop chain=input comment="Drop Invalid" connection-state=invalid
add action=drop chain=forward connection-state=invalid
add chain=input comment=Established connection-state=established
add chain=input comment=Tunnels protocol=gre
add chain=input comment=WinBox dst-port=8291 protocol=tcp
add chain=input comment=NTP dst-port=123 protocol=udp
add chain=input comment="From Local" src-address=192.168.0.0/16
add chain=input comment=Ping protocol=icmp
add action=drop chain=input comment="DONT MOVE - DROP" in-interface=\
ISP2
add action=drop chain=input comment="DONT MOVE - DROP" in-interface=\
ISP1
/ip firewall mangle
add action=change-mss chain=forward new-mss=clamp-to-pmtu protocol=tcp \
tcp-flags=syn tcp-mss=1460-65535
/ip firewall nat
add action=masquerade chain=srcnat out-interface=\
ISP2
add action=masquerade chain=srcnat out-interface=ISP1
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set irc disabled=yes
set h323 disabled=yes
set sip disabled=yes
set pptp disabled=yes
/ip route
add check-gateway=ping distance=1 gateway=8.8.8.8
add distance=2 gateway=10.10.20.1
add check-gateway=ping distance=1 dst-address=8.8.8.8/32 gateway=10.10.10.1 scope=10
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set api disabled=yes
/routing ospf interface
add interface=FIRST network-type=point-to-point
add interface=FIRST_BAK network-type=point-to-point
add interface=SECOND network-type=point-to-point
add interface=SECOND_BAK network-type=point-to-point
add interface=THIRD network-type=point-to-point
add interface=THIRD_BAK network-type=point-to-point
add interface=FOURTH network-type=point-to-point
add interface=FOURTH_BAK network-type=point-to-point
add interface=FIFTH network-type=point-to-point
add interface=FIFTH_BAK network-type=point-to-point
add interface=SIX network-type=point-to-point
add interface=SIX_BAK network-type=point-to-point
add interface=SEVENTH network-type=point-to-point
add interface=SEVENTH_BAK network-type=point-to-point
add interface=ether1 network-type=broadcast passive=yes
/routing ospf network
add area=backbone network=10.0.0.0/22
add area=FIRST network=192.168.0.0/24
/system clock
set time-zone-autodetect=no time-zone-name=Europe/Kiev
/system identity
set name=MikroTik
/system resource irq rps
set ether1 disabled=yes
set ether2 disabled=yes
set ether3 disabled=yes
set ether4 disabled=yes
set ether5 disabled=yes
set ether6 disabled=yes
set ether7 disabled=yes
set ether8 disabled=yes
set ISP1 disabled=yes
set ISP2 disabled=yes
set ether11 disabled=yes
/system scheduler
add interval=6h name=schedule1 on-event=BackupToMail policy=\
ftp,reboot,read,write,policy,test,password,sniff,sensitive start-date=\
oct/28/2014 start-time=01:00:00
add interval=6h name=schedule2 on-event=BackupToFTP policy=\
ftp,reboot,read,write,policy,test,password,sniff,sensitive start-date=\
oct/28/2014 start-time=01:00:00
/system script
add name=BackupToMail owner=root policy=\
ftp,reboot,read,write,policy,test,password,sniff,sensitive source="{\r\
\n:log info \"Starting Backup Script...\";\r\
\n:local sysname [/system identity get name];\r\
\n:local sysver [/system package get system version];\r\
\n:log info \"Flushing DNS cache...\";\r\
\n/ip dns cache flush;\r\
\n:delay 2;\r\
\n:log info \"Deleting last Backups...\";\r\
\n:foreach i in=[/file find] do={:if ([:typeof [:find [/file get \$i name]\
\_\\\r\
\n\"\$sysname-backup-\"]]!=\"nil\") do={/file remove \$i}};\r\
\n:delay 2;\r\
\n:local smtpserv [:resolve \"smtp.gmail.com\"];\r\
\n:local Eaccount \"xxxxxx@gmail.com\";\r\
\n:local TOaccount \"xxxxxx@gmail.com\";\r\
\n:local pass \"xxxxxx\";\r\
\n:local backupfile (\"\$sysname-backup-\" . \\\r\
\n[:pick [/system clock get date] 7 11] . [:pick [/system \\\r\
\nclock get date] 0 3] . [:pick [/system clock get date] 4 6] . \".backup\
\");\r\
\n:log info \"Creating new Full Backup file...\";\r\
\n/system backup save name=\$backupfile;\r\
\n:delay 2;\r\
\n:log info \"Sending Full Backup file via E-mail...\";\r\
\n/tool e-mail send from=\"<\$Eaccount>\" to=\$TOaccount server=\$smtpserv\
\_\\\r\
\nport=587 user=\$Eaccount password=\$pass tls=yes file=\$backupfile \\\r\
\nsubject=(\"\$sysname Full Backup (\" . [/system clock get date] . \")\")\
\_\\\r\
\nbody=(\"\$sysname full Backup file see in attachment.\\nRouterOS version\
: \\\r\
\n\$sysver\\nTime and Date stamp: \" . [/system clock get time] . \" \" . \
\\\r\
\n[/system clock get date]);\r\
\n:delay 5;\r\
\n:local exportfile (\"\$sysname-backup-\" . \\\r\
\n[:pick [/system clock get date] 7 11] . [:pick [/system \\\r\
\nclock get date] 0 3] . [:pick [/system clock get date] 4 6] . \".rsc\");\
\r\
\n:log info \"Creating new Setup Script file...\";\r\
\n/export file=\$exportfile;\r\
\n:delay 2;\r\
\n:log info \"Sending Setup Script file via E-mail...\";\r\
\n/tool e-mail send from=\"<\$Eaccount>\" to=\$TOaccount server=\$smtpserv\
\_\\\r\
\nport=587 user=\$Eaccount password=\$pass tls=yes file=\$exportfile \\\r\
\nsubject=(\"\$sysname Setup Script Backup (\" . [/system clock get date] \
. \\\r\
\n\")\") body=(\"\$sysname Setup Script file see in attachment.\\nRouterOS\
\_\\\r\
\nversion: \$sysver\\nTime and Date stamp: \" . [/system clock get time] .\
\_\" \\\r\
\n\" . [/system clock get date]);\r\
\n:delay 5;\r\
\n:log info \"All System Backups emailed successfully.\\nBackuping complet\
ed.\";\r\
\n}"
add name=BackupToFTP owner=antony policy=\
ftp,reboot,read,write,policy,test,password,sniff,sensitive source="# Set l\
ocal variables. Change the value in \"\" to reflect your environment.\r\
\n\r\
\n:local hostname [/system identity get name];\r\
\n:local password \"xxxxxx\"\r\
\n:local username \"xxxxxx\"\r\
\n:local ftpserver \"xxx.xxx.xxx.xxx\"\r\
\n\r\
\n# Set Filename variables. Do not change this unless you want to edit the\
\_format of the filename.\r\
\n\r\
\n:local time [/system clock get time];\r\
\n:local date ([:pick [/system clock get date] 0 3] \\\r\
\n. [:pick [/system clock get date] 4 6] \\\r\
\n. [:pick [/system clock get date] 7 11]);\r\
\n:local filename \"\$hostname-\$date-\$time\";\r\
\n\r\
\n# Create backup file and export the config.\r\
\n\r\
\nexport compact file=\"\$filename\"\r\
\n/system backup save name=\"\$filename\"\r\
\n\r\
\n:log info \"Backup Created Successfully\"\r\
\n\r\
\n# Upload config file to FTP server.\r\
\n\r\
\n/tool fetch address=\$ftpserver src-path=\"\$filename.rsc\" \\\r\
\nuser=\$username mode=ftp password=\$password \\\r\
\ndst-path=\"\$filename.rsc\" upload=yes\r\
\n\r\
\n# Upload backup file to FTP server.\r\
\n\r\
\n/tool fetch address=\$ftpserver src-path=\"\$filename.backup\" \\\r\
\nuser=\$username mode=ftp password=\$password \\\r\
\ndst-path=\"\$filename.backup\" upload=yes\r\
\n\r\
\n:log info \"Backup Uploaded Successfully\"\r\
\n\r\
\n# Delete created backup files once they have been uploaded\r\
\n# so they don't accumulate and fill up storage space on the router.\r\
\n\r\
\n/file remove \"\$filename.rsc\"\r\
\n/file remove \"\$filename.backup\"\r\
\n\r\
\n:log info \"Local Backup Files Deleted Successfully\""
/system watchdog
set automatic-supout=no watchdog-timer=no
/tool e-mail
set address=64.233.161.109 from= password=xxxxxx port=587 \
user=xxxxxx@gmail.com
UPD: this config is merged from the test lab, and not from the sell. Since the writing of this article in the prod, they managed to abandon this scheme in favor of L2VPN from providers with OSPF for routing.