Distributed Cluster of Two Fortigate

Zwerg October 17, 2016 at 23:56

Distributed Cluster of Two Fortigate

Tutorial

Recently completed a project with a client to build a fault-tolerant network infrastructure. As long as the memories are alive - and the documentation is at hand, I want to share unique know-how. What is unique? The fact that such a configuration is not mentioned anywhere in official Fortinet documents is not recommended and generally does not officially exist. Also, no mention of such a configuration was found on the Internet.

So, what we have available:
Two platforms spaced in space, an office and a data center (DC). Each site has independent access to the Internet and local subnet. Between sites there is a dedicated connection with the ability to create your own VLANs. Of the equipment, there are two Fortigate 100Ds and four managed Cisco Catalyst 2960XR switches with a Stacking Module. Those. two switches and one fortigate per site.

Objective:
To ensure fault tolerance and continuity of communication
1) in case of failure of any one piece of iron.
2) in the event of a fall in any of the communication channels.
3) in the event of failure of any one port on any of the glands

If we had 2 fortigates per site - there would be no problem at all - we set up a standard cluster of two fortigates at each site, dynamic routing and enjoy life. However, with the equipment we have what we have, so we have to bother.

At the beginning of the equipment there was even less - one switch and one fortigate per site, but in this article I will describe the final configuration.

So

here we go : Denote SW1 and FG1 - the stack of switches in the office and fortigate in the office. SW2 and FG2 - similar pieces of iron in a data center.

For the correct configuration of everything, we need 5 logical ports on each fortigate (5 logical = 10 physical). Let's designate them as:

Office_Internet
DC_Internet
Office_lan
DC_Lan
Interconnect

We also use both HA ports to ensure cluster synchronization, which adds 2 more ports to those used on the switch. In total, 6 ports will be involved on each physical switch (or 12 ports on the stack of switches) only for connecting fortigates.

Layer1 configuration (physical connections): I
assume that the stack of switches has already been assembled, because there are no difficulties there.
We connect fortigates to switches according to the following scheme:

For DC_Internet

FG1-WAN1 - SW1-1/0/1
FG1-WAN2 - SW1-2/0/1

For Office_Internet

FG1-DMZ   - SW1-1/0/2
FG1-Port1  - SW1-2/0/2

For Office_Lan

FG1-DMZ   - SW1-1/0/3
FG1-Port1  - SW1-2/0/3

For DC_Lan

FG1-Port2   - SW1-1/0/4
FG1-Port3   - SW1-2/0/4

For interconnect

FG1-Port4   - SW1-1/0/5
FG1-Port5   - SW1-2/0/5

For HA1 and HA2

FG1-HA1   - SW1-1/0/6
FG1-HA2   - SW1-2/0/6

Repeat the operation in the same way for FG2 and SW2.

Now you need to stick the links coming from the provider into the switches. In my case, these are 2 Internet connections and a channel between the DC and the office, respectively, the configuration will slightly differ between the two sites. Again, in my particular case, the providers provided two links to the Internet and 2 links for interconnect on each side.

If you can’t get 2 links for each channel - then one of the requirements will remain partially unfulfilled - if the port exits on the switch where the link from the provider is connected, the connection will be lost.

Here we will connect downlinks to Access switches in the office and DC.

We connect in the office:

SW1-1/0/22 - Даунлинк в Office_LAN линк 1
SW1-2/0/22 - Даунлинк в Office_LAN линк 2
SW1-1/0/23 - Интернет от провайдера в офисе линк 1
SW1-2/0/23 - Интернет от провайдера в офисе линк 2
SW1-1/0/24 - Канал Офис-ДЦ линк 1
SW1-2/0/24 - Канал Офис-ДЦ линк 2

We connect in DC:

SW1-1/0/22 - Даунлинк в DC_LAN линк 1
SW1-2/0/22 - Даунлинк в DC_LAN линк 2
SW2-1/0/23 - Интернет от провайдера в ДЦ линк 1
SW2-2/0/23 - Интернет от провайдера в ДЦ линк 2
SW2-1/0/24 - Канал Офис-ДЦ линк 1
SW2-2/0/24 - Канал Офис-ДЦ линк 2

With Layer 1 done, you can move on to Port Aggregation and VLAN configurations.

Layer 2
At this stage, you need to configure pairs of ports on switches and fortigates, and assign a VLAN to distinguish between traffic.

- We create the necessary VLANs on the switches and assign ports to them:

on the switches in configuration mode, create the necessary VLANs

vlan 10
name DC_Internet
vlan 20 
name Office_Internet
vlan 30
name Office_Lan
vlan 40 
name DC_LAN
vlan 50
name Interconnect
vlan 60
name HA

On the switches, create a port-channel and assign ports to them:

For DC_Internet:

Interface port-channel1
  switchport access vlan 10    # VLAN 10 = DC_Internet
  switchport mode access
Interface GigabitEthernet1/0/1
 switchport access vlan 10
 switchport mode access
 channel-group 1 mode passive
Interface GigabitEthernet2/0/1
 switchport access vlan 10
 switchport mode access
 channel-group 1 mode passive

Repeat for the above pairs of ports and get port-channel 2-4

For Interconnect:

Interface port-channel5
  switchport access vlan 50
  switchport mode access
Interface GigabitEthernet1/0/5
 switchport access vlan 50
 switchport mode access
 channel-group 5 mode passive
Interface GigabitEthernet2/0/5
 switchport access vlan 50
 switchport mode access
 channel-group 5 mode passive

A slightly different configuration for HA1 and HA2 because there is no port aggregation
for HA1

Interface GigabitEthernet1/0/6  
 switchport access vlan 60         # Vlan 60 = HA
 switchport mode access

For HA2

Interface GigabitEthernet2/0/6
 switchport access vlan 60        # Vlan 60 = HA
 switchport mode access

For the Office-DC Channel:

Interface port-channel6
    switchport mode trunk      # !!! Trunk
Interface GigabitEthernet1/0/24
 switchport mode trunk
 channel-group 6 mode passive
Interface GigabitEthernet2/0/24
  switchport mode trunk
 channel-group 6 mode passive

The above configuration for Layer2 is the same for the office and DC, the different parts of the config are shown below.

In the office:

Interface port-channel7
  switchport access vlan 30           # VLAN 30 = Office_Lan
  switchport mode access
Interface GigabitEthernet1/0/22
 switchport access vlan 30
 switchport mode access
 channel-group 7 mode passive
Interface GigabitEthernet2/0/22
 switchport access vlan 30
 switchport mode access
 channel-group 7 mode passive
Interface port-channel8
  switchport access vlan 20          # VLAN 20 = Office_Internet 
  switchport mode access
Interface GigabitEthernet1/0/23
 switchport access vlan 20
 switchport mode access
 channel-group 8 mode passive
Interface GigabitEthernet2/0/23
 switchport access vlan 20
 switchport mode access
 channel-group 8 mode passive

In DC:

Interface port-channel7
  switchport access vlan 40           # VLAN 40 = DC_Lan
  switchport mode access
Interface GigabitEthernet1/0/22
 switchport access vlan 40
 switchport mode access
 channel-group 7 mode passive
Interface GigabitEthernet2/0/22
 switchport access vlan 40
 switchport mode access
 channel-group 7 mode passive
Interface port-channel8
  switchport access vlan 10           #  VLAN 10 = DC_Internet
  switchport mode access
Interface GigabitEthernet1/0/23
 switchport access vlan 10
 switchport mode access
 channel-group 8 mode passive
Interface GigabitEthernet2/0/23
 switchport access vlan 10
 switchport mode access
 channel-group 8 mode passive

From the side of fortigates:

1. Turning on VDOM support
2. Configuring Virtual Cluster support
3. Scattering ports - each in its own VDOM
4. Creating pairs of Port Aggregation 802.3ad ports

Here it looks like code:

A lot of code

config system interface
    edit "wan1"
        set vdom "DataCenter"
        set type physical
        set snmp-index 1
    next
    edit "dmz"
        set vdom "Office"
        set type physical
        set snmp-index 2
    next
    edit "wan2"
        set vdom "DataCenter"
        set type physical
        set snmp-index 7
    next
    edit "ha1"
        set vdom "root"
        set type physical
        set snmp-index 9
    next
    edit "ha2"
        set vdom "root"
        set status down
        set type physical
        set snmp-index 10
    next
    edit "port1"
        set vdom "Office"
        set type physical
        set snmp-index 12
    next
    edit "port2"
        set vdom "DataCenter"
        set type physical
        set snmp-index 13
    next
    edit "port3"
        set vdom "Office"
        set type physical
        set snmp-index 17
    next
    edit "port4"
        set vdom "DataCenter"
        set type physical
        set snmp-index 18
    next
    edit "port5"
        set vdom "Office"
        set type physical
        set snmp-index 15
    next
    edit "port6"
        set vdom "Office"
        set type physical
        set snmp-index 16
    next
    edit "port7"
        set vdom "DataCenter"
        set type physical
        set snmp-index 19
    next
    edit "port8"
        set vdom "Office"
        set status down
        set type physical
        set snmp-index 20
    next
    edit "port9"
        set vdom "DataCenter"
        set status down
        set type physical
        set snmp-index 21
    next
    edit "port10"
        set vdom "root"
        set status down
        set type physical
        set snmp-index 22
    next
    edit "lan"
        set vdom "root"
        set status down
        set type hard-switch
        set snmp-index 11
    next
    edit "DC_Internet"
        set vdom "DataCenter"
        set ip 2.2.2.2 255.255.255.224
        set allowaccess ping https ssh
        set type aggregate
        set member "wan1" "wan2"
        set snmp-index 14
    next
    edit "Office_Internet"
        set vdom "Office"
        set ip 1.1.1.1 255.255.255.248
        set allowaccess ping https
        set type aggregate
        set member "port5" "dmz"
        set snmp-index 23
    next
    edit "Office-LAN"
        set vdom "Office"
        set dhcp-relay-service enable
        set ip 192.168.100.1 255.255.255.0
        set allowaccess ping https ssh
        set type aggregate
        set member "port6" "port1"
        set snmp-index 24
     next
    edit "DC-LAN"
        set vdom "DataCenter"
        set ip 192.168.200.1 255.255.255.0
        set allowaccess ping https ssh
        set type aggregate
        set member "port7" "port2"
        set snmp-index 25
    next
    edit "Intercon-Office"
        set vdom "Office"
        set ip 192.168.150.1 255.255.255.252
        set allowaccess ping
        set type aggregate
        set member "port8" "port3"
        set snmp-index 26
    next
    edit "Intercon-DC"
        set vdom "DataCenter"
        set ip 192.168.150.2 255.255.255.252
        set allowaccess ping
        set type aggregate
        set member "port9" "port4"
        set snmp-index 27
    next
end
config system ha
    set group-name "aaa.com"
    set mode a-p
    set password ENC xxx
    set hbdev "ha1" 50
    set arps 30
    set arps-interval 1
    set session-pickup enable
    set link-failed-signal enable
    set ha-mgmt-status enable
    set ha-mgmt-interface "mgmt"
    set vcluster2 enable
    set override enable
    set priority 100
    set monitor "Office_internet" "Office_LAN" "Intercon_Office"
        config secondary-vcluster
            set override enable
            set priority 250
            set monitor "DC_Internet" "DC_LAN" "Intercon_DC"
            set vdom "DataCenter"
        end
end

This is all for Layer 2, go to Layer 3+ . Here we need to:

- enable routing
- configure routing monitoring
- configure firewall rules
- configure a VPN in case the channel falls between the office and the DC

There is no need to do anything on the switches. On fortigite (after setting up the cluster - we control one piece of iron):

Routing and Routing Monitoring

config vdom
edit DataCenter
config router static
    edit 1
        set gateway 2.2.2.3
        set device "DC_Internet"
    next
    edit 2
        set dst 192.168.100.0 255.255.255.0
        set gateway 192.168.150.1
        set device "Intercon-DC"
    next
    edit 3
        set gateway 192.168.150.1
        set distance 15
        set device "Intercon-DC"
    next
end
config system link-monitor
    edit "DC-Internet"
        set srcintf "DC_Internet"
        set server "2.2.2.3"
        set gateway-ip 2.2.2.3
        set update-cascade-interface disable
    next
    edit "Interconnect"
        set srcintf "Intercon-DC"
        set server "192.168.150.1"
        set gateway-ip 192.168.150.1
        set update-cascade-interface disable
    next
end
end 
end
config vdom
edit Office
config router static
    edit 1
        set gateway 1.1.1.2
        set distance 9
        set device "Office_Internet"
    next
    edit 2
        set dst 192.168.200.0 255.255.255.0
        set gateway 192.168.150.2
        set device "Intercon-Office"
    next
    edit 3
        set gateway 192.168.150.2
        set distance 15
        set device "Intercon-Office"
    next
end
config system link-monitor
    edit "Office-Internet-Check"
        set srcintf "Office_Internet"
        set server "1.1.1.2"
        set gateway-ip 1.1.1.2
        set update-cascade-interface disable
    next
    edit "Interconnect"
        set srcintf "Intercon-Office"
        set server "192.168.150.2"
        set gateway-ip 192.168.150.2
        set update-cascade-interface disable
    next
end

Now, in the event of a fall of any Internet channel, the static route to the Internet will be deleted and the march will be activated through the officer-dts channel, in case of restoration of the channel, everything will return as it was.

It remains to create firewall rules (I hope to master this yourself). You just need to remember that the same traffic can come from different interfaces (Internet or interconnect).

And also create a VPN connection to connect sites in the event of a drop in the DC - office channel.

VPN configuration

config vdom
edit DataCenter
config vpn ipsec phase1
    edit "site-to-site"
        set interface "DC_Internet"
        set nattraversal disable
        set proposal aes256-sha256
        set dhgrp 5
        set remote-gw 1.1.1.1
        set psksecret ENC xxx
    next
end
config vpn ipsec phase2
    edit "p2"
        set phase1name "site-to-site"
        set proposal aes256-sha256
        set pfs disable
        set replay disable
        set auto-negotiate enable
        set src-addr-type name
        set dst-addr-type name
        set src-name "192.168.200.x"
        set dst-name "192.168.100.x"
    next
end
config vdom
edit Office
config vpn ipsec phase1
    edit "site-to-site"
        set interface "Office_Internet"
        set nattraversal disable
        set proposal aes256-sha256
        set dhgrp 5
        set remote-gw 2.2.2.2
        set psksecret ENC xxx
    next
end
config vpn ipsec phase2
    edit "p2"
        set phase1name "site-to-site"
        set proposal aes256-sha256
        set pfs disable
        set replay disable
        set keepalive enable
        set auto-negotiate enable
        set src-addr-type name
        set dst-addr-type name
        set src-name "192.168.100.x"
        set dst-name "192.168.200.x"
    next
end

And the corresponding firewall rules:

Firewall rules

config vdom
edit Office
config firewall policy
    edit 20
        set srcintf "Office-LAN"
        set dstintf "Office_Internet"
        set srcaddr "192.168.100.x"
        set dstaddr "192.168.200.x"
        set action ipsec
        set schedule "always"
        set service "ALL"
        set inbound enable
        set outbound enable
        set vpntunnel "site-to-site"
    next
end
config vdom
edit DataCenter
    edit 9
        set srcintf "DC-LAN"
        set dstintf "DC_Internet"
        set srcaddr "192.168.200.x"
        set dstaddr "192.168.100.x"
        set action ipsec
        set schedule "always"
        set service "ALL"
        set inbound enable
        set outbound enable
        set vpntunnel "site-to-site"
    next

As a result, the VPN will only rise if the office-dts channel falls.

What we got as a result:

1. Port aggregation protects us from the fall of either one port or the switch as a whole (with the exception of channels with one wire from the provider).

2. The virtual fortigate cluster will act as a router for the network, and in a normal situation, the fortigate in the office will always forward packets for its network zone, and the fortigate in the DC for its own. If one of the fortigates falls, all traffic will be routed by the second fortigate.

3. In the event of a fall of one of the Internet channels, all traffic will go through another channel, and will return as soon as the channel to the Internet rises.

4.If the office-dts channel falls, the traffic will go through an automatically rising VPN and will go through it until the channel recovers.

It turned out a lot and probably at first glance it is difficult - but if you have questions, I am ready to answer them.

Tags: