NetApp Storage Clustering Using On-Off Switches
Clustering storage systems is gaining momentum, especially in light of the rapidly developing Flash technologies that require more controllers capable of handling high-performance drives. In addition to the supported cluster switches, there are many others that can be temporarily used for these purposes. In this article, I would like to give an example of setting up several switches that I have tested for a cluster network. I’m sure that many other switches will work as well, because this is a regular Ethernet
Switch connection scheme for Cluster Interconnect
When you have (at least temporarily) more than two FAS controllers on your hands , a cluster switch may be needed if:
Both for temporary cluster configurations consisting of more than two controllers, as well as for permanent ones, a supported cluster switch is required. At the moment, there are only two such switches: NetApp CN1610 and Cisco Nexus 5596. If you bought four or more controllers at once, then of course you have this switch. And if this is a temporary configuration from the examples above, most likely there is no such switch. And what to do? The answer is obvious: configure the switch that you now have at hand to create a temporary configuration.
Despite the fact that other switches are not officially supported, they certainly work:
The following switches were tested with MTU 1500 :
With the MTU 9000 Cisco Nexus 5548UP :
The following operational systems cDOT 8.3RC1, 8.3.1RC1 with storages were tested:
If necessary, reconfigure MTU for cluster ports NetApp:
In production, NetApp does not recommend using Cluster Interconnect switches for anything except for cluster connection of NetApp storages; for this it is necessary to have separate switches, although it is technically possible to use the same switches for both tasks.
Let me remind you that starting with cDOT 8.3, a new SLM feature works by default, and in the case of online migration, you must enable the nodes that accept the migrating moons to allow the hosts to be told to the hosts that the moons are now available in new, additional ways.
Make sure the host multipassing driver has detected these new paths.
If you migrate a volume, which is used for the NFS / CIFS (SMB) file protocols, after the migration, do not forget to move the LIF to the node where the volume moved.
More details on how to get nodes out of a cluster .
Learn more about how to configure zoning for clustered storage .
Please send comments on errors and suggestions for corrections in the text to the LAN .
Switch connection scheme for Cluster Interconnect
When you have (at least temporarily) more than two FAS controllers on your hands , a cluster switch may be needed if:
- Converting FAS 2240 / 255x to a shelf and then connecting to a new FAS system (which also has a shelf with disks), and online transfer is required without stopping access to data. Both systems must run cDOT
- Update 7M to cDOT followed by online data migration back . Learn more about the benefits of Cluster-Mode .
- Controller firmware upgrade (although it can be performed without stopping, using only two controllers, but if you have a couple more controllers with disks, why not use them)
- If you need to perform a more rational re-partitioning of disks (for example, to get the ADP feature : Root-Data Partitioning or Storage Pool )
- And etc.
Both for temporary cluster configurations consisting of more than two controllers, as well as for permanent ones, a supported cluster switch is required. At the moment, there are only two such switches: NetApp CN1610 and Cisco Nexus 5596. If you bought four or more controllers at once, then of course you have this switch. And if this is a temporary configuration from the examples above, most likely there is no such switch. And what to do? The answer is obvious: configure the switch that you now have at hand to create a temporary configuration.
Despite the fact that other switches are not officially supported, they certainly work:
The following switches were tested with MTU 1500 :
- HP Pro Curve 6120XG (Image stamp Z.14.44, 1564) from HP c7000 with 10Gb links to 2240 and 8020
- Cisco WS-C3850-48T (IOS 03.03.03SE RELEASE SOFTWARE (fc2)) with 1Gb links to 2240 and 8020
Settings HP Pro Curve 6120XG from HP c7000
# HP Pro Curve 6120XG from HP c7000
swbl2# sh ver
Image stamp: /ws/swbuildm/Z_zinfandel_fip_t4b_qaoff/code/build/vern(Z_zinfandel_fip_t4b_qaoff)
Aug 26 2013 16:32:58
Z.14.44
1564
Boot Image: Primary
swbl2# sh vlan
Status and Counters - VLAN Information
Maximum VLANs to support : 256
Primary VLAN : TECH
Management VLAN :
VLAN ID Name | Status Voice Jumbo
------- -------------------- + ---------- ----- -----
1 DEFAULT_VLAN | Port-based No No
210 NetApp-Cluster | Port-based No No
vlan 210
name "NetApp-Cluster"
untagged 19-20,22
tagged Trk1-Trk2
# Trk1-Trk2 - connection of both HP switches to each other and to Cisco WS-C3850-48T
no ip address
exit
wbl1# sh vlans 210
Status and Counters - VLAN Information - VLAN 210
VLAN ID : 210
Name : NetApp-Cluster
Status : Port-based
Voice : No
Jumbo : Yes
Port Information Mode Unknown VLAN Status
---------------- -------- ------------ ----------
22 Untagged Learn Up
swbl2# sh cdp neighbors 22 detail
CDP neighbors information for port 22
Port : 22
Device ID : clA-02
Address Type : IP
Address : 169.254.166.38
Platform : NetApp Release 8.3RC1: Fri Oct 31 20:13:33 PDT 2014FAS8020
Capability :
Device Port : e0b
Version : NetApp Release 8.3RC1: Fri Oct 31 20:13:33 PDT 2014FAS8020
Configure Cisco WS-C3850
#Cisco WS-C3850-48T
interface Vlan210
description NetApp-Cluster
no ip address
interface GigabitEthernet2/0/10
switchport access vlan 210
switchport mode access
flowcontrol receive on
spanning-tree portfast
!
core1-co2#sh cdp neighbors
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
D - Remote, C - CVTA, M - Two-port Mac Relay
Device ID Local Intrfce Holdtme Capability Platform Port ID
netapp2 Gig 1/0/10 149 H FAS2240-2 e0b
netapp2 Gig 1/0/9 149 H FAS2240-2 e0a
netapp2 Gig 2/0/9 149 H FAS2240-2 e0d
netapp2 Gig 2/0/10 149 H FAS2240-2 e0c
netapp2 Gig 1/0/40 149 H FAS2240-2 e0M
netapp1 Gig 2/0/8 166 H FAS2240-2 e0d
netapp1 Gig 2/0/7 166 H FAS2240-2 e0c
netapp1 Gig 1/0/8 166 H FAS2240-2 e0a
netapp1 Gig 1/0/7 166 H FAS2240-2 e0b
netapp1 Gig 1/0/39 166 H FAS2240-2 e0M
clA-02 Gig 2/0/12 142 H FAS8020 e0f
clA-02 Gig 2/0/6 142 H FAS8020 e0e
clA-01 Gig 2/0/11 168 H FAS8020 e0f
clA-01 Gig 1/0/6 168 H FAS8020 e0e
core1-co2#sh vlan
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
1 default active Te1/1/3, Te1/1/4, Gi2/0/37, Gi2/0/38, Gi2/0/41, Gi2/0/48, Te2/1/3
Te2/1/4
210 NetApp-Cluster active Gi1/0/6, Gi1/0/7, Gi1/0/8, Gi1/0/9, Gi1/0/10, Gi2/0/6, Gi2/0/7
Gi2/0/8, Gi2/0/9, Gi2/0/10, Gi2/0/11, Gi2/0/12
VLAN Type SAID MTU Parent RingNo BridgeNo Stp BrdgMode Trans1 Trans2
---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------
210 enet 100210 1500 - - - - - 0 0
ore1-co2#sh ver
Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software
(CAT3K_CAA-UNIVERSALK9-M), Version 03.03.03SE RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Sun 27-Apr-14 18:33 by prod_rel_team
ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 1.2, RELEASE SOFTWARE (P)
Ipbase
License Type: Permanent
Next reload license Level: Ipbase
cisco WS-C3850-48T (MIPS) processor with 4194304K bytes of physical memory.
Processor board ID FOC1801U0WU
11 Virtual Ethernet interfaces
104 Gigabit Ethernet interfaces
8 Ten Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
250456K bytes of Crash Files at crashinfo:.
250456K bytes of Crash Files at crashinfo-2:.
1609272K bytes of Flash at flash:.
1609272K bytes of Flash at flash-2:.
0K bytes of Dummy USB Flash at usbflash0:.
0K bytes of Dummy USB Flash at usbflash0-2:.
0K bytes of at webui:.
Base Ethernet MAC Address : 50:1c:bf:6c:65:00
Motherboard Assembly Number : 73-14444-06
Motherboard Serial Number : FOC180112WY
Model Revision Number : M0
Motherboard Revision Number : A0
Model Number : WS-C3850-48T
System Serial Number : FOC1801U0WU
Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 56 WS-C3850-48T 03.03.03SE cat3k_caa-universalk9 INSTALL
2 56 WS-C3850-48T 03.03.03SE cat3k_caa-universalk9 INSTALL
With the MTU 9000 Cisco Nexus 5548UP :
Nexus 5548UP Software version 7.1 (1) N1 (1)
banner motd # Nexus 5548 NetApp Reference Configuration File (RCF) version 1.3-48p (2012-10-15)
#
feature lacp
cdp enable
cdp advertise v1
cdp timer 5
snmp-server community cshm1! ro
errdisable recovery interval 30
errdisable recovery cause pause-rate-limit
policy-map type network-qos cluster
class type network-qos class-default
mtu 9216
system qos
service-policy type network-qos cluster
spanning-tree port type edge default
port-channel load-balance ethernet source-dest-port
interface port-channel1
switchport mode trunk
spanning-tree port type network
interface Ethernet1/1
description Cluster Node 1
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/2
description Cluster Node 2
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/3
description Cluster Node 3
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/4
description Cluster Node 4
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/5
description Cluster Node 5
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/6
description Cluster Node 6
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/7
description Cluster Node 7
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/8
description Cluster Node 8
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/9
description Cluster Node 9
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/10
description Cluster Node 10
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/11
description Cluster Node 11
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/12
description Cluster Node 12
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/13
description Cluster Node 13
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/14
description Cluster Node 14
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/15
description Cluster Node 15
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/16
description Cluster Node 16
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/17
description Cluster Node 17
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/18
description Cluster Node 18
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/19
description Cluster Node 19
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/20
description Cluster Node 20
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/21
description Cluster Node 21
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/22
description Cluster Node 22
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/23
description Cluster Node 23
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/24
description Cluster Node 24
no lldp transmit
no lldp receive
spanning-tree port type edge
spanning-tree bpduguard enable
interface Ethernet1/25
description Inter-Cluster Switch ISL Port 25 (port channel)
no lldp transmit
no lldp receive
switchport mode trunk
channel-group 1 mode active
interface Ethernet1/26
description Inter-Cluster Switch ISL Port 26 (port channel)
no lldp transmit
no lldp receive
switchport mode trunk
channel-group 1 mode active
interface Ethernet1/27
description Inter-Cluster Switch ISL Port 27 (port channel)
no lldp transmit
no lldp receive
switchport mode trunk
channel-group 1 mode active
interface Ethernet1/28
description Inter-Cluster Switch ISL Port 28 (port channel)
no lldp transmit
no lldp receive
switchport mode trunk
channel-group 1 mode active
interface Ethernet1/29
description Inter-Cluster Switch ISL Port 29 (port channel)
no lldp transmit
no lldp receive
switchport mode trunk
channel-group 1 mode active
interface Ethernet1/30
description Inter-Cluster Switch ISL Port 30 (port channel)
no lldp transmit
no lldp receive
switchport mode trunk
channel-group 1 mode active
interface Ethernet1/31
description Inter-Cluster Switch ISL Port 31 (port channel)
no lldp transmit
no lldp receive
switchport mode trunk
channel-group 1 mode active
interface Ethernet1/32
description Inter-Cluster Switch ISL Port 32 (port channel)
no lldp transmit
no lldp receive
switchport mode trunk
channel-group 1 mode active
end
The following operational systems cDOT 8.3RC1, 8.3.1RC1 with storages were tested:
- FAS 2240
- FAS 8020
If necessary, reconfigure MTU for cluster ports NetApp:
network port broadcast-domain modify -broadcast-domain Cluster -mtu 1500-or-9000 -ipspace Cluster
If necessary, convert SwitchLess-Cluster to Switched:
cluster1::*> network options switchless-cluster modify -enabled true
If necessary, migrate cluster LIFs
Если у ноды только один кластерный LIF, создайте второй и мигрируйте их по-очереди, так чтобы все ноды видели друг-друга хотябы по одному пути: через swichless коммутацию или через свичи.
Не мигрируйте все LIF'ы одной ноды сразу, отвалившаяся нода с включённым HA сразу ребутнётся!
HA конечно же сработает, но зачем, если можно без этого обойтись?
Не мигрируйте все LIF'ы одной ноды сразу, отвалившаяся нода с включённым HA сразу ребутнётся!
HA конечно же сработает, но зачем, если можно без этого обойтись?
cluster1::> network interface show -role cluster
cluster1::> network interface modify -vserver vsm0 -lif clus1_1 -home-node node01 -home-port e0c -auto-revert false
cluster1::> network interface modify -vserver vsm0 -lif clus2_1 -home-node node02 -home-port e0c -auto-revert false
#Мигрируем LIF
#Выполняем из Cluster Node Management консоли, той ноды на которой расположен LIF.
cluster1-01::> network interface revert -vserver vsm0 -lif clus1_1
cluster1-02::> network interface revert -vserver vsm0 -lif clus2_1
Be sure to test a clustered network from advanced mode:
cluster1::*> cluster ping-cluster -node local
Host is node1
Getting addresses from network interface table...
Local = 10.254.231.102 10.254.91.42
Remote = 10.254.42.25 10.254.16.228
Ping status:
....
Basic connectivity succeeds on 4 path(s)
Basic connectivity fails on 0 path(s)
................
Detected 1500 byte MTU on 4 path(s):
Local 10.254.231.102 to Remote 10.254.16.228
Local 10.254.231.102 to Remote 10.254.42.25
Local 10.254.91.42 to Remote 10.254.16.228
Local 10.254.91.42 to Remote 10.254.42.25
Larger than PMTU communication succeeds on 4 path(s)
RPC status:
2 paths up, 0 paths down (tcp check)
2 paths up, 0 paths down (udp check)
In production, NetApp does not recommend using Cluster Interconnect switches for anything except for cluster connection of NetApp storages; for this it is necessary to have separate switches, although it is technically possible to use the same switches for both tasks.
Block Access:
Let me remind you that starting with cDOT 8.3, a new SLM feature works by default, and in the case of online migration, you must enable the nodes that accept the migrating moons to allow the hosts to be told to the hosts that the moons are now available in new, additional ways.
cluster1::> lun mapping add-reporting-nodes -vserver vserver_name -path lun_path -igroup igroup_name [-destination-aggregate aggregate_name | -destination-volume volume_name]Make sure the host multipassing driver has detected these new paths.
NAS:
If you migrate a volume, which is used for the NFS / CIFS (SMB) file protocols, after the migration, do not forget to move the LIF to the node where the volume moved.
cluster1::> network interface migrate -vserver vs0 -lif datalif1 -source-node vs0 -dest-node node2 -dest-port e0c
More details on how to get nodes out of a cluster .
Learn more about how to configure zoning for clustered storage .
Please send comments on errors and suggestions for corrections in the text to the LAN .