NetApp Storage Clustering Using On-Off Switches

Published on September 15, 2015

NetApp Storage Clustering Using On-Off Switches

    Clustering storage systems is gaining momentum, especially in light of the rapidly developing Flash technologies that require more controllers capable of handling high-performance drives. In addition to the supported cluster switches, there are many others that can be temporarily used for these purposes. In this article, I would like to give an example of setting up several switches that I have tested for a cluster network. I’m sure that many other switches will work as well, because this is a regular Ethernet


    Switch connection scheme for Cluster Interconnect

    When you have (at least temporarily) more than two FAS controllers on your hands , a cluster switch may be needed if:

    • Converting FAS 2240 / 255x to a shelf and then connecting to a new FAS system (which also has a shelf with disks), and online transfer is required without stopping access to data. Both systems must run cDOT
    • Update 7M to cDOT followed by online data migration back . Learn more about the benefits of Cluster-Mode .
    • Controller firmware upgrade (although it can be performed without stopping, using only two controllers, but if you have a couple more controllers with disks, why not use them)
    • If you need to perform a more rational re-partitioning of disks (for example, to get the ADP feature : Root-Data Partitioning or Storage Pool )
    • And etc.


    Both for temporary cluster configurations consisting of more than two controllers, as well as for permanent ones, a supported cluster switch is required. At the moment, there are only two such switches: NetApp CN1610 and Cisco Nexus 5596. If you bought four or more controllers at once, then of course you have this switch. And if this is a temporary configuration from the examples above, most likely there is no such switch. And what to do? The answer is obvious: configure the switch that you now have at hand to create a temporary configuration.

    Despite the fact that other switches are not officially supported, they certainly work:
    The following switches were tested with MTU 1500 :
    • HP Pro Curve 6120XG (Image stamp Z.14.44, 1564) from HP c7000 with 10Gb links to 2240 and 8020
    • Cisco WS-C3850-48T (IOS 03.03.03SE RELEASE SOFTWARE (fc2)) with 1Gb links to 2240 and 8020

    Settings HP Pro Curve 6120XG from HP c7000
    # HP Pro Curve 6120XG from HP c7000
    swbl2# sh ver
    Image stamp:    /ws/swbuildm/Z_zinfandel_fip_t4b_qaoff/code/build/vern(Z_zinfandel_fip_t4b_qaoff)
                    Aug 26 2013 16:32:58
                    Z.14.44
                    1564
    Boot Image:     Primary
    swbl2# sh vlan
     Status and Counters - VLAN Information
      Maximum VLANs to support : 256
      Primary VLAN : TECH
      Management VLAN :
      VLAN ID Name                 | Status     Voice Jumbo
      ------- -------------------- + ---------- ----- -----
      1       DEFAULT_VLAN         | Port-based No    No
      210     NetApp-Cluster       | Port-based No    No
    vlan 210
       name "NetApp-Cluster"
       untagged 19-20,22
       tagged Trk1-Trk2
    # Trk1-Trk2 - connection of both HP switches to each other and to Cisco WS-C3850-48T
       no ip address
       exit
    wbl1# sh vlans 210
     Status and Counters - VLAN Information - VLAN 210
      VLAN ID : 210
      Name : NetApp-Cluster
      Status : Port-based
      Voice : No
      Jumbo : Yes
      Port Information Mode     Unknown VLAN Status
      ---------------- -------- ------------ ----------
      22               Untagged Learn        Up
    swbl2# sh cdp neighbors 22 detail
     CDP neighbors information for port 22
      Port : 22
      Device ID : clA-02
      Address Type : IP
      Address      : 169.254.166.38
      Platform     : NetApp Release 8.3RC1: Fri Oct 31 20:13:33 PDT 2014FAS8020
      Capability   :
      Device Port  : e0b
      Version      : NetApp Release 8.3RC1: Fri Oct 31 20:13:33 PDT 2014FAS8020
    


    Configure Cisco WS-C3850
    #Cisco WS-C3850-48T
    interface Vlan210
     description NetApp-Cluster
     no ip address
    interface GigabitEthernet2/0/10
     switchport access vlan 210
     switchport mode access
     flowcontrol receive on
     spanning-tree portfast
    !
    core1-co2#sh cdp neighbors
    Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                      S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
                      D - Remote, C - CVTA, M - Two-port Mac Relay
    Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
    netapp2          Gig 1/0/10        149               H    FAS2240-2 e0b
    netapp2          Gig 1/0/9         149               H    FAS2240-2 e0a
    netapp2          Gig 2/0/9         149               H    FAS2240-2 e0d
    netapp2          Gig 2/0/10        149               H    FAS2240-2 e0c
    netapp2          Gig 1/0/40        149               H    FAS2240-2 e0M
    netapp1          Gig 2/0/8         166               H    FAS2240-2 e0d
    netapp1          Gig 2/0/7         166               H    FAS2240-2 e0c
    netapp1          Gig 1/0/8         166               H    FAS2240-2 e0a
    netapp1          Gig 1/0/7         166               H    FAS2240-2 e0b
    netapp1          Gig 1/0/39        166               H    FAS2240-2 e0M
    clA-02           Gig 2/0/12        142               H    FAS8020   e0f
    clA-02           Gig 2/0/6         142               H    FAS8020   e0e
    clA-01           Gig 2/0/11        168               H    FAS8020   e0f
    clA-01           Gig 1/0/6         168               H    FAS8020   e0e
    core1-co2#sh vlan
    VLAN Name                             Status    Ports
    ---- -------------------------------- --------- -------------------------------
    1    default                          active    Te1/1/3, Te1/1/4, Gi2/0/37, Gi2/0/38, Gi2/0/41, Gi2/0/48, Te2/1/3
                                                    Te2/1/4
    210  NetApp-Cluster                   active    Gi1/0/6, Gi1/0/7, Gi1/0/8, Gi1/0/9, Gi1/0/10, Gi2/0/6, Gi2/0/7
                                                    Gi2/0/8, Gi2/0/9, Gi2/0/10, Gi2/0/11, Gi2/0/12
    VLAN Type  SAID       MTU   Parent RingNo BridgeNo Stp  BrdgMode Trans1 Trans2
    ---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------
    210  enet  100210     1500  -      -      -        -    -        0      0
    ore1-co2#sh ver
    Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software
    (CAT3K_CAA-UNIVERSALK9-M), Version 03.03.03SE RELEASE SOFTWARE (fc2)
    Technical Support: http://www.cisco.com/techsupport
    Copyright (c) 1986-2014 by Cisco Systems, Inc.
    Compiled Sun 27-Apr-14 18:33 by prod_rel_team
    ROM: IOS-XE ROMMON
    BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 1.2, RELEASE SOFTWARE (P)
    Ipbase
    License Type: Permanent
    Next reload license Level: Ipbase
    cisco WS-C3850-48T (MIPS) processor with 4194304K bytes of physical memory.
    Processor board ID FOC1801U0WU
    11 Virtual Ethernet interfaces
    104 Gigabit Ethernet interfaces
    8 Ten Gigabit Ethernet interfaces
    2048K bytes of non-volatile configuration memory.
    4194304K bytes of physical memory.
    250456K bytes of Crash Files at crashinfo:.
    250456K bytes of Crash Files at crashinfo-2:.
    1609272K bytes of Flash at flash:.
    1609272K bytes of Flash at flash-2:.
    0K bytes of Dummy USB Flash at usbflash0:.
    0K bytes of Dummy USB Flash at usbflash0-2:.
    0K bytes of  at webui:.
    Base Ethernet MAC Address          : 50:1c:bf:6c:65:00
    Motherboard Assembly Number        : 73-14444-06
    Motherboard Serial Number          : FOC180112WY
    Model Revision Number              : M0
    Motherboard Revision Number        : A0
    Model Number                       : WS-C3850-48T
    System Serial Number               : FOC1801U0WU
    Switch Ports Model              SW Version        SW Image              Mode
    ------ ----- -----              ----------        ----------            ----
    *    1 56    WS-C3850-48T       03.03.03SE        cat3k_caa-universalk9 INSTALL
         2 56    WS-C3850-48T       03.03.03SE        cat3k_caa-universalk9 INSTALL 
    


    With the MTU 9000 Cisco Nexus 5548UP :
    Nexus 5548UP Software version 7.1 (1) N1 (1)
    banner motd # Nexus 5548 NetApp Reference Configuration File (RCF) version 1.3-48p (2012-10-15)
    #
    feature lacp
    cdp enable
    cdp advertise v1
    cdp timer 5
    snmp-server community cshm1! ro
    errdisable recovery interval 30
    errdisable recovery cause pause-rate-limit
    policy-map type network-qos cluster
      class type network-qos class-default
        mtu 9216
    system qos
      service-policy type network-qos cluster
    spanning-tree port type edge default
    port-channel load-balance ethernet source-dest-port
    interface port-channel1
      switchport mode trunk
      spanning-tree port type network
    interface Ethernet1/1
      description Cluster Node 1  
      no lldp transmit
      no lldp receive
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/2
      description Cluster Node 2  
      no lldp transmit
      no lldp receive
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/3
      description Cluster Node 3
      no lldp transmit
      no lldp receive  
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/4
      description Cluster Node 4
      no lldp transmit
      no lldp receive  
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/5
      description Cluster Node 5
      no lldp transmit
      no lldp receive  
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/6
      description Cluster Node 6
      no lldp transmit
      no lldp receive  
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/7
      description Cluster Node 7
      no lldp transmit
      no lldp receive  
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/8
      description Cluster Node 8
      no lldp transmit
      no lldp receive  
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/9
      description Cluster Node 9
      no lldp transmit
      no lldp receive  
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/10
      description Cluster Node 10
      no lldp transmit
      no lldp receive  
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/11
      description Cluster Node 11
      no lldp transmit
      no lldp receive  
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/12
      description Cluster Node 12
      no lldp transmit
      no lldp receive 
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/13
      description Cluster Node 13
      no lldp transmit
      no lldp receive
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/14
      description Cluster Node 14
      no lldp transmit
      no lldp receive
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/15
      description Cluster Node 15
      no lldp transmit
      no lldp receive
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/16
      description Cluster Node 16
      no lldp transmit
      no lldp receive
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/17
      description Cluster Node 17
      no lldp transmit
      no lldp receive
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/18
      description Cluster Node 18
      no lldp transmit
      no lldp receive
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/19
      description Cluster Node 19
      no lldp transmit
      no lldp receive
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/20
      description Cluster Node 20
      no lldp transmit
      no lldp receive
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/21
      description Cluster Node 21
      no lldp transmit
      no lldp receive
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/22
      description Cluster Node 22
      no lldp transmit
      no lldp receive
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/23
      description Cluster Node 23
      no lldp transmit
      no lldp receive
      spanning-tree port type edge
      spanning-tree bpduguard enable
     interface Ethernet1/24
      description Cluster Node 24
      no lldp transmit
      no lldp receive
      spanning-tree port type edge
      spanning-tree bpduguard enable
    interface Ethernet1/25
      description Inter-Cluster Switch ISL Port 25 (port channel)
      no lldp transmit
      no lldp receive
      switchport mode trunk
      channel-group 1 mode active
    interface Ethernet1/26
      description Inter-Cluster Switch ISL Port 26 (port channel)
      no lldp transmit
      no lldp receive
      switchport mode trunk
      channel-group 1 mode active
    interface Ethernet1/27
      description Inter-Cluster Switch ISL Port 27 (port channel)
      no lldp transmit
      no lldp receive
      switchport mode trunk
      channel-group 1 mode active
    interface Ethernet1/28
      description Inter-Cluster Switch ISL Port 28 (port channel)
      no lldp transmit
      no lldp receive
      switchport mode trunk
      channel-group 1 mode active
    interface Ethernet1/29
      description Inter-Cluster Switch ISL Port 29 (port channel)
      no lldp transmit
      no lldp receive
      switchport mode trunk
      channel-group 1 mode active
    interface Ethernet1/30
      description Inter-Cluster Switch ISL Port 30 (port channel)
      no lldp transmit
      no lldp receive
      switchport mode trunk
      channel-group 1 mode active
    interface Ethernet1/31
      description Inter-Cluster Switch ISL Port 31 (port channel)
      no lldp transmit
      no lldp receive
      switchport mode trunk
      channel-group 1 mode active
    interface Ethernet1/32
      description Inter-Cluster Switch ISL Port 32 (port channel)
      no lldp transmit
      no lldp receive
      switchport mode trunk
      channel-group 1 mode active
    end
    


    The following operational systems cDOT 8.3RC1, 8.3.1RC1 with storages were tested:
    • FAS 2240
    • FAS 8020

    If necessary, reconfigure MTU for cluster ports NetApp:
    network port broadcast-domain modify -broadcast-domain Cluster -mtu 1500-or-9000 -ipspace Cluster
    

    If necessary, convert SwitchLess-Cluster to Switched:
    cluster1::*> network options switchless-cluster modify -enabled true
    


    If necessary, migrate cluster LIFs
    Если у ноды только один кластерный LIF, создайте второй и мигрируйте их по-очереди, так чтобы все ноды видели друг-друга хотябы по одному пути: через swichless коммутацию или через свичи.
    Не мигрируйте все LIF'ы одной ноды сразу, отвалившаяся нода с включённым HA сразу ребутнётся!
    HA конечно же сработает, но зачем, если можно без этого обойтись?
    cluster1::> network interface show -role cluster
    cluster1::> network interface modify -vserver vsm0 -lif clus1_1 -home-node node01 -home-port e0c -auto-revert false
    cluster1::> network interface modify -vserver vsm0 -lif clus2_1 -home-node node02 -home-port e0c -auto-revert false
    #Мигрируем LIF
    #Выполняем из Cluster Node Management консоли, той ноды на которой расположен LIF.
    cluster1-01::> network interface revert -vserver vsm0 -lif clus1_1
    cluster1-02::> network interface revert -vserver vsm0 -lif clus2_1
    


    Be sure to test a clustered network from advanced mode:
    cluster1::*> cluster ping-cluster -node local
    Host is node1
    Getting addresses from network interface table...
    Local = 10.254.231.102  10.254.91.42
    Remote = 10.254.42.25    10.254.16.228
    Ping status:
    ....
    Basic connectivity succeeds on 4 path(s)
    Basic connectivity fails on 0 path(s)
    ................
    Detected 1500 byte MTU on 4 path(s):
        Local 10.254.231.102 to Remote 10.254.16.228
        Local 10.254.231.102 to Remote 10.254.42.25
        Local 10.254.91.42 to Remote 10.254.16.228
        Local 10.254.91.42 to Remote 10.254.42.25
    Larger than PMTU communication succeeds on 4 path(s)
    RPC status:
    2 paths up, 0 paths down (tcp check)
    2 paths up, 0 paths down (udp check)
    



    In production, NetApp does not recommend using Cluster Interconnect switches for anything except for cluster connection of NetApp storages; for this it is necessary to have separate switches, although it is technically possible to use the same switches for both tasks.

    Block Access:


    Let me remind you that starting with cDOT 8.3, a new SLM feature works by default, and in the case of online migration, you must enable the nodes that accept the migrating moons to allow the hosts to be told to the hosts that the moons are now available in new, additional ways.
    cluster1::> lun mapping add-reporting-nodes -vserver vserver_name -path lun_path -igroup igroup_name [-destination-aggregate aggregate_name | -destination-volume volume_name]

    Make sure the host multipassing driver has detected these new paths.

    NAS:


    If you migrate a volume, which is used for the NFS / CIFS (SMB) file protocols, after the migration, do not forget to move the LIF to the node where the volume moved.
    cluster1::> network interface migrate -vserver vs0 -lif datalif1 -source-node vs0 -dest-node node2 -dest-port e0c
    


    More details on how to get nodes out of a cluster .
    Learn more about how to configure zoning for clustered storage .

    Please send comments on errors and suggestions for corrections in the text to the LAN .