Networking for storage systems of SMB sector Part 2. Practical issues of organizing small infrastructures

    Preface to Part 2

    The first part dealt with the basic aspects of building data storage systems. Now we will talk about the practical component, in particular, the use of the selected equipment, exchange protocols and the topology of the internal network.

    Network Protocols and Required Features

    Before you start building a data storage system, you need to decide which one you have to use.

    Today, there are many different protocols that can be applied one way or another during the exchange between the server and the storage system. However, it is best to choose the most universal, and, at the same time, the most undemanding of them.

    All storage systems and protocols for data exchange can be conditionally divided by type of access into: file and block.

    File-access storages are network resources with which data is exchanged via popular application-level protocols. Most commonly used: CIFS, NFS, AFP. Other file transfer protocols are used, such as FTP, TFTP, SFTP, and so on. Such devices are called Network Attached Storage, for short - NAS. In fact, NAS is a file server, the hardware and software of which is optimized for the tasks of storing and exchanging files.

    Block-access storages provide for use specially organized volumes with unique numbers - Logical Unit Number, LUN. At the same time, connected storage resources from the point of view of logical organization look like local hard drives. In this case, special access protocols are used, such as FCP (for Fiber Channel), and iSCSI, FCoE, ATAoE and others - for Ethernet networks. Storage systems of this type are called Storage Area Network, abbreviated as “SAN” and are disk arrays connected to a high-performance data network. ISCSI protocol is very popular among budget storage systems due to the simplicity of implementation and unpretentiousness to resources.

    IMPORTANT NOTICE. In the framework of this article, it is impossible to talk about all aspects of the creation and operation of storage systems. Therefore, for additional information, you should contact other sources. Some of them are indicated at the end of the article.

    Block access storage systems (like a local hard drive) during normal use do not allow more than one connection without the risk of data loss, while file storages may be available for connecting to multiple clients.

    NOTE . Under certain conditions, block storage can be connected to several clients, for example, when using file systems of a shared disk. In particular, VMware ESX (i) host servers can simultaneously connect and work with block access storage in the presence of the VMFS file system.

    The restriction requires certain access control measures. In Fiber Channel networks, zoning is used for this, for the Ethernet family, a password authorization mechanism and separation into virtual subnets (VLANs) are used.

    NOTE . Virtual Local Area Network (VLAN) - an artificially limited segment of the local network in the form of a group of ports with a single set of properties, for example, with the same security requirements. Allows you to exchange information with each other within a single virtual broadcast domain, regardless of the physical location of the connected devices.

    Another important parameter that block access dictates to us is the reservation of the communication channel. If in the event of a line break when copying to a network resource you just have to overwrite the file, then with block access this is fraught with the appearance of serious errors in the file system. The combination of several data lines in the Ethernet family of networks into one logical channel will come in handy here. This feature is called Link Aggregation, or LAGG for short. In addition to increasing fault tolerance, link aggregation can increase bandwidth.

    In operating UNIX-like systems: Linux, BSD, Solaris, which are taken as the basis for building various storage systems, the Link Aggregation Control Protocol is most often used. This open standard protocol, described in the IEEE 802.3ad and IEEE 802.1aq documents, is commonly used for link aggregation in simple systems.

    To improve system performance, it is critical to reduce the cost of traffic maintenance. One way is to use larger data packets. Accordingly, if fewer packets are required to transmit the same information, this will reduce processing time and save hardware resources. Therefore, the Jumbo Frame function will also be out of place.

    NOTE. Jumbo frame is a mechanism in the Ethernet family of networks, thanks to which it is possible to transmit data blocks exceeding 1500 bytes (the value specified by the IEEE 802.3 group standards) - or, as they say, increase the MTU by more than 1500 bytes). The most popular practice in storage systems based on IP networks is to set the frame size to 9000 bytes of data, for example, to speed up the exchange via iSCSI, NFS, GIFS (SMB) protocols.

    All of the above features: VLAN, LAGG (LACP) and Jumbo Frames - are available in our Zyxel XS1920 Series switch.

    Storage Topology

    The construction of small storage systems for medium and small businesses, on the one hand, imposes certain restrictions in terms of cost, on the other hand, it allows some concessions in the field of fault tolerance and security.
    First, consider the simplest diagram in Figure 1.

    In this case, there is a virtual system of 4 servers connected to the same storage through a dedicated switch. To increase throughput, all connections are duplicated through the port aggregation function. Thus, 10 RJ45 ports are used on 10 Gigabit Ethernet. The remaining 2 universal 10 Gigabit Ethernet ports are reserved and can later be used to connect another storage system or server.

    Figure 1. Simple storage network.

    Please note that the switch in the above diagram is used completely offline. There is no connection with other segments of the network (Uplink / Downlink) and, apart from servers and the storage system, nothing else is connected to it. Such isolation is justified both from the point of view of fault tolerance, and for optimization and simplification of network exchange.

    When building this simple network, it should be noted that most storage systems and servers have a special port for management, for example, via IPMI. In addition, to improve control over home-made storage systems, they often use an additional network card, which displays the control interface via HTTP / HTTPS or Telnet / SSH.

    NOTE. IPMI - Intelligent Platform Management Interface, - a dedicated interface for monitoring and control. Most notable implementations: HP iLO and IMM2, developed by IBM. IPMI is available even when the host device is inactive. Through IPMI, you can turn off, turn on, restart the system, intercept the console for control, take sensors, for example, temperature, power, and so on.

    As a low-cost solution for organizing such a network, any switch with Fast Ethernet (100Mb / s) or Gigabit Ethernet interface is suitable. In order to minimize the costs of its acquisition, as well as management, staff training, and so on, it makes sense to choose equipment from the same manufacturer as the main storage network. For example, in our case, the Zyxel GS1100-8HP model - a switch for 8 gigabit ports, will do well.

    Figure 2. An example of the organization of a dedicated management network.

    If the requirements for fault tolerance are high, you must use a circuit with two switches. In order to avoid conflict when accessing the same iSCSI volumes on different connections, the multipathing mechanism is used. An additional bonus in this case will be the ability to balance and increase the number of ports that can be used both for connecting devices and for expanding the channel.

    NOTE. Multipathing - technology for connecting storage nodes using several independent lines. For example, a single storage device can be connected to a server using multiple iSCSI channels. In the event of a failure of one of the connections, the operating system will use the remaining workable communication lines to access the device. This architecture increases the fault tolerance of the system and allows you to distribute the load, which leads to an increase in overall performance.

    Figure 3. Diagram with two switches. (To simplify the perception, the control network is not shown).

    The Zyxel XS1920 Series is well-suited for small data storage. For larger infrastructures, we can recommend another model - the 24-port 10GbE L2 + XS3700 Series switch.


    Building an inexpensive storage system that meets business requirements is not an easy task, but it is feasible. Of course, in a short narrative, it is not possible to cover all aspects of this vast area of ​​IT. For example, such issues as duplication of storage systems for fault tolerance, the creation of geographically distributed storage systems and so on remained behind the scenes. However, as a first step, the experience of building simple solutions for centralized data storage will certainly be useful and will allow you to further organize an effective infrastructure for various purposes.


    1. Berezhnoy A. The use of iSCSI in the construction of data storage systems // "System Administrator", No. 3, 2017
    2. Vladan Seget How to configure ESXi 6.5 for iSCSI Shared Storage
    3. LACP (description) -
    4. Ethernet Alliance. Ethernet Jumbo Frames
    5. Zyxel XS1920 Series Switch Description
    6. Intelligent Platform Management Interface (IPMI)
    7. Description of Unmanaged 8/10/16/24 GbE Switches - Zyxel GS1100 Series
    8. Multipath (description)
    9. Zyxel XS3700 Series Switch Description

    Also popular now: