May 25, 2017 at 16:34

Key storage protocols: usage and prospects



    In this article, we will give a short overview of the most popular protocols that are used to build storage area networks (SANs). We also consider the prospects for the development and use of individual protocols, based on publicly accessible roadmaps of manufacturers.

    It is no secret that today performance problems are moving from the storage sector to the SAN area, since storage systems have already achieved huge performance indicators in GB / s and millions of IOps, and current SAN networks do not allow such data volumes to be pumped through itself.

    General trends


    Existing FC 8Gb / s and Ethernet 10Gb / s networks can no longer cope with modern storage systems, even Ethernet 25/50 Gb / s cannot provide acceptable delays when working with the latest storage models using NVMe drives.

    Many IT professionals involved in setting up and administering the storage infrastructure are wondering about upgrading SAN networks. Why is this so important and necessary today? We formulate several main reasons:

    • Firstly, the data storage density is constantly growing, the volumes of both HDD disks and flash drives are increasing. Already on the market are available 10TB HDD and SSD up to 3.2TB.
    • Secondly, the speed of flash and hybrid storage grows in two aspects: increased throughput and reduced latency. For example, the latest models of NVMe-drives have a performance of about 3 GB / s and a delay of 20 μs.
    • Third, the need for processing millions of IOps. Already, one NVMe drive is capable of delivering up to 700+ thousand IOps.
    • Fourth, the need to increase performance by one connection port. Previously, a maximum of 56 Gb / s was received from one QSFP + port on the InfiniBand protocol, but already this year there are solutions with a capacity of 200 Gb / s per port.

    With all these factors in mind, the key characteristics of access protocols are high throughput and fast response time.

    According to existing forecasts, over the next two years (according to Gartner, within 18 months) all-flash arrays will be equipped with more efficient SSD disks, and in combination with fast interconnect (for example, NVMe) and new protocols (for example, iWARP) all-flash arrays will increase by another two orders of magnitude, which will lead to the need to upgrade the SAN.

    Protocol Comparison


    All SAN networks, whether Ethernet, PCI Express, SAS, NVMeoF, FC, FCoE or InfiniBand, must support the same functionality, namely:

    • identify host and storage
    • be able to route traffic;
    • split the network into subnets and isolate traffic in them;
    • provide the ability to use multiple paths to storage;
    • manage the connection of devices to the network;
    • prioritize traffic.

    Given the similar functionality being implemented, we can say that the differences between the protocols are reduced to usability, performance parameters and implementation costs. So, some protocols have more advanced management and monitoring tools, which makes them easier to implement, administer and operate. Other protocols have better performance, but seem more complex in terms of integration into current SAN networks.

    Today, data storage protocols can be divided into two conditional groups:

    1. Used for connecting application servers (FC, FCoE, iSCSI, NFS, SMB);
    2. used to connect within a cluster or as an interconnect within storage systems (InfiniBand, NVMe, PCIe).




    Fiber Channel (FC)


    Fiber Channel is a popular storage protocol that provides low latency and high throughput due to its architectural features. Fiber Channel is not demanding on resources and is great for transferring a large amount of data, since all FC operations are performed on the HBA side, offloading the central processor.

    New versions of the Fiber Channel protocol are backward compatible with previous editions, which opens up good prospects for modernization and scaling. For example, if you implement FC 32Gb / s, then you can still use FC 8Gb / s and 16Gb / s, i.e. It is possible to gradually change FC switches and FC adapters.

    In the near future FC will be upgraded to 64Gb / s and 128Gb / s (now there are switches that support the aggregation of 4 32Gb / s ports into one 128Gb / s channel for connecting switches).

    Ease of configuration and ease of administration have enabled FC to become one of the most common storage protocols. Most SAN administrators around the world know how it works and what benefits it provides when solving various problems. At the same time, FC is still more complicated than Ethernet, although it has a large number of control and monitoring tools.

    FCoE (Fiber Channel over Ethernet)


    The idea of ​​creating FCoE was to consolidate the I / O operations and, as a result, ensure the safe placement of various types of traffic in one "wire". Such an idea meant reducing the total cost of ownership of the system (TCO) by reducing the number of cables and adapters, as well as lowering energy consumption indicators. At the same time, FCoE availability and performance indicators are not comparable with FC indicators, since data transfer requires additional overhead for Ethernet encapsulation.

    In addition, FCoE adds complexity to the deployment and administration of the entire system, increasing the level of requirements for maintenance staff and solution support. Despite the hopes of FCoE manufacturers for widespread adoption, so far this protocol has not been able to supplant FC and is developing at a very slow pace. Distribution of FCoE in the SAN market is currently minimal.



    According to the Fiber Channel Industry Association (FCIA), FCoE predicted protocol speed depends on Ethernet implementations.

    iSCSI (Internet Small Computer System Interface)


    iSCSI builds on the two most commonly used protocols:

    • SCSI - a protocol for exchanging data blocks between a computer and storage
    • IP is a network transport protocol widely used in corporate Ethernet networks.

    iSCSI is a low-cost implementation solution. Administration of such installations is very simple, although to ensure fault tolerance it is necessary to build a dedicated network for iSCSI, which brings us closer to a network implementation very similar to FC SAN.

    ISCSI 10Gbps is thought to provide the same amount of IOps and bandwidth as its comparable 8Gbps FC, but this is not entirely true. Although iSCSI bandwidth is higher, its efficiency is lower than that of FC due to additional overhead.

    ISCSI performance depends on your existing Ethernet infrastructure (the minimum recommended network for iSCSI today is 10Gbps). In the near future (according to Gartner, 10-12 months), it is worth planning to switch to 25/40 / 50GbE, if there is a need to use high-performance all-flash storage.

    Ethernet-based SANs


    The iSCSI, NFS, SMB, FCoE protocols use Ethernet networks for data transmission, therefore it is not advisable to use these protocols in shared networks. This leads us to the need to deploy dedicated networks to use them as SANs.

    Such networks should work in parallel with other main networks and separate user traffic from server data traffic. However, network separation increases administration complexity and total cost of ownership.

    The only exception is the use of shared networks for object storage, as in this case, ensuring minimum delays and maximum performance is not so critical.

    NFS (Network File System)


    Network File System (NFS) is a network file system access protocol originally developed by Sun Microsystems in 1984. NFS provides clients transparent access to the server’s files and file system.

    NFS is easy to configure and administer, as Used over Ethernet networks. As in other protocols using Ethernet, the speed and delays depend entirely on the network implementation at the lower level and most often “rest” against the limitations of the 10GbE standard.

    NFS is often used as an entry-level protocol to build a SAN network for virtualization. According to Gartner, this trend will continue over the next 10 years.

    SMB (Server Message Block)


    SMB is a network file sharing protocol that allows computer applications to read and write files, as well as request server program services on a computer network. Designed by Microsoft to implement the Microsoft Windows Network and File and Printer Sharing. With an increase in the bandwidth of data networks, SMB has become one of the main file access protocols used in storage systems. In storage systems, SMB is most often used in 10GbE networks, because of this, its performance depends heavily on the implementation, settings and network components used.

    Prior to SMB 3.0, the protocol was mainly used to transfer files on local networks. With the new version supporting SMB-Direct technology (using RDMA), it has become actively used in virtualization clusters based on MS Hyper-V as a protocol for accessing a shared storage pool.

    Like NFS, SMB is often used as an entry-level protocol when building a SAN network for virtualization. This trend should also continue in the coming decade.

    InfiniBand (IB)


    InfinBand is a high-speed protocol that provides very high bandwidth and low latency. It is used mainly in the industry of high-performance computing (HPC) and as an interconnect in the creation of high-performance storage systems.

    The most obvious drawback of this technology is the difficulty in setting up and administering. As a result, working with IB requires qualified personnel.
    Among the difficulties of the widespread distribution of InfinBand, limited tools for monitoring performance and diagnosing problems, as well as not the best compatibility with different operating systems, can be noted.

    Compared to FC and Ethernet, the InfiniBand protocol is more expensive to implement and maintain. In addition, there are only a few companies that produce equipment and software for working with IB.



    NVMe (NVM Express)


    NVMe is a fairly new high-performance protocol for accessing solid-state drives connected via the PCI Express bus. The abbreviation "NVM" in the name refers to non-volatile memory (Non-Volatile Memory), which is commonly used in the SSD flash memory type NAND.

    The protocol was developed from scratch. The main goals of its development are low latencies and efficient use of high parallelism of solid state drives. The last task is solved through the use of a new set of commands and a mechanism for processing I / O queues optimized for working with modern processors.

    At the moment, NVMe technology is not yet widespread. Basically, the protocol is used for internal connections in servers and storage. The NVMe specification also allows it to be encapsulated in other protocols, such as Ethernet, FC, and InfiniBand, and to scale the protocol on larger networks. However, since NVMe uses direct memory access (RDMA), the latency of the carrier protocol must be very low for the protocol to function properly.

    In 2017, the active implementation of NVMe is expected, as many manufacturers of server platforms introduce new models with support for dual-port NVMe devices, which will allow designing fault-tolerant storage solutions.

    In the next few years, NVMe is expected to be used as an external gateway, similar to PCIe and InfiniBand. For example, it can be used in small specialized storage networks of dozens of nodes or devices in a homogeneous storage environment. Much wider NVMe will be used as an internal interconnect.



    PCIe


    PCIe has very low latencies and is used mainly in servers for connecting expansion cards, including high-performance NVMe drives. Some new products from well-known manufacturers use PCIe as a protocol for connecting servers through small PCIe switches, i.e. PCIe can only be used in small SAN networks.

    Only a few storage manufacturers in the world use PCIe for external connections, which determines the highly specialized application of this protocol.

    Since PCIe does not use SCSI and requires its own protocol for data transfer, it can increase throughput by reducing latency, in fact, working at the speed of a clean PCIe line. Such solutions require the use of proprietary drivers, which makes them difficult to administer and makes it impossible to create heterogeneous infrastructures, as well as integrate such solutions into existing SAN networks.

    To date, the main technology implementation used is PCIe 3.x, in which performance is increased by 40% compared to PCIe 2.x.

    By the number of PCIe lines it scales from 1GB / s to 16GB / s. In 2017, the new PCIe 4.x standard is released, which will increase productivity by 2 times, i.e. maximum performance will reach 32GB / s.



    Object Storage Protocols


    Object storage includes a large number of new protocols that work on top of HTTP through the API. Today, many new applications are developed taking into account the use of these protocols, this is especially true for backup software.

    The performance of these protocols is highly dependent on the lower level, most often it is Ethernet. The main functional application of object storage protocols is working with unstructured data. In view of this, they are used within the framework of data centers, and are also widely used to access data and write to cloud storage pools.

    Object access protocols require the transfer and storage of a large amount of metadata, which always adds overhead, especially when compared to block access protocols.

    Over the past few years, many object access protocols have been developed. The most popular of them are SOAP, S3, OpenStack Swift. In the near future (according to Gartner, for 5 years) these technologies will be of particular interest.

    Conclusion


    The development of SAN protocols is directly dependent on the development of applications and load profiles.
    Applications and types of loads, such as in HPC, Big Data analytics and active archives, will move the SAN towards creating storage systems with very low latencies and high throughput, as well as support for shared access using NVMe.

    Protocols that will be implemented in the near future (such as 40 / 100GbE), file protocols (such as NFS over RDMA and SMB-Direct) and current block protocols (such as FC 16Gb / s) are already too slow for the next generation of Flash and hybrid storage.
    The main protocol for the next decade will remain FC, since all the necessary infrastructure has been created for it. Many will upgrade to FC 16GB / s, then to FC 32GB / s and newer versions.

    InfinBand, PCIe, and NVMe will remain protocols for connecting end devices, inter-node connections in clusters, or low-latency interconnects. At the same time, there will be small highly specialized solutions for SAN-networks, in which minimum delays and maximum throughput are required. FCoE, iSCSI, NFS, SMB or object protocols will be used primarily as external protocols.

    With every year, interest in object storage systems is growing. This is due to an increase in the amount of unstructured data, the emergence of new tasks for their processing and new requirements for the storage of information.

    Sources used:

    • The Fiber Channel Roadmap, FCIA 2016.
    • Dennis Martin. Demartek Storage Networking Interface Comparison, Demartek 2016.
    • Valdis Filks, Stanley Zaffos. The Future of Storage Protocols, Gartner 2017.
    Tags:

    Also popular now: