324 Infiniband Ports in a Single Rack or Voltaire Grid Director 4700 Switch

    About the data center as a whole, we have already talked and shown almost everything. Now we will gradually focus on the interesting equipment that is installed there, or will be installed in the near future. The topic, of course, is very controversial, but from time to time there are so curious devices that it is simply impossible to resist the temptation. How about a 324-port QDR-Infiniband switch, for example, with a total bandwidth of 50 Terabits? We will talk about him today.

    image



    The switch is used in the cloud computing system of our partners, the company Oversan-Skalaksi and is installed in the data center Oversan-Mercury. As with any other equipment of this class (and price), its selection and purchase was preceded by a thorough study of the solutions available on the market. What was needed was an ultra-fast switch capable of organizing data transfer in the GPFS cluster file system and transferring ethernet traffic. This allows you to combine storage with disks and client machines into one network, while also providing Internet access.

    The data transfer bus must meet the requirements of cluster systems, that is, have a very high data transfer rate, with a margin of several years, and low latency. 40-Gigabit Ethernet is still quite an exotic and expensive bus, and besides, the estimated cost of ownership (TCO) for a system of this size turned out to be completely space. As a result, the choice fell on Infiniband, as a proven technology, run-in, and relatively more affordable. Well, of course, QDR-Infiniband can operate at speeds up to 40 Gb / s, and the switch provides a delay of 100 to 300 nanoseconds when transmitting between ports.

    The Oversan-Skalaxi specialists conducted a whole study on which switching matrix could meet all the requirements. Ultimately, after long discussions, contacts and negotiations with vendors, Gta Director 4700 from Voltaire was chosen as the main switch. Few people know about her and her products, which makes the subject of discussion even more interesting. Meanwhile, the company was formed back in 1997 and has Israeli-American roots. The initial guideline was the release of products for data centers, highly loaded and cluster computing. Apparently, subsequently, cloud projects were added to this list. The company manufactures products, including powerful switching arrays, on Infiniband and Ethernet buses.

    Here's what they say to themselves: "The line of switching matrices for servers and storage systems, as well as advanced management software, can increase productivity in mission-critical applications, increase efficiency and reduce costs by consolidating infrastructure and lower power consumption."

    To the credit of Voltaire, their switches are used in systems that are in the first positions with world ratings of supercomputers, and those or other Voltaire solutions are used in systems Bull, HP, IBM, NEC, Rackable, SGI and Sun.

    About how the switch was purchased, delivered and installed, you can write a separate book. We will not disclose these secrets for now, but it is better to dwell directly on the hardware.

    In the pictures, the switch is shown during the installation process so that you can take a picture of the "insides".

    image

    The four thousandth series of Grid Director switches is designed for the market of cluster computing, where against the background of an ever-growing size and complexity of clusters, lightning speed, excellent controllability and scalable architecture are still required. The same thing is obviously required in the "clouds". In short, the Grid Director 4700 is a management switch that contains 324 40 Gb / s Infiniband ports in a 19-unit chassis. The main module board of the switch (backplane) provides a total throughput of 51.8 Terabits per second, while the delay when switching between ports is in the range from 100 to 300 nanoseconds. The numbers, as they say, speak for themselves. For connection, special copper connectors QSFP and the corresponding cables are used. At the same time, the switch itself determines the characteristics of the connected cable and sets the optimal operation parameters. On the one hand, this seems to be a very useful feature, and on the other hand, if you don’t want surprises, buy Voltaire branded cables. And they, like, in general, everything related to high-speed data transfer, are now not cheap.

    Grid Director 4700 via HP DL-series servers with IB-modules connected HP MSA disk storage with GPFS cluster file system implemented on them. This provides good disk I / O throughput in virtual instances. Actually the solution to the problem of input-output (I / O bottleneck) and the solutions of Voltaire are famous. Accordingly, using Grid Director, Oversan-Skalaksi solves the main problem of all Russian hosting services - low disk I / O speed. Ethernet traffic is also transmitted through Infiniband using the proprietary Oversan-Scalaxi design. And finally, the high bandwidth of Infiniband is needed to provide customers with a wide interconnect between virtual instances and a corresponding channel to the Internet through powerful uplinks of the Oversan-Mercury data center.

    General Ethernet switching and the organization of Internet channels are assigned to Juniper EX82 switches. We also plan to talk about them in detail in the near future.

    image

    Up to 18 slots can be installed in the chassis, each of which contains a line card for QDR connectors. There are, in turn, 18 of them on the board. This is where the 324 Infiniband ports come from. The same number of ports is achieved in a different way - nine commutation matrices with 36 ports each are installed in the chassis. All installed modules are managed using the Unified Fabric Manager proprietary software.

    Matrix connectors on a modular board.

    image

    The entire engine is powered by six reserved hot-swappable power supplies. This was done mainly for reliability, the consumption of the system itself is not so great - a maximum of 12 watts per port. At first glance it may seem a lot, but do not forget that the Grid Director 4700 alone occupies almost half the rack.

    image

    For cooling, two fan modules are responsible, vertical and horizontal. At the same time, the general direction of air movement remains standard, from front to back. This is done so that the Grid Director can be installed in ordinary racks in standard data centers, although all ports on the switch are located at the rear, which is not so common.

    image

    And one more touch - when fully loaded with modules, the switch weighs 170 kilograms. It’s solid, but not as much as it could be - two engineers can easily cope with the installation.

    Below is cooling and power.

    image

    Some storages are connected directly via SAS to servers with Infiniband modules.

    image

    Side view.

    image

    Indicators on the modules.

    image

    Nameplate close-up. Modest but tasteful.

    image

    Also popular now: