What should we build an open data center? Some details about the structure of Facebook DC in Altoon

Over the past year, Facebook has brought many interesting things to the standards for the development of network equipment. Moreover, if most developers leave their projects proprietary, then Facebook opens up innovation for others. Actually, for a company whose goal is to provide the public with an exchange of information, this model of work makes sense. At the same time, an open scheme of work allows Facebook to save money. According to James Taylor, vice president of infrastructure for the past 3 years, Facebook has saved more than $ 2 billion by allowing members of the Open Compute Project to work on their own specifications.

In addition, Wedge , an open top-of-rack switch developed by the OCP community , is catching on . This was followed by a 6-pack ,FBOSS and OpenBNC . Facebook has built its new data center, based on the development of the Open Compute Project, in Altoon, Iowa, USA. At the same time, the company provided all the necessary information about this project . There are several ideas that can be used in data centers of other companies, and DCs of any size.

Facebook Cluster Design

The first image shows the aggregated cluster design of DC in Altun. The developers call the architecture of the 4-post data center. Here, up to 255 racks can be combined through ToR switches into high-density cluster switches (CSW). RSW can have up to 44 10G downlinks and 4 or 8 10G uplinks. Four CWS and connected RSWs constitute a cluster.



Four aggregated FatCat (FC) switches combine clusters into a single system. Each CSW has a 40G connection with one of four FCs. The 80G guard ring integrates CWS within each cluster, and FCs are connected to the 160G guard ring.

This is a really good structure for several reasons, including the reliability and practicality of the system. However, for Facebook this was not enough. The fact is that many problems in this kind of architecture are a consequence of the need to install very large switches for CSW and FC.

What's up with Altoona?

In the architecture of the new generation data center from Altoona, most of the problems of cluster architecture have been resolved, while retaining the best features of this type of architecture.

For example, not a few large switches are used here, but many small ones. Moreover, each switch is responsible for a small percentage of the load, and failure of one switch is not a significant problem;

Also in such DCs, capital and operating costs are reduced;

An increase in the size and power of a data center of this type is carried out in a very short time, much cheaper than in conventional DCs.

The network topology of such a DC is shown in the following image, where Clos can be quickly recognized.. Instead of working with hundreds of racks in cluster design, here each of the topological units is responsible for 48 racks.





Below is a three-dimensional diagram of the topology of this kind of data center.



Representatives of Facebook argue that the modular design of the data center allows you to quickly change the structure of the data center, adding or removing certain elements. All changes are made in the minimum time with the minimum expenses. This point of view is explained in more detail here:



The advantage of the new type of data center from Facebook is the ability to use small switches in an architecture that allows you to scale an object to any size, without the need to change the base blocks.

Switches can be used from Accton, Quanta, Celestica, Dell and some other companies. At the same time, the cost of a switch from Quanta with 32 40G ports is $ 7,495, while the Juniper QFX5100 with 24 40G ports costs a little less than 30 thousand.

Hyperscaling - what is it?

Most telecommunications professionals apply this term only to giants like Amazon, Google, Facebook. However, the term means the ability to change scaling in a very short time. A hyper-scaled data center can be relatively small, but it can be increased at any time without fundamental changes to the infrastructure. Also in this case, it should be possible to use the same switches and connections that were used initially.

There can be only a few racks in a DC, but it can already be a hyperscale-DC.

Another misconception in terms of hyper-scaling is the belief that DCs of this type are “sharpened” for optimal operation with one or more main applications. This is not entirely true. Ideally, hyperscale-design means the ability to support hundreds of business applications with the same ease with which DC works with big-data, search applications or social media.

As for Facebook DC, here you can add additional blocks and layers without any problems, and as much as you need at a particular moment.