Where to store data: a tour to the IXcellerate data center
We often hear that storing data and putting its systems to IT outsourcing is dangerous and unreliable. Allegedly, the data may be unavailable due to technical problems with the provider, the data center can become the object of a hacker attack, and simply calmer when all the IT equipment and all company data are near, “close by” and not stored and maintained “on side. "
I propose to see how IXcellerate data center is arranged, in which we store our data and customer data. We are a service provider of Onlanta . Our cloud OnCloud.ru is based on several geographically distributed data centers. IXcellerate data center is one of them.
IXcellerate data processing center is located on the territory of the former industrial zone and reinforced concrete plant in the Altufyevo area on a site of more than 15 thousand square meters. m - this is one of the largest data centers in Moscow.
The choice of the territory of the concrete products plant to create a data center was not accidental - such enterprises usually have powerful lines for the input of electricity, which is exactly what is needed for the data center. There are two such lines on the territory of the data center - from two different distribution substations. The distance between their entry into the building is at least 150 m. This greatly increases the reliability and availability of data center services, since the probability of failures due to accidents on the power grid is greatly reduced.
The design capacity of the data center is 13.7 megawatts, and the capacity is 1835 racks, and this number is growing. The building area is 6000 square meters. m. In addition, the data center has more than 600 square meters. m warehouse for customers where they can store the necessary equipment for the work.
Two access roads lead to the data center building. The territory is removed from airports, railways, highways, various shopping and entertainment centers, as well as residential buildings.
Building a data center with installed cooling devices (chillers)
In addition to the main power supply from two power substations in the suburbs, the data center also has backup diesel generators that generate electricity in emergency situations, for example, if both substations fail during a disaster.
For diesel generators supplied special fuel - summer and winter. Special protective concrete blocks were placed near the diesel generators so that they would not be accidentally damaged by moving equipment.
Street container with a diesel generator behind protective concrete blocks
The efficiency of diesel engines is monitored around the clock online, and the duty officer performs a detour every four hours. During our visit, diesel generators were just serviced, and there were vehicles with test equipment on the territory.
In the event of a power failure, the batteries create the required time interval until the backup diesel generators, after starting, reach the required power.
Maintenance of diesel generators
The data center strictly complies with the requirements of both physical and information security, as well as the security of electrical grids. The data center building is faced with so-called. sandwich panels both outside and inside. They provide thermal insulation and physical protection. The data center was built according to the “box-in-box” principle, that is, from the street, interior rooms are separated by trees, a concrete fence, a lawn and sandwich panels.
The roof of the building is located at a height of 14 m, and the roof of the engine room is 9 m. Between them there is a protective layer against leakage.
Multi-layering allows you to maintain certain temperature and humidity parameters that are optimal for the operation of server equipment. Microclimate is also created by air coolers (chillers) and precision air conditioners.
The IXcellerate data center was built modularly, in three phases. The machine hall of the first phase during construction in 2012 was the largest in Moscow. The halls of the second and third phases round the hall of the first phase around the perimeter of the letter G.
Each machine hall has its own loading and unloading zones. A separate discharge zone is also provided for the fire station.
The IXcellerate
IXcellerate data center scheme was created as a “neutral” data center, independent of telecom and cloud service providers, and focused exclusively on co-location services, i.e. placement of customer equipment, its protection and provision of the required engineering infrastructure with an established level of service under a standard contract.
It happens that customers ask about cloud services. For such cases, the data center has an ecosystem of partners, on the equipment of which cloud solutions are provided. We are doing just that.
The data center requires employees and authorized clients to undergo dual identification, including a standard ID card and biometric control (hand palm scanner).
Access control for employees
Access to cells (“cells”) where client equipment is installed is under round-the-clock control, that is, only data center employees authorized by the client can get there. These cells are separated from each other by a metal grid.
The cells also have their own physical protection, such as combination locks and video cameras. In total, there are five lines of physical security with a constantly working building monitoring and control system (BMS) Computrols.
High quality electricity is used to power the equipment of the customers. What does the word “quality” mean for electricity? It is no secret that industrial electric current sometimes has peak bursts that can damage customer equipment. Therefore, there is a special “electric room” in the data center, where these emissions are eliminated. The result is an electric current with a frequency of 50 Hertz, which has the form of a "pure" sinusoid, without emissions and peaks. Electrical distribution equipment constantly monitors nine parameters of current quality.
Data center power supply redundancy is organized at the 2N level: as already mentioned, two independent power supply beams approach the data center. Damage to one route of laying will not entail stopping the data center or its part, as there is always a backup route.
This is how incoming lines from power substations on one of the laying routes look.
Power supply from the transformer substation
Here is a small touch to the topic of electrical and fire safety. Electrical wiring from the switches is laid in metal pipes so that when a short circuit is possible, no sparks fly out.
Switches for lighting lamps with wiring in metal pipes
All power cables run under a raised floor 1 m high. Under the raised floor, cold air with a given humidity is supplied to equipment racks in full compliance with ASHRAE international standard.
Temperature and humidity sensors are installed on the cells with client equipment, which not only visually display these parameters, but also regularly transmit them to the centralized control system of the engineering infrastructure of the data center.
Temperature and Humidity Sensor
For powering each rack under the raised floor there are two sockets - each from a separate input line from different electrical substations. For even greater reliability, they are laid in separate gutters.
Two rack power sockets (through the raised floor grille)
Some customers also install their own video cameras in their cells with equipment to increase security.
The data center has “telecom zones”, where the equipment of 44 telecom operators is installed, including Orange, Sprint, Beeline, MegaFon, MTS, Telia, other operators and Internet providers, as well as the MSK-IX node and the Eurasia Peering traffic exchange platform IX, uniting 38 resident operators.
In the data center for the installation of client equipment accepted any size and functionality, even tape drives.
One of our clients - a television channel with a multimillion audience - makes reservations here for broadcasting in HD.
The ability to quickly and effectively extinguish the fire, and it is better not to allow it at all - one of the most important characteristics of any data center.
IXcellerate uses a three-tier fire detection and extinguishing system. The first level is the Vesda ultra-early smoke detection system that reacts to a change in air density and the amount of impurities and dust of just 0.01%. A self-learning system with elements of artificial intelligence keeps in mind a certain matrix of air parameters. Such a system can recognize smoke at a very early stage, as well as determine the sector where the change in parameters occurred.
Apollo - an integrated fire alarm and warning system - the next step in the fire safety data center.
But IXcellerate's pride is the HI-FOG automatic fire extinguishing system, which is capable of localizing fires on an area of 4 x 4 square meters. m (with the possibility of finding staff directly in the gym and without impact and consequences for other areas). To do this, the HI-FOG system uses water sprays (sprinklers), which are under constant pressure and are placed over each local perimeter of 4 x 4 m.
It is important to note that the HI-FOG uses a dry pipe system, i.e. a substance used in the normal mode to extinguish the fire - distilled water - is stored in the tank, which is placed in the fire and pumping station.
One of the sprinklers
in the sprinkler is ball with ethereal substance, which breaks at a temperature of 57 to C and then started automatically extinguishing system.
The HI-FOG system creates a fine mist that closes the fire zone, displaces oxygen from it and lowers the temperature. That is, the system allows to eliminate two causes of their three possible fires. The third theoretically possible cause of a fire is the presence of combustible material. However, the already mentioned sandwich panels, which are faced with both external and internal walls, as well as raised floor plates in the hall are made of non-combustible material. These panels can provide up to 90 minutes of fire protection. In addition, only low smoke cables are used in the data center.
The HI-FOG system is perhaps optimal for use in the data center, because it provides local extinguishing, rather than filling the entire hall with gas foam, causing damage to equipment that is not covered by fire. In addition, its use does not require mandatory evacuation of people.
The FM Approved certificate, issued by a global leader in independent testing, allows the use of HI-FOG technology also in the energy industry, in the processing and processing industry, as well as in submarines and cruise ships.
Fire Station A
tank with distilled water (blue, visible in the back of the photo) is designed to extinguish the entire area of the data center. The same system works in the Moscow metro and the Bolshoi Theater.
The 24 x 7 control and monitoring control panel contains duty officers who monitor the work of engineering systems and receive applications from clients. The video surveillance system allows you to quickly move from displaying the general scheme of the data center to images of individual rooms, cells, corridors and equipment racks, components of the fire extinguishing system (including sprinklers), power supplies, and air conditioning.
The standard SLA agreement provides that the first response to a request for technical support must be given within 30 minutes. However, in the data center, the actual time of customer feedback does not exceed six minutes, according to statistics from the Zendesk portal, which is used to register requests.
Support is provided in Russian, English and Chinese.
Control and monitoring system console
Since the launch of the data center since its launch in 2013, there has not been a single downtime (or downtime) or a single violation of the SLA.
The data center and its well-coordinated team leaves the impression of a living organism.
Inside the data center there is even a place for music: when you cross the corridor from one hall to another, you can not miss the equipment for the rock band in which Cliff Gauntlett once played and who performs at corporate events and the mandatory annual open air festival IXcellerate - Rockin ' Russia.
The corridor between the halls of the data center
IXсellerate plans to build new sites, in addition to the Moscow One there will be data centers Moscow Two, Moscow Three, etc.
Our company " Onlanta“I started cooperation with IXcellerate from two pillars to implement an ambitious geodistributed cluster project at that time. At that time, no one on the Russian market had such a solution. The dark optics channels cost us space money compared to the profit from the services we provided. Nobody bought a separate service for a disaster-resistant cloud, so we just started developing our cloud at two sites at once. Then a third one appeared, as it became obvious that it was impossible to provide a full-fledged Resilience and load balancing without a separate “witness" who would not be tied to one of the two sites. But objectively, we can say that of all three sites, IXcellerate is the best - both in terms of service organization and overall management adequacy.
Share and you what you value most in your data center and, conversely, what annoys you most?
I propose to see how IXcellerate data center is arranged, in which we store our data and customer data. We are a service provider of Onlanta . Our cloud OnCloud.ru is based on several geographically distributed data centers. IXcellerate data center is one of them.
IXcellerate data processing center is located on the territory of the former industrial zone and reinforced concrete plant in the Altufyevo area on a site of more than 15 thousand square meters. m - this is one of the largest data centers in Moscow.
The choice of the territory of the concrete products plant to create a data center was not accidental - such enterprises usually have powerful lines for the input of electricity, which is exactly what is needed for the data center. There are two such lines on the territory of the data center - from two different distribution substations. The distance between their entry into the building is at least 150 m. This greatly increases the reliability and availability of data center services, since the probability of failures due to accidents on the power grid is greatly reduced.
IXcellerate data center began to be built in 2012 and a year later the first machine hall was put into operation. The founder of IXcellerate is a well-known international investor and British businessman Guy Wilner .Guy Willner (right)
Co-founder Guy Wilner was another international businessman, Cliff Gauntlett, who started working in the Soviet Union in the late 1980s. He turned out to be one of those foreigners who came here during perestroika, fell in love with the country, and, despite all her problems, remained here forever (Cliff died in 2013). Probably many people remember phone cards ROL. They were issued by the Russian provider Golden Telecom, where Cliff worked as a senior vice president. In addition, Cliff was also an avid "rocker" - headed by the famous ROL Band, and his songs are even in the repertoire of the Scorpions group. Hence the motto of the IXcellerate data center - Rockin'Russia.Corpine Festival Rockin'Russia: Cliff Gauntlett with a guitar, behind the drums - Alexander Vershbow, US Ambassador to Russia. (photo www.passportmagazine.ru )
The first machine room of the IXcellerate data center was named in honor of Cliff Gauntlett.Cliff's guitar still hangs in the data center, with Paul McCartney’s original autograph.
The design capacity of the data center is 13.7 megawatts, and the capacity is 1835 racks, and this number is growing. The building area is 6000 square meters. m. In addition, the data center has more than 600 square meters. m warehouse for customers where they can store the necessary equipment for the work.
Two access roads lead to the data center building. The territory is removed from airports, railways, highways, various shopping and entertainment centers, as well as residential buildings.
In addition to the main power supply from two power substations in the suburbs, the data center also has backup diesel generators that generate electricity in emergency situations, for example, if both substations fail during a disaster.
For diesel generators supplied special fuel - summer and winter. Special protective concrete blocks were placed near the diesel generators so that they would not be accidentally damaged by moving equipment.
The efficiency of diesel engines is monitored around the clock online, and the duty officer performs a detour every four hours. During our visit, diesel generators were just serviced, and there were vehicles with test equipment on the territory.
In the event of a power failure, the batteries create the required time interval until the backup diesel generators, after starting, reach the required power.
The data center strictly complies with the requirements of both physical and information security, as well as the security of electrical grids. The data center building is faced with so-called. sandwich panels both outside and inside. They provide thermal insulation and physical protection. The data center was built according to the “box-in-box” principle, that is, from the street, interior rooms are separated by trees, a concrete fence, a lawn and sandwich panels.
The roof of the building is located at a height of 14 m, and the roof of the engine room is 9 m. Between them there is a protective layer against leakage.
Multi-layering allows you to maintain certain temperature and humidity parameters that are optimal for the operation of server equipment. Microclimate is also created by air coolers (chillers) and precision air conditioners.
Data Center Awards
Дата-центр имеет сертификаты ISO 27001и ISO 9001, сертификат по стандарту безопасности данных для платёжных карт PCI DSS v3.2, а также сертификат соответствия уровню надежности IBM Level 3 в категориях «Проект», «Строительство» и «Эксплуатация». В 2018 году дата-центр IXcellerate получил сертификат Uptime Institute на соответствие уровню Tier III в категории «Дизайн».
Уровень обслуживания SLA, предоставляемый для клиентов составляет, как говорят ИТ-специалисты, «пять девяток». Это означает, что инфраструктура дата-центра доступна 99,999% времени рабочего времени. То есть время возможного простоя в год не превышает пяти минут и шестнадцати секунд.
Дата-центр третий год подряд является победителем и обладателем награды Russian DC Awards за самое эффективное решение по охлаждению и электроснабжению, за инновационное решение в области снижения совокупной стоимости владения ТСО (Total Cost of Ownership), а также локализацию персональных данных.
Вложения в дата-центр составляют порядка 50 миллионов долларов, при этом уровень операционной безубыточности был достигнут после 4 лет эксплуатации, а не за обычные для дата-центров такого класса 8-10 лет.
«Тумба гордости» с наградами и сертификатами дата-центра
Уровень обслуживания SLA, предоставляемый для клиентов составляет, как говорят ИТ-специалисты, «пять девяток». Это означает, что инфраструктура дата-центра доступна 99,999% времени рабочего времени. То есть время возможного простоя в год не превышает пяти минут и шестнадцати секунд.
Дата-центр третий год подряд является победителем и обладателем награды Russian DC Awards за самое эффективное решение по охлаждению и электроснабжению, за инновационное решение в области снижения совокупной стоимости владения ТСО (Total Cost of Ownership), а также локализацию персональных данных.
Вложения в дата-центр составляют порядка 50 миллионов долларов, при этом уровень операционной безубыточности был достигнут после 4 лет эксплуатации, а не за обычные для дата-центров такого класса 8-10 лет.
Creating a data center
The IXcellerate data center was built modularly, in three phases. The machine hall of the first phase during construction in 2012 was the largest in Moscow. The halls of the second and third phases round the hall of the first phase around the perimeter of the letter G.
Each machine hall has its own loading and unloading zones. A separate discharge zone is also provided for the fire station.
IXcellerate data center scheme was created as a “neutral” data center, independent of telecom and cloud service providers, and focused exclusively on co-location services, i.e. placement of customer equipment, its protection and provision of the required engineering infrastructure with an established level of service under a standard contract.
It happens that customers ask about cloud services. For such cases, the data center has an ecosystem of partners, on the equipment of which cloud solutions are provided. We are doing just that.
Let's see what's inside
The data center requires employees and authorized clients to undergo dual identification, including a standard ID card and biometric control (hand palm scanner).
Access to cells (“cells”) where client equipment is installed is under round-the-clock control, that is, only data center employees authorized by the client can get there. These cells are separated from each other by a metal grid.
The cells also have their own physical protection, such as combination locks and video cameras. In total, there are five lines of physical security with a constantly working building monitoring and control system (BMS) Computrols.
In the data center, it has become a good tradition to call each machine room in honor of the company or the person who made the greatest contribution to its creation.
The first phase hall, named after Cliff Gauntlett, is designed for 220 seats with customer equipment.
During the construction of the hall of the second phase, the investments of the Japanese company Sumitomo Corp. were used, therefore it is called The Summit Hall. The third-phase hall is named “Height” in honor of the technical director of the data center, Sergey Vladimirovich Vyshemirsky, who used to work at the Chkalovsky airfield.Plate with the name of the third engine room
Power and Cooling
High quality electricity is used to power the equipment of the customers. What does the word “quality” mean for electricity? It is no secret that industrial electric current sometimes has peak bursts that can damage customer equipment. Therefore, there is a special “electric room” in the data center, where these emissions are eliminated. The result is an electric current with a frequency of 50 Hertz, which has the form of a "pure" sinusoid, without emissions and peaks. Electrical distribution equipment constantly monitors nine parameters of current quality.
Data center power supply redundancy is organized at the 2N level: as already mentioned, two independent power supply beams approach the data center. Damage to one route of laying will not entail stopping the data center or its part, as there is always a backup route.
This is how incoming lines from power substations on one of the laying routes look.
Here is a small touch to the topic of electrical and fire safety. Electrical wiring from the switches is laid in metal pipes so that when a short circuit is possible, no sparks fly out.
All power cables run under a raised floor 1 m high. Under the raised floor, cold air with a given humidity is supplied to equipment racks in full compliance with ASHRAE international standard.
Temperature and humidity sensors are installed on the cells with client equipment, which not only visually display these parameters, but also regularly transmit them to the centralized control system of the engineering infrastructure of the data center.
For powering each rack under the raised floor there are two sockets - each from a separate input line from different electrical substations. For even greater reliability, they are laid in separate gutters.
Some customers also install their own video cameras in their cells with equipment to increase security.
The data center has “telecom zones”, where the equipment of 44 telecom operators is installed, including Orange, Sprint, Beeline, MegaFon, MTS, Telia, other operators and Internet providers, as well as the MSK-IX node and the Eurasia Peering traffic exchange platform IX, uniting 38 resident operators.
In the data center for the installation of client equipment accepted any size and functionality, even tape drives.
One of our clients - a television channel with a multimillion audience - makes reservations here for broadcasting in HD.
Fire safety
The ability to quickly and effectively extinguish the fire, and it is better not to allow it at all - one of the most important characteristics of any data center.
IXcellerate uses a three-tier fire detection and extinguishing system. The first level is the Vesda ultra-early smoke detection system that reacts to a change in air density and the amount of impurities and dust of just 0.01%. A self-learning system with elements of artificial intelligence keeps in mind a certain matrix of air parameters. Such a system can recognize smoke at a very early stage, as well as determine the sector where the change in parameters occurred.
Apollo - an integrated fire alarm and warning system - the next step in the fire safety data center.
But IXcellerate's pride is the HI-FOG automatic fire extinguishing system, which is capable of localizing fires on an area of 4 x 4 square meters. m (with the possibility of finding staff directly in the gym and without impact and consequences for other areas). To do this, the HI-FOG system uses water sprays (sprinklers), which are under constant pressure and are placed over each local perimeter of 4 x 4 m.
It is important to note that the HI-FOG uses a dry pipe system, i.e. a substance used in the normal mode to extinguish the fire - distilled water - is stored in the tank, which is placed in the fire and pumping station.
in the sprinkler is ball with ethereal substance, which breaks at a temperature of 57 to C and then started automatically extinguishing system.
The HI-FOG system creates a fine mist that closes the fire zone, displaces oxygen from it and lowers the temperature. That is, the system allows to eliminate two causes of their three possible fires. The third theoretically possible cause of a fire is the presence of combustible material. However, the already mentioned sandwich panels, which are faced with both external and internal walls, as well as raised floor plates in the hall are made of non-combustible material. These panels can provide up to 90 minutes of fire protection. In addition, only low smoke cables are used in the data center.
The HI-FOG system is perhaps optimal for use in the data center, because it provides local extinguishing, rather than filling the entire hall with gas foam, causing damage to equipment that is not covered by fire. In addition, its use does not require mandatory evacuation of people.
The FM Approved certificate, issued by a global leader in independent testing, allows the use of HI-FOG technology also in the energy industry, in the processing and processing industry, as well as in submarines and cruise ships.
tank with distilled water (blue, visible in the back of the photo) is designed to extinguish the entire area of the data center. The same system works in the Moscow metro and the Bolshoi Theater.
Remote control and monitoring
The 24 x 7 control and monitoring control panel contains duty officers who monitor the work of engineering systems and receive applications from clients. The video surveillance system allows you to quickly move from displaying the general scheme of the data center to images of individual rooms, cells, corridors and equipment racks, components of the fire extinguishing system (including sprinklers), power supplies, and air conditioning.
The standard SLA agreement provides that the first response to a request for technical support must be given within 30 minutes. However, in the data center, the actual time of customer feedback does not exceed six minutes, according to statistics from the Zendesk portal, which is used to register requests.
Support is provided in Russian, English and Chinese.
Since the launch of the data center since its launch in 2013, there has not been a single downtime (or downtime) or a single violation of the SLA.
The data center and its well-coordinated team leaves the impression of a living organism.
Inside the data center there is even a place for music: when you cross the corridor from one hall to another, you can not miss the equipment for the rock band in which Cliff Gauntlett once played and who performs at corporate events and the mandatory annual open air festival IXcellerate - Rockin ' Russia.
IXсellerate plans to build new sites, in addition to the Moscow One there will be data centers Moscow Two, Moscow Three, etc.
Our company " Onlanta“I started cooperation with IXcellerate from two pillars to implement an ambitious geodistributed cluster project at that time. At that time, no one on the Russian market had such a solution. The dark optics channels cost us space money compared to the profit from the services we provided. Nobody bought a separate service for a disaster-resistant cloud, so we just started developing our cloud at two sites at once. Then a third one appeared, as it became obvious that it was impossible to provide a full-fledged Resilience and load balancing without a separate “witness" who would not be tied to one of the two sites. But objectively, we can say that of all three sites, IXcellerate is the best - both in terms of service organization and overall management adequacy.
Share and you what you value most in your data center and, conversely, what annoys you most?
Or maybe you want to work with us?