PhosAgro: Data Center for four months
Today we would like to talk about one of the latest projects to create a corporate data center. This is the Cherepovets Data Center of PhosAgro, the main activity of which is the production of phosphorus fertilizers, phosphate raw materials, as well as feed phosphates, nitrogen fertilizers and ammonia. The PhosAgro Group includes Apatit JSC and its Balakovo branch, PhosAgro-Cherepovets JSC, Metakhim JSC, PhosAgro-Trans LLC, PhosAgro-Region LLC and NIUIF JSC. The group is the largest European producer of phosphate fertilizers, as well as the world's largest producer of high-grade phosphorus raw materials.
The main goals of creating a new data center were the centralization of the PhosAgro server complex in one place, as well as the reorganization of the network core. The project was implemented in just four months, and, according to classical technology, with the overhaul of the premises. How did you manage to achieve this, you ask? First of all, thanks to an efficiently working team, timely delivery of equipment, coordinated installation of various subsystems. Well, of course, a competent choice of technical solutions helped. Details are below.
Before dwelling on the construction of their own data center using classical technology, PhosAgro experts considered other options for solving existing IT problems. One of them is the use of commercial data center services. But after analyzing the offers of several providers, we realized that high operating costs (OPEX) lead to the fact that this option is unprofitable (above TCO) with a large planning horizon. In addition, the concentration of the data center in Moscow and St. Petersburg meant the need for a difficult and expensive transfer of the server complex to one of the capitals - another minus of this option.
Next, the possibility of installing a container data center was considered. However, the benefit was not obvious in view of the technical circumstances and the fact that the existing building had enough resources, and its modernization was not so costly. Such a solution is in demand in cases where there is a shortage of physical space for equipment or if it is assumed that it can be relocated to another place - neither of these is relevant for PhosAgro.
As a result, it was decided to quickly create a data center in a three-story building of the IT service on the territory of PhosAgro.
The main criterion for creating a data center was fault tolerance, so it was decided to purchase all the basic engineering equipment from one vendor. The mono-vendor solution also provides 100% compatibility of all components of the engineering infrastructure, and reduces the cost of operation and maintenance. It eliminates disputes between contractors about the causes of malfunctions and contributes to their prompt resolution. Another advantage of a single vendor data center is a single service contract. It is easier to respond to changes and deviations from the norm - the vendor identifies the reasons, he makes a decision, for example, to replace a failed component. No need to look on whose side the problem is.
Thus, PhosAgro considered only mono-vendor solutions and almost immediately the offer of Schneider Electric was chosen. The quality and accumulated positive operating experience of Schneider Electric solutions have contributed to this decision. Schneider Electric equipment is successfully operated at the PhosAgro site in Kirovsk, and the staff has specialists trained by this company.
At the next stage, a tender was held in which seven large integrators with high Schneider Electric partner status and experience in implementing similar projects participated. The IBS Platformix company won, offering the best price and terms of the project.
The project was required to be implemented in a short time, since the state of engineering systems in the existing server rooms no longer made it possible to provide the required reliability indicators. Thanks to the coordinated work of a team of customer, manufacturer and integrator specialists, prompt delivery and installation of equipment, the data center was built and put into operation in a record short time. The results of the tender were announced in August 2016, and the main work on the project was completed before the end of winter.
Interestingly, the work of specialists in various fields during the construction of the data center was carried out simultaneously. For example, the delivery dates of chillers were agreed with the terms of construction and installation works - the equipment was installed in its place literally “from the wheels”, without additional movements around the facility. A clear synchronization of work together with representatives of the vendor has reduced the time to complete them.
Despite the prompt creation of a new site, the PhosAgro data center cannot be called typical. It consists of two rooms, one of which is intended for standard IT equipment, the second - for racks where data storage systems and non-standard form factor servers are located. In the first hall, 28 racks were installed (6 racks of 7 kW each, the rest - 3.5 kW each) and insulation of the hot corridor was implemented.
EcoAisle Hot Corridor Air Isolation System
In the second hall, where IT equipment of non-standard dimensions is located, a slightly different scheme is implemented: a cold corridor is also highlighted between two rows of racks, but it is not isolated. The data center areas allow you to place all the equipment available in the Group and leave a reserve for growth.
The design load of the data center for IT equipment is 250 kW, engineering systems consume the same amount. Power is supplied to the data center from a medium-voltage substation with a total capacity of 1 MW (two transformers of 500 kW each). At the site, two UPSs with a capacity of 500 kW each are deployed according to the mutual redundancy scheme and can power not only IT, but also critical elements of the engineering infrastructure, and the modular architecture eliminates the presence of a single point of failure. Installed devices Symmetra PX is a modular uninterruptible power supply system with high efficiency - 95.6%, which allows to reduce the cost of power supply for both the UPS and its conditioning system (for this purpose, a standard freon split system is used).
Symmetra PX Schneider Electric Modular Scalable Three-Phase Uninterruptible Power System
For short-term stand-alone power supply of the IT load, a set of rechargeable batteries has been installed, designed to provide stand-alone power supply for 10-15 minutes - this time is enough to guarantee the diesel engine starts. But while the data center is not loaded at full capacity, the batteries are able to guarantee an autonomy time of more than an hour. DGU is used as a backup source.
Another important requirement for the new data center was high energy efficiency. The PhosAgro enterprise in Cherepovets is well supplied with energy, so there was no problem getting the required capacity (although a thorough “cleaning” of the electricity coming from the plant was required). Energy efficiency was important not only because of reduced energy consumption, but rather for the sake of reducing environmental impact. This is all the more symbolic, since 2017 is the year of ecology in Russia. PhosAgro IT executives wanted to prove that they could quickly and efficiently build a reliable and energy-efficient facility.
A lot of factors affect energy efficiency, including, in particular, the efficiency of installed UPS systems. But the main contribution to reducing energy consumption is the active use of free cooling (freecooling). The new Data Center has the innovative Aquaflair chiller system, which uses the free-cooling function from October to April. In this mode, only the circulation pump that pumps the coolant (ethylene glycol) works, while consuming a minimum of electricity.
Schneider Electric Uniflair TRAF Series Chillers with Freecooling
To increase cooling efficiency, the already mentioned EcoAisle hot corridor insulation system and InRow row air conditioners in a compact case (only 300mm wide) are provided. The cooling capacity of such an air conditioner is up to 62 kW with an energy consumption of 1.8 kW. Thus, the efficiency factor of the air conditioner is more than 34 kW of cold per 1 kW of electricity.
InRowtm APC by Schneider Electric
The new data center is built in accordance with the requirements for reliability and fault tolerance for Tier III facilities, which, in particular, allows you to disable the components of the engineering infrastructure for their preventive maintenance and repair without interrupting the operation of the data center.
For some key components, even triple redundancy is provided. For example, three chillers are installed: one of them is in working mode, the second is in standby, and the third is in “standby” mode. Every few hours they change roles. This increases fault tolerance and ensures uniform production of equipment life. Any unit can be taken out of service for maintenance without disrupting the operation of services.
In addition, the PhosAgro IT system construction concept envisaged the creation of two geographically dispersed data centers for disaster tolerance - in Cherepovets and Kirovsk (Murmansk Region). The site in Kirovsk, now a backup data center was built three years ago. It is similar to the new, basic, but it uses refrigeration systems on freon. There are also two rooms in the Kirovsk data center, but they are designed for fewer racks. Currently, PhosAgro IT systems are distributed across these two data centers.
The StruxureWare Data Center Expert system is used to monitor engineering systems in the new data center. This is a scalable monitoring system independent of the equipment of a particular manufacturer, which collects data from various sensors and “smart” devices, provides the generation of alarm messages and other functions, forming a unified view of the complex physical infrastructure of the facility.
A centralized management system for all engineering infrastructure facilities provides end-to-end monitoring of all levels of engineering systems and improves the efficiency of the service. StruxureWare not only monitors various environmental parameters, but also stores the information received for subsequent analysis.
Even before the main tender, at the design stage, Schneider Electric experts carried out an examination of the project, made their comments, prepared a specification of the equipment. Representatives of the manufacturer were present at the site at all stages: installation of the hot corridor, electrical and cold supply systems, including the most difficult stage of commissioning and commissioning of equipment.
Schneider Electric quickly completed the installation supervision and tested the equipment. Under the contract, she will serve both sites for three years. The vendor conducts regular inspections of equipment (cooling and power systems) for scheduled preventive repairs. At a predetermined time, a Schneider Electric engineer conducts a comprehensive survey of refrigeration systems twice a year and power systems once a year.
Notifications for sending alarm messages to the Help Desk are configured in StruxureWare, cases are started automatically, and the engineer promptly takes action. All components of engineering systems conduct self-tests on a regular basis, and any errors in their operation are displayed in the monitoring system, which allows you to immediately respond to emergency situations.
In recent years, PhosAgro Group has been investing heavily in the development of IT, automation of production and business. The centralization of IT resources in Cherepovets contributed to the implementation of such projects as a single electronic archive with the exchange of documents between holding sites. Business-critical applications now run on diversity clusters. The sites are connected by a redundant channel of 400 Mbps. User access in the new data center is organized at a speed of 1 Gbit / s, and for other company sites in the five largest cities of Russia - 200 Mbit / s.
It is clear that the processes of "digital transformation" are impossible without modern technologies. And with the help of the platform, which the engineering infrastructure of Schneider Electric will provide for the PhosAgro data center, reliable and efficient operation and sustainable development of IT systems are implemented. The data center allowed providing IT equipment with high-quality uninterrupted power supply, creating the necessary climatic conditions for its operation. Now the data center’s power supply load is only 15-20% - engineering systems were designed with a view to the future growth of the server fleet in the next 5-7 years.
The main goals of creating a new data center were the centralization of the PhosAgro server complex in one place, as well as the reorganization of the network core. The project was implemented in just four months, and, according to classical technology, with the overhaul of the premises. How did you manage to achieve this, you ask? First of all, thanks to an efficiently working team, timely delivery of equipment, coordinated installation of various subsystems. Well, of course, a competent choice of technical solutions helped. Details are below.
Options Analysis
Before dwelling on the construction of their own data center using classical technology, PhosAgro experts considered other options for solving existing IT problems. One of them is the use of commercial data center services. But after analyzing the offers of several providers, we realized that high operating costs (OPEX) lead to the fact that this option is unprofitable (above TCO) with a large planning horizon. In addition, the concentration of the data center in Moscow and St. Petersburg meant the need for a difficult and expensive transfer of the server complex to one of the capitals - another minus of this option.
Next, the possibility of installing a container data center was considered. However, the benefit was not obvious in view of the technical circumstances and the fact that the existing building had enough resources, and its modernization was not so costly. Such a solution is in demand in cases where there is a shortage of physical space for equipment or if it is assumed that it can be relocated to another place - neither of these is relevant for PhosAgro.
As a result, it was decided to quickly create a data center in a three-story building of the IT service on the territory of PhosAgro.
Vendor selection
The main criterion for creating a data center was fault tolerance, so it was decided to purchase all the basic engineering equipment from one vendor. The mono-vendor solution also provides 100% compatibility of all components of the engineering infrastructure, and reduces the cost of operation and maintenance. It eliminates disputes between contractors about the causes of malfunctions and contributes to their prompt resolution. Another advantage of a single vendor data center is a single service contract. It is easier to respond to changes and deviations from the norm - the vendor identifies the reasons, he makes a decision, for example, to replace a failed component. No need to look on whose side the problem is.
Thus, PhosAgro considered only mono-vendor solutions and almost immediately the offer of Schneider Electric was chosen. The quality and accumulated positive operating experience of Schneider Electric solutions have contributed to this decision. Schneider Electric equipment is successfully operated at the PhosAgro site in Kirovsk, and the staff has specialists trained by this company.
At the next stage, a tender was held in which seven large integrators with high Schneider Electric partner status and experience in implementing similar projects participated. The IBS Platformix company won, offering the best price and terms of the project.
In a short time
The project was required to be implemented in a short time, since the state of engineering systems in the existing server rooms no longer made it possible to provide the required reliability indicators. Thanks to the coordinated work of a team of customer, manufacturer and integrator specialists, prompt delivery and installation of equipment, the data center was built and put into operation in a record short time. The results of the tender were announced in August 2016, and the main work on the project was completed before the end of winter.
Interestingly, the work of specialists in various fields during the construction of the data center was carried out simultaneously. For example, the delivery dates of chillers were agreed with the terms of construction and installation works - the equipment was installed in its place literally “from the wheels”, without additional movements around the facility. A clear synchronization of work together with representatives of the vendor has reduced the time to complete them.
"Anatomy" of the data center
Despite the prompt creation of a new site, the PhosAgro data center cannot be called typical. It consists of two rooms, one of which is intended for standard IT equipment, the second - for racks where data storage systems and non-standard form factor servers are located. In the first hall, 28 racks were installed (6 racks of 7 kW each, the rest - 3.5 kW each) and insulation of the hot corridor was implemented.
EcoAisle Hot Corridor Air Isolation System
In the second hall, where IT equipment of non-standard dimensions is located, a slightly different scheme is implemented: a cold corridor is also highlighted between two rows of racks, but it is not isolated. The data center areas allow you to place all the equipment available in the Group and leave a reserve for growth.
The design load of the data center for IT equipment is 250 kW, engineering systems consume the same amount. Power is supplied to the data center from a medium-voltage substation with a total capacity of 1 MW (two transformers of 500 kW each). At the site, two UPSs with a capacity of 500 kW each are deployed according to the mutual redundancy scheme and can power not only IT, but also critical elements of the engineering infrastructure, and the modular architecture eliminates the presence of a single point of failure. Installed devices Symmetra PX is a modular uninterruptible power supply system with high efficiency - 95.6%, which allows to reduce the cost of power supply for both the UPS and its conditioning system (for this purpose, a standard freon split system is used).
Symmetra PX Schneider Electric Modular Scalable Three-Phase Uninterruptible Power System
For short-term stand-alone power supply of the IT load, a set of rechargeable batteries has been installed, designed to provide stand-alone power supply for 10-15 minutes - this time is enough to guarantee the diesel engine starts. But while the data center is not loaded at full capacity, the batteries are able to guarantee an autonomy time of more than an hour. DGU is used as a backup source.
Energy efficiency
Another important requirement for the new data center was high energy efficiency. The PhosAgro enterprise in Cherepovets is well supplied with energy, so there was no problem getting the required capacity (although a thorough “cleaning” of the electricity coming from the plant was required). Energy efficiency was important not only because of reduced energy consumption, but rather for the sake of reducing environmental impact. This is all the more symbolic, since 2017 is the year of ecology in Russia. PhosAgro IT executives wanted to prove that they could quickly and efficiently build a reliable and energy-efficient facility.
A lot of factors affect energy efficiency, including, in particular, the efficiency of installed UPS systems. But the main contribution to reducing energy consumption is the active use of free cooling (freecooling). The new Data Center has the innovative Aquaflair chiller system, which uses the free-cooling function from October to April. In this mode, only the circulation pump that pumps the coolant (ethylene glycol) works, while consuming a minimum of electricity.
Schneider Electric Uniflair TRAF Series Chillers with Freecooling
To increase cooling efficiency, the already mentioned EcoAisle hot corridor insulation system and InRow row air conditioners in a compact case (only 300mm wide) are provided. The cooling capacity of such an air conditioner is up to 62 kW with an energy consumption of 1.8 kW. Thus, the efficiency factor of the air conditioner is more than 34 kW of cold per 1 kW of electricity.
InRowtm APC by Schneider Electric
Failure and disaster tolerance
The new data center is built in accordance with the requirements for reliability and fault tolerance for Tier III facilities, which, in particular, allows you to disable the components of the engineering infrastructure for their preventive maintenance and repair without interrupting the operation of the data center.
For some key components, even triple redundancy is provided. For example, three chillers are installed: one of them is in working mode, the second is in standby, and the third is in “standby” mode. Every few hours they change roles. This increases fault tolerance and ensures uniform production of equipment life. Any unit can be taken out of service for maintenance without disrupting the operation of services.
In addition, the PhosAgro IT system construction concept envisaged the creation of two geographically dispersed data centers for disaster tolerance - in Cherepovets and Kirovsk (Murmansk Region). The site in Kirovsk, now a backup data center was built three years ago. It is similar to the new, basic, but it uses refrigeration systems on freon. There are also two rooms in the Kirovsk data center, but they are designed for fewer racks. Currently, PhosAgro IT systems are distributed across these two data centers.
Monitoring and management
The StruxureWare Data Center Expert system is used to monitor engineering systems in the new data center. This is a scalable monitoring system independent of the equipment of a particular manufacturer, which collects data from various sensors and “smart” devices, provides the generation of alarm messages and other functions, forming a unified view of the complex physical infrastructure of the facility.
A centralized management system for all engineering infrastructure facilities provides end-to-end monitoring of all levels of engineering systems and improves the efficiency of the service. StruxureWare not only monitors various environmental parameters, but also stores the information received for subsequent analysis.
One contract, one area of responsibility
Even before the main tender, at the design stage, Schneider Electric experts carried out an examination of the project, made their comments, prepared a specification of the equipment. Representatives of the manufacturer were present at the site at all stages: installation of the hot corridor, electrical and cold supply systems, including the most difficult stage of commissioning and commissioning of equipment.
Schneider Electric quickly completed the installation supervision and tested the equipment. Under the contract, she will serve both sites for three years. The vendor conducts regular inspections of equipment (cooling and power systems) for scheduled preventive repairs. At a predetermined time, a Schneider Electric engineer conducts a comprehensive survey of refrigeration systems twice a year and power systems once a year.
Notifications for sending alarm messages to the Help Desk are configured in StruxureWare, cases are started automatically, and the engineer promptly takes action. All components of engineering systems conduct self-tests on a regular basis, and any errors in their operation are displayed in the monitoring system, which allows you to immediately respond to emergency situations.
In recent years, PhosAgro Group has been investing heavily in the development of IT, automation of production and business. The centralization of IT resources in Cherepovets contributed to the implementation of such projects as a single electronic archive with the exchange of documents between holding sites. Business-critical applications now run on diversity clusters. The sites are connected by a redundant channel of 400 Mbps. User access in the new data center is organized at a speed of 1 Gbit / s, and for other company sites in the five largest cities of Russia - 200 Mbit / s.
It is clear that the processes of "digital transformation" are impossible without modern technologies. And with the help of the platform, which the engineering infrastructure of Schneider Electric will provide for the PhosAgro data center, reliable and efficient operation and sustainable development of IT systems are implemented. The data center allowed providing IT equipment with high-quality uninterrupted power supply, creating the necessary climatic conditions for its operation. Now the data center’s power supply load is only 15-20% - engineering systems were designed with a view to the future growth of the server fleet in the next 5-7 years.
