Immersion Cooling, Underwater Servers: Immersion-2 for 3M ™ Novec ™ Delivers Magical Results, Practice in Hong Kong
Immersion cooling - immersion cooling. In a previous article on immersion cooling , the benefits of hosting servers in mineral oil were essentially considered. I was deeply mistaken when it seemed to me that this was the most effective solution globally. Having got acquainted with some other developments, I realized that the basic principle itself is effective - immersion cooling, but everything else, as they say “depends on” ...
Today I want to tell you about the relatively new two-phase cooling system Immersion-2, which uses all the advantages of “dry water »3M Novec to achieve truly magical results, turning flaws into virtues!
What is 3M Novec we will not repeat, we have already written about this in RuNet many times, and even on Habréan excellent article , but how it can be used to cool servers in practice and why “dry water” is more effective than mineral oil, liquid systems for cooling electronics in closed circuits, we will now consider.
A system for cooling electronics in liquids in closed circuits has existed and has been used for several decades, for example, in the military and aviation sectors. It is problematic and expensive in production and operation, since pressure can increase over time in closed systems. Among other things, these systems are aimed at smaller parts of hardware with a lower power density, which leads to duplication of components in use.
For example, 20 motherboards will require 20 enclosed cases, 20 heat exchangers, 40 connectors for fluid circulation (20 incoming and 20 outgoing), 20 power connectors, 20 monitoring devices, etc. All of these components will be very difficult to maintain. In an open approach, thanks to 3M Novec, they use one immersion cooling half-open bath with 2-phase cooling, where all the equipment can be placed in one housing as a whole.
Due to the fact that 3M Novec begins to boil at atmospheric pressure already at a temperature of 34 ° C, 49 ° C or 56 ° C (the boiling point varies depending on the type used), it undergoes a phase transition from a liquid to a gaseous state. This property can be used to cool equipment without using hydraulic pumps, as in the case of mineral oil cooling, since there is no need to pump liquid inside the tank, 3M Novec boils and mixes, removes excess heat due to physical processes of evaporation and condensation, two phase transitions, due to which system design is greatly simplified.
For a better understanding, I will give the stages of 2-phase immersion cooling:
Like immersion cooling, “sealed servers” use a single-phase cooling process. This is much less efficient compared to a 2-phase system (heat transfer coefficient is much higher) and requires more components. High density, for example, in passive 2-phase systems, is simply not possible, because the mixing process goes well without the use of pumps. Technically sealed servers use a dielectric fluid, such as mineral oil or a fluid similar to that used in passive 2-phase immersion cooling. But the process of heat removal does not occur due to evaporation and condensation, the main process is mixing with the help of pumps, which in any case only increases energy consumption and reduces cooling efficiency.
In general, it is worth noting that the idea of Immersion-2 is essentially not new. Back in the 1970s, relevant studies were conducted using a C6F14 fluid, which was boiling on the surface of a silicon chip. However, the chip itself was far from perfect and the fluid had a negative effect on it. In short, computer and chip technology has matured only in the last few decades. What we could not do 20 years ago is now possible thanks to a modern package of microcircuits that include heat distributors with high-performance interfaces, ssd disks in which there are no rotating components and sealed HDD disks, which can be used in a liquid environment. Another factor that influenced the fact that the technology has not been previously investigated,
The Allied Control company seriously took up the development of this idea, put it into practice, and here’s what came of it:
As a test project in Hong Kong, a mini data center was built in an office building on a small area, which became the most energy-efficient data center not only in Asia, but perhaps in the whole world, since the PUE energy efficiency coefficient of 1.01 was achieved!
According to Allied Control, the maximum power per rack when using their cooling system can be up to 225 kW. For test purposes, a cooling system with power up to 500 kW was installed for 24 racks. This office mini data center is quieter than the library!
The funny thing is that the world's first most energy-efficient data center in history was not used for scientific work, but for mining the BitCoin cryptocurrency, the realities of the modern world, but innovation needs to be paid back somehow! An additional area has even been reserved for the growth of the mining farm:
And this is how the “blades” look, which can be up to 92 in each unit:
The liquid cooling system itself is reserved:
Of course, as are the communication channels and power supply.
- 4000 times more efficient than air cooling (electronic components are immersed in a dielectric fluid, which has much greater thermal conductivity and heat capacity than air);
- energy efficiency 1.01 (the ability to remove from 18 to 225 kW of heat from the rack);
- “cool hundreds of kilowatts” using only 1,500 watts (energy savings for the current site of 24 cabinets filled with equipment is more than $ 60,000 / month);
- low boiling point promotes mixing and passive cooling due to evaporation, eliminates the need for pumps (fluid with a boiling point of 34 ° C (93 ° F) under the name 3M Novec 7000 is used);
- higher density (the area occupied by equipment is reduced many times, thereby allowing you to save data center on all other engineering systems);
- improved performance and reliability (there are no fluctuations in temperature and hot spots, there are no unnecessary details, since the cooling system is passive, it is possible to get rid of duplication of hardware, heat transfer occurs only once);
- incredible cleanliness (two-phase immersion cooling system has useful side effects - massive air flow, dust and noise are excluded from the object, something immersed in the fluid not only does not need to be cleaned and dried during extraction, but it even becomes cleaner due to evaporation fluid from the surface);
- an extremely high degree of environmental friendliness and fire safety (does not contain substances that destroy ozone, moreover, it has the property of displacing oxygen, thereby preventing ignition processes, which is actively used in gas fire extinguishing systems based on 3M Novec);
- long life (over 30 years) and various boiling points of the coolant (34 ° C, 49 °, 56 ° C, etc., mainly Novec 7000 and Novec 649 with a temperature of 34 and 49 degrees, respectively);
- safety (when working due to low boiling points it is impossible to get a burn, and due to the absence of toxicity - poisoning);
- not patented (the possibility of universal design for a specific project);
- a ready-made modular solution for the customer with a quick commissioning time (up to 6 months).
The last paragraph will dwell in more detail. The serial solution is called DataTank and provides the ability to remove up to 1.4 MW of heat from equipment located in one container:
DataTank modules are third-generation systems and can operate efficiently even in hot and humid climates of Asia, while providing PUE 1.01 or even less. Multiple containers can be combined to deploy a large-scale container farm.
In general, we can conclude that the future lies with immersion cooling, although of course I would like such innovations to be primarily used not only for BitCoin mining, but also for scientific and industrial purposes!
Today I want to tell you about the relatively new two-phase cooling system Immersion-2, which uses all the advantages of “dry water »3M Novec to achieve truly magical results, turning flaws into virtues!
What is 3M Novec we will not repeat, we have already written about this in RuNet many times, and even on Habréan excellent article , but how it can be used to cool servers in practice and why “dry water” is more effective than mineral oil, liquid systems for cooling electronics in closed circuits, we will now consider.
A system for cooling electronics in liquids in closed circuits has existed and has been used for several decades, for example, in the military and aviation sectors. It is problematic and expensive in production and operation, since pressure can increase over time in closed systems. Among other things, these systems are aimed at smaller parts of hardware with a lower power density, which leads to duplication of components in use.
For example, 20 motherboards will require 20 enclosed cases, 20 heat exchangers, 40 connectors for fluid circulation (20 incoming and 20 outgoing), 20 power connectors, 20 monitoring devices, etc. All of these components will be very difficult to maintain. In an open approach, thanks to 3M Novec, they use one immersion cooling half-open bath with 2-phase cooling, where all the equipment can be placed in one housing as a whole.
Due to the fact that 3M Novec begins to boil at atmospheric pressure already at a temperature of 34 ° C, 49 ° C or 56 ° C (the boiling point varies depending on the type used), it undergoes a phase transition from a liquid to a gaseous state. This property can be used to cool equipment without using hydraulic pumps, as in the case of mineral oil cooling, since there is no need to pump liquid inside the tank, 3M Novec boils and mixes, removes excess heat due to physical processes of evaporation and condensation, two phase transitions, due to which system design is greatly simplified.
For a better understanding, I will give the stages of 2-phase immersion cooling:
Like immersion cooling, “sealed servers” use a single-phase cooling process. This is much less efficient compared to a 2-phase system (heat transfer coefficient is much higher) and requires more components. High density, for example, in passive 2-phase systems, is simply not possible, because the mixing process goes well without the use of pumps. Technically sealed servers use a dielectric fluid, such as mineral oil or a fluid similar to that used in passive 2-phase immersion cooling. But the process of heat removal does not occur due to evaporation and condensation, the main process is mixing with the help of pumps, which in any case only increases energy consumption and reduces cooling efficiency.
In general, it is worth noting that the idea of Immersion-2 is essentially not new. Back in the 1970s, relevant studies were conducted using a C6F14 fluid, which was boiling on the surface of a silicon chip. However, the chip itself was far from perfect and the fluid had a negative effect on it. In short, computer and chip technology has matured only in the last few decades. What we could not do 20 years ago is now possible thanks to a modern package of microcircuits that include heat distributors with high-performance interfaces, ssd disks in which there are no rotating components and sealed HDD disks, which can be used in a liquid environment. Another factor that influenced the fact that the technology has not been previously investigated,
The Allied Control company seriously took up the development of this idea, put it into practice, and here’s what came of it:
As a test project in Hong Kong, a mini data center was built in an office building on a small area, which became the most energy-efficient data center not only in Asia, but perhaps in the whole world, since the PUE energy efficiency coefficient of 1.01 was achieved!
According to Allied Control, the maximum power per rack when using their cooling system can be up to 225 kW. For test purposes, a cooling system with power up to 500 kW was installed for 24 racks. This office mini data center is quieter than the library!
The funny thing is that the world's first most energy-efficient data center in history was not used for scientific work, but for mining the BitCoin cryptocurrency, the realities of the modern world, but innovation needs to be paid back somehow! An additional area has even been reserved for the growth of the mining farm:
And this is how the “blades” look, which can be up to 92 in each unit:
The liquid cooling system itself is reserved:
Of course, as are the communication channels and power supply.
Benefits
- 4000 times more efficient than air cooling (electronic components are immersed in a dielectric fluid, which has much greater thermal conductivity and heat capacity than air);
- energy efficiency 1.01 (the ability to remove from 18 to 225 kW of heat from the rack);
- “cool hundreds of kilowatts” using only 1,500 watts (energy savings for the current site of 24 cabinets filled with equipment is more than $ 60,000 / month);
- low boiling point promotes mixing and passive cooling due to evaporation, eliminates the need for pumps (fluid with a boiling point of 34 ° C (93 ° F) under the name 3M Novec 7000 is used);
- higher density (the area occupied by equipment is reduced many times, thereby allowing you to save data center on all other engineering systems);
- improved performance and reliability (there are no fluctuations in temperature and hot spots, there are no unnecessary details, since the cooling system is passive, it is possible to get rid of duplication of hardware, heat transfer occurs only once);
- incredible cleanliness (two-phase immersion cooling system has useful side effects - massive air flow, dust and noise are excluded from the object, something immersed in the fluid not only does not need to be cleaned and dried during extraction, but it even becomes cleaner due to evaporation fluid from the surface);
- an extremely high degree of environmental friendliness and fire safety (does not contain substances that destroy ozone, moreover, it has the property of displacing oxygen, thereby preventing ignition processes, which is actively used in gas fire extinguishing systems based on 3M Novec);
- long life (over 30 years) and various boiling points of the coolant (34 ° C, 49 °, 56 ° C, etc., mainly Novec 7000 and Novec 649 with a temperature of 34 and 49 degrees, respectively);
- safety (when working due to low boiling points it is impossible to get a burn, and due to the absence of toxicity - poisoning);
- not patented (the possibility of universal design for a specific project);
- a ready-made modular solution for the customer with a quick commissioning time (up to 6 months).
The last paragraph will dwell in more detail. The serial solution is called DataTank and provides the ability to remove up to 1.4 MW of heat from equipment located in one container:
DataTank modules are third-generation systems and can operate efficiently even in hot and humid climates of Asia, while providing PUE 1.01 or even less. Multiple containers can be combined to deploy a large-scale container farm.
In general, we can conclude that the future lies with immersion cooling, although of course I would like such innovations to be primarily used not only for BitCoin mining, but also for scientific and industrial purposes!