Dedicated servers under water, literally !? Prospects for fish farming in servers ?!
We all know that water and electronics are a dangerous combination, but is it always so? Are modern technologies capable of changing this view?
In this article, we will consider the possibility, advantages and disadvantages of placing servers in a liquid and discuss possible operational problems. We show how it all can look in practice and really work. And also discuss the question of why fish can or cannot fish in servers :)
For a long time, energy losses and cooling costs during the operation of the servers did not give rest to many, including us, since the number of servers used by our subscribers is constantly growing, we are increasingly thinking about creating our own data center (DPC) in the foreseeable future. And when more than half of the energy consumed by the entire data center is spent on air cooling, due to which it is possible to remove the generated heat from the equipment with no more than 1.7 efficiency factor, you are not free to wonder, but how can you improve the cooling efficiency and minimize energy losses?
From the course of physics it is known that air is an extremely inefficient heat conductor, since its thermal conductivity is 25 times lower than the thermal conductivity of water. It is more suitable for thermal insulation, rather than for heat removal. He also has a very small heat capacity, which means that he constantly needs to be mixed intensively and supplied with large volumes for cooling. Another thing is water and liquids. They are used in the cooling systems of the data centers in the form of a heat exchanger in order to increase the overall efficiency coefficient, but they don’t contact the servers directly, only through the air gap and / or the radiator (for example, to cool the chipset), which allows to increase the mechanical coefficient of efficiency of the cooling system ( mPUE) up to 1.2 or even up to 1.15 when using the external environment for cooling purposes.
But how to cool the server most efficiently? There is only one way out - to place it completely in a liquid (of course, a dielectric), preferably with the highest possible thermal conductivity and heat capacity, which will not have a negative effect on the server components. And such a dielectric may be mineral oil. The idea, alas, and fortunately, was not new - several companies have been developing and implementing it for several years in various variations and with different efficiencies. Modern technologies allow you to build an "underwater" Data Center! But what are the advantages and disadvantages of this solution?
Liquid cooling already saves up to 95 percent of the energy that is usually used for cooling in Data Centers and, as a result, up to 50% of all the energy that Data Center consumes.
The liquid cooling system saves up to 60% in the construction of the Data Center, since there is no need to purchase expensive chillers, HVAC (heating ventilation air cooling) systems, build cold / hot corridors, use a raised floor, etc.
SSD disks can be immersed in the coolant, of course, while maintaining operability :), without any modifications, as in the rest of the other standard server components, with the exception of hard drives. Hard drives will require the use of additional devices, because they will not be able to effectively rotate in liquids.
Since the coolant is a dielectric (does not conduct electricity) - there is no need to dry the servers and drain the entire system to work in a cabinet or with a specific server. Nevertheless, this liquid should be non-toxic, odorless (with minimal evaporation) and not be aggressive towards the server components, for example, do not dissolve the rubber insulation of the wires, etc. Choosing the right and effective mineral oil is not an easy task. Not every mineral oil is suitable for the task of cooling in a liquid. And depending on the selected oil, we will get different permissible power of the equipment in a 42-unit cabinet, the heat from which the system can remove.
If we talk about the cooling efficiency in the liquid as a whole, then the system allows you to achieve PUE 1.03. But how is this possible, you ask, if the use of mineral oil for cooling saves only 95% of energy? How can we get an additional efficiency of 2%?
The answer here is simple, liquid cooling allows you to save the energy that the servers consume, due to the fact that they no longer need to install coolers for cooling, and also because the leakage of currents from the chips is reduced, since they are reliably isolated and work at a constant temperature (temperature changes contribute to leakage of currents). And as a result, we save on the cooling system, since it can now occupy a smaller volume, because it needs to remove less heat. This gives a gain of those cherished 2 percent on cooling, but we get not only that. The servers themselves begin to consume 10-20 percent less energy than servers with other cooling. The PUE of the entire Data Center is growing.
Mineral oil is able to effectively protect against corrosion and dust, due to the fact that, unlike air, does not contain water and oxygen, to extend the life of the equipment. It is non-toxic and odorless, which means it practically does not evaporate. But it is of varying effectiveness and the selection of the right mineral oil is a real art.
Various companies have been closely involved in these issues for a long time and have various successes due to the use of different mineral oils, and create their own know-how. About a month ago, Intel and SGI announced the “news” that they are able, thanks to the use of mineral oil and their own developed cooling system based on it, which provides for the operation of servers in liquids, to provide heat removal from the cabinet of several tens of kilowatts and even more. But they still have problems, in particular, in their message it is mentioned that ordinary optical cables in their mineral oil are unlikely to work, for which reason, alas, it is not indicated, apparently the oil is aggressive for them. The solution is far from commercial operation.
Another company, GRC, has been using a much more effective mineral oil for a long time, offers a ready-made commercial solution and does not have similar problems, has not published it as “news” for a long time, and according to them they are able to remove heat from the cabinet up to 100 kW and more, which means significantly surpassed the success of Intel! So you need to be more critical of all the information from the news. If one company claims “know-how”, it doesn’t mean at all that the other one hasn’t come up with a better one, some may just be at the beginning of their journey in a new direction for them :) As mentioned above, Intel is still very far from commercial operation this decision, but without it, ultimately, the decision of any company will not do.
On today's motherboards, circuits are laid out at a “huge” distance from each other to maximize heat dissipation for use as an air cooler, which is an awfully inefficient cooler. Thanks to the cooling in the liquid, it is possible to start the production of servers with more tightly packed circuits that take into account the work in the liquid and the property of heat dissipation by the liquid, because the liquid has not only higher thermal conductivity than air, but much higher heat capacity. The most effective mineral oils to date have a heat capacity that exceeds the heat capacity of air by more than 1200 times!
This all allows not only to remove heat much more efficiently, but also in case of stopping the cooling system, to get much more time to repair it before the work transitions to a critical state due to temperature increase, since the properties of the liquid (large heat capacity and density) allow it to absorb much more heat, while the liquid does not become overheated by itself, thereby pushing the threshold of "critical overheating" in time.
Very great prospects are also opening up for supercomputers operating in liquids, saving energy and space during the operation of high-performance equipment is huge.
Most likely in the future there will be no computing and server equipment that could work without immersion in liquid. The advantages are huge, there are practically no shortcomings, except that the cabinets now need to be placed not vertically, but horizontally, which is somewhat unusual. Due to this, it is possible to increase the "density" of equipment in the Data Center, as well as provide an additional level of security. If the Data Center is suddenly flooded with water as a result of a natural disaster - the water will not affect the servers, as they are already immersed in liquid, albeit with a different density, but at the same time reliably sealed in cabinets.
- there is no need to purchase expensive chillers, HVAC (heating ventilation air cooling) systems;
- there is no need for the construction of cold / hot corridors, the use of a raised floor;
- the number of generators, uninterruptible power supply (UPS) batteries is reduced by N units of equipment, due to reduced consumption - - energy by this equipment when working in liquid;
- the cost of infrastructure per Watt is 73% lower than when building an air-cooled data center, and 55% if the data center uses an external environment for cooling;
- The cost of infrastructure per server gives a gain of 86 and 70 percent, respectively.
- the equipment, being in the liquid, consumes 10-20% less energy, depending on the type, due to the lack of coolers and current losses from the chips, due to their being in the dielectic and ensuring their constant temperature;
- 90-95% of the energy is saved due to the cooling of the servers in the liquid and the absence of large-sized cooling systems in the data center, since now the heat from the cabinet with the servers in mineral oil can be effectively removed using an evaporative cooling tower (no mechanics, only evaporation of water) or using the circuit with cold water;
- there are no expenses associated with the amortization of conventional cooling systems, the cost of energy supply systems are significantly reduced in terms of N units, due to the fact that you need to contain fewer UPS batteries including;
mineral oil is almost forever, it does not need to be changed and almost no need to be added (except in cases of leakage), unlike other coolers in data centers;
- if the average server consumes about 230-270 watts of power and 50-170 watts of cooling, depending on the cooling method used, then the use of cooling in a liquid reduces the average energy consumption of the server to 210 watts, and the energy required to cool it is about 10 Watt!
It is possible to divert over 100 kW of heat from a 42 unit cabinet immersed in mineral oil! And also significantly reduce the cost of server hardware, up to 50% for various components, and all because now the constant operating temperature is about 20 degrees lower than in air, there is the possibility to use even desktop components without fear, since they work when much lower temperatures.
Of course, this idea has not and will not receive such widespread application in the personal computer market, simply because most have long switched to laptops and other gadgets, tower workstations are often used only by professionals, since they need great performance and removal of a large amount of heat. So for them, immersion of their priceless iron in a liquid can be very useful!
It turns out to realize this at home is not difficult and possible, moreover, it has been done by many fans of modding for quite some time. Some companies even offer a turnkey solution., based on the Crystal Plus 70T mineral oil, which is commercially available and is ideal for this task according to the experimenters, has a heat capacity of 750 times higher than that of air and a lower density than water.
Mixing the liquid can be carried out by passing air through mineral oil or even an ordinary computer cooler, which rotates by itself much slower in mineral oil than in air, but retains its efficiency. To the question of what to do with water vapor that will get into the mineral oil when air is passed for mixing, the developers answer that, due to different densities (higher at the water), water will accumulate at the very bottom of the “aquarium”, where it’s not located any electrical components, but they have not yet seen that during the long operation at least a scanty amount of water appears, otherwise the tank would begin to resemble a “lava lamp”.
I dare to admit that watching various FISH under water is an unforgettable pleasure. I’m a diver and wouldn’t refuse a workstation at home in the form of an aquarium with tropical or not very tropical fish.
But alas, genetic engineering is still behind the fashion in IT, the fish die, if they are placed in mineral oil, its properties are extremely far from the water so needed by the fish. Let's hope that this idea, either through the use of new technologies, or through genetic engineering, will be possible for someone to implement in the future, but for now you can use robotic fish!
I will be the first to order a robotic boxfish!
Well, of course, do not forget to support the construction of our future underwater Data Center (whose ideas I have no doubt) - order dedicated servers with us, so far in the Netherlands and the USA and from $ 49, and in the future on the Barrier Reef!
In this article, we will consider the possibility, advantages and disadvantages of placing servers in a liquid and discuss possible operational problems. We show how it all can look in practice and really work. And also discuss the question of why fish can or cannot fish in servers :)
For a long time, energy losses and cooling costs during the operation of the servers did not give rest to many, including us, since the number of servers used by our subscribers is constantly growing, we are increasingly thinking about creating our own data center (DPC) in the foreseeable future. And when more than half of the energy consumed by the entire data center is spent on air cooling, due to which it is possible to remove the generated heat from the equipment with no more than 1.7 efficiency factor, you are not free to wonder, but how can you improve the cooling efficiency and minimize energy losses?
From the course of physics it is known that air is an extremely inefficient heat conductor, since its thermal conductivity is 25 times lower than the thermal conductivity of water. It is more suitable for thermal insulation, rather than for heat removal. He also has a very small heat capacity, which means that he constantly needs to be mixed intensively and supplied with large volumes for cooling. Another thing is water and liquids. They are used in the cooling systems of the data centers in the form of a heat exchanger in order to increase the overall efficiency coefficient, but they don’t contact the servers directly, only through the air gap and / or the radiator (for example, to cool the chipset), which allows to increase the mechanical coefficient of efficiency of the cooling system ( mPUE) up to 1.2 or even up to 1.15 when using the external environment for cooling purposes.
But how to cool the server most efficiently? There is only one way out - to place it completely in a liquid (of course, a dielectric), preferably with the highest possible thermal conductivity and heat capacity, which will not have a negative effect on the server components. And such a dielectric may be mineral oil. The idea, alas, and fortunately, was not new - several companies have been developing and implementing it for several years in various variations and with different efficiencies. Modern technologies allow you to build an "underwater" Data Center! But what are the advantages and disadvantages of this solution?
Advantages and disadvantages of placing servers in a liquid
Liquid cooling already saves up to 95 percent of the energy that is usually used for cooling in Data Centers and, as a result, up to 50% of all the energy that Data Center consumes.
The liquid cooling system saves up to 60% in the construction of the Data Center, since there is no need to purchase expensive chillers, HVAC (heating ventilation air cooling) systems, build cold / hot corridors, use a raised floor, etc.
SSD disks can be immersed in the coolant, of course, while maintaining operability :), without any modifications, as in the rest of the other standard server components, with the exception of hard drives. Hard drives will require the use of additional devices, because they will not be able to effectively rotate in liquids.
Since the coolant is a dielectric (does not conduct electricity) - there is no need to dry the servers and drain the entire system to work in a cabinet or with a specific server. Nevertheless, this liquid should be non-toxic, odorless (with minimal evaporation) and not be aggressive towards the server components, for example, do not dissolve the rubber insulation of the wires, etc. Choosing the right and effective mineral oil is not an easy task. Not every mineral oil is suitable for the task of cooling in a liquid. And depending on the selected oil, we will get different permissible power of the equipment in a 42-unit cabinet, the heat from which the system can remove.
If we talk about the cooling efficiency in the liquid as a whole, then the system allows you to achieve PUE 1.03. But how is this possible, you ask, if the use of mineral oil for cooling saves only 95% of energy? How can we get an additional efficiency of 2%?
The answer here is simple, liquid cooling allows you to save the energy that the servers consume, due to the fact that they no longer need to install coolers for cooling, and also because the leakage of currents from the chips is reduced, since they are reliably isolated and work at a constant temperature (temperature changes contribute to leakage of currents). And as a result, we save on the cooling system, since it can now occupy a smaller volume, because it needs to remove less heat. This gives a gain of those cherished 2 percent on cooling, but we get not only that. The servers themselves begin to consume 10-20 percent less energy than servers with other cooling. The PUE of the entire Data Center is growing.
Successes of various companies in the field of server cooling in liquid
Mineral oil is able to effectively protect against corrosion and dust, due to the fact that, unlike air, does not contain water and oxygen, to extend the life of the equipment. It is non-toxic and odorless, which means it practically does not evaporate. But it is of varying effectiveness and the selection of the right mineral oil is a real art.
Various companies have been closely involved in these issues for a long time and have various successes due to the use of different mineral oils, and create their own know-how. About a month ago, Intel and SGI announced the “news” that they are able, thanks to the use of mineral oil and their own developed cooling system based on it, which provides for the operation of servers in liquids, to provide heat removal from the cabinet of several tens of kilowatts and even more. But they still have problems, in particular, in their message it is mentioned that ordinary optical cables in their mineral oil are unlikely to work, for which reason, alas, it is not indicated, apparently the oil is aggressive for them. The solution is far from commercial operation.
Another company, GRC, has been using a much more effective mineral oil for a long time, offers a ready-made commercial solution and does not have similar problems, has not published it as “news” for a long time, and according to them they are able to remove heat from the cabinet up to 100 kW and more, which means significantly surpassed the success of Intel! So you need to be more critical of all the information from the news. If one company claims “know-how”, it doesn’t mean at all that the other one hasn’t come up with a better one, some may just be at the beginning of their journey in a new direction for them :) As mentioned above, Intel is still very far from commercial operation this decision, but without it, ultimately, the decision of any company will not do.
Prospects
On today's motherboards, circuits are laid out at a “huge” distance from each other to maximize heat dissipation for use as an air cooler, which is an awfully inefficient cooler. Thanks to the cooling in the liquid, it is possible to start the production of servers with more tightly packed circuits that take into account the work in the liquid and the property of heat dissipation by the liquid, because the liquid has not only higher thermal conductivity than air, but much higher heat capacity. The most effective mineral oils to date have a heat capacity that exceeds the heat capacity of air by more than 1200 times!
This all allows not only to remove heat much more efficiently, but also in case of stopping the cooling system, to get much more time to repair it before the work transitions to a critical state due to temperature increase, since the properties of the liquid (large heat capacity and density) allow it to absorb much more heat, while the liquid does not become overheated by itself, thereby pushing the threshold of "critical overheating" in time.
Very great prospects are also opening up for supercomputers operating in liquids, saving energy and space during the operation of high-performance equipment is huge.
Most likely in the future there will be no computing and server equipment that could work without immersion in liquid. The advantages are huge, there are practically no shortcomings, except that the cabinets now need to be placed not vertically, but horizontally, which is somewhat unusual. Due to this, it is possible to increase the "density" of equipment in the Data Center, as well as provide an additional level of security. If the Data Center is suddenly flooded with water as a result of a natural disaster - the water will not affect the servers, as they are already immersed in liquid, albeit with a different density, but at the same time reliably sealed in cabinets.
Liquid cooling in numbers
Saving over 60% of funds during construction:
- there is no need to purchase expensive chillers, HVAC (heating ventilation air cooling) systems;
- there is no need for the construction of cold / hot corridors, the use of a raised floor;
- the number of generators, uninterruptible power supply (UPS) batteries is reduced by N units of equipment, due to reduced consumption - - energy by this equipment when working in liquid;
- the cost of infrastructure per Watt is 73% lower than when building an air-cooled data center, and 55% if the data center uses an external environment for cooling;
- The cost of infrastructure per server gives a gain of 86 and 70 percent, respectively.
Savings in excess of 50% during operation:
- the equipment, being in the liquid, consumes 10-20% less energy, depending on the type, due to the lack of coolers and current losses from the chips, due to their being in the dielectic and ensuring their constant temperature;
- 90-95% of the energy is saved due to the cooling of the servers in the liquid and the absence of large-sized cooling systems in the data center, since now the heat from the cabinet with the servers in mineral oil can be effectively removed using an evaporative cooling tower (no mechanics, only evaporation of water) or using the circuit with cold water;
- there are no expenses associated with the amortization of conventional cooling systems, the cost of energy supply systems are significantly reduced in terms of N units, due to the fact that you need to contain fewer UPS batteries including;
mineral oil is almost forever, it does not need to be changed and almost no need to be added (except in cases of leakage), unlike other coolers in data centers;
- if the average server consumes about 230-270 watts of power and 50-170 watts of cooling, depending on the cooling method used, then the use of cooling in a liquid reduces the average energy consumption of the server to 210 watts, and the energy required to cool it is about 10 Watt!
It is possible to divert over 100 kW of heat from a 42 unit cabinet immersed in mineral oil! And also significantly reduce the cost of server hardware, up to 50% for various components, and all because now the constant operating temperature is about 20 degrees lower than in air, there is the possibility to use even desktop components without fear, since they work when much lower temperatures.
From “underwater” servers to liquid-cooled PCs, or how to create a workstation in liquid at home
Of course, this idea has not and will not receive such widespread application in the personal computer market, simply because most have long switched to laptops and other gadgets, tower workstations are often used only by professionals, since they need great performance and removal of a large amount of heat. So for them, immersion of their priceless iron in a liquid can be very useful!
It turns out to realize this at home is not difficult and possible, moreover, it has been done by many fans of modding for quite some time. Some companies even offer a turnkey solution., based on the Crystal Plus 70T mineral oil, which is commercially available and is ideal for this task according to the experimenters, has a heat capacity of 750 times higher than that of air and a lower density than water.
Mixing the liquid can be carried out by passing air through mineral oil or even an ordinary computer cooler, which rotates by itself much slower in mineral oil than in air, but retains its efficiency. To the question of what to do with water vapor that will get into the mineral oil when air is passed for mixing, the developers answer that, due to different densities (higher at the water), water will accumulate at the very bottom of the “aquarium”, where it’s not located any electrical components, but they have not yet seen that during the long operation at least a scanty amount of water appears, otherwise the tank would begin to resemble a “lava lamp”.
I want the fish to swim in the servers! Throw fish there!
I dare to admit that watching various FISH under water is an unforgettable pleasure. I’m a diver and wouldn’t refuse a workstation at home in the form of an aquarium with tropical or not very tropical fish.
But alas, genetic engineering is still behind the fashion in IT, the fish die, if they are placed in mineral oil, its properties are extremely far from the water so needed by the fish. Let's hope that this idea, either through the use of new technologies, or through genetic engineering, will be possible for someone to implement in the future, but for now you can use robotic fish!
I will be the first to order a robotic boxfish!
Well, of course, do not forget to support the construction of our future underwater Data Center (whose ideas I have no doubt) - order dedicated servers with us, so far in the Netherlands and the USA and from $ 49, and in the future on the Barrier Reef!
