High power immersion cooling

Modern electronic components are working faster every year. Growing speeds, increasing consumption and heat dissipation. Current trends in immersion cooling of processors and video cards are increasingly entering our lives.

There are many offers of immersion cooling systems on the market, but when they first met, their fundamental differences are not so easy to determine. We conducted a comparison of technologies empirically and revealed their disadvantages and advantages.

Overheating equipment as a scourge of modern electronics

Everyone knows that modern electronics work with electrical power. In this device, there is either a battery or it needs to be plugged in. And they are all united by one more common feature - they heat up. For example, modern phones actively generate heat when performing demanding tasks: games, high quality video recording, etc., and gamers know that large and efficient coolers are necessary for the smooth operation of their powerful computers.

Electric current from the power source passes through the chip, consisting mainly of semiconductors of complex structure. A semiconductor is a kind of material that partially conducts electrical current, and partly does not. Its conductivity depends on voltage, temperature and other conditions.

If you take several different semiconductors and arrange them in three layers, you can achieve an unexpected result. If you apply voltage to the 1st and 3rd layer, the current through this “sandwich” does not leak. And if we start a very small current on the 2nd layer, then between the 1st and 3rd layer the current starts to flow almost unchecked.

A device operating according to the specified principle is called a transistor. Of course, its structure, of course, is more complex, but the rule remains the same - control of the flow of current through the control gate. This effect can be compared with a faucet.
Particular attention in the operation of the transistor is given to the process of transition from the closed state (current does not flow) to the open state (current flows freely). Common sense suggests that the transition from one state to another can not be instantaneous, and takes, though very short, but still not a zero time interval. It is at the moment of switching between these states that the current passes badly, which causes the transistor to heat up.

Modern processors operate at frequencies up to 4 GHz, which means that the transistors in the processor make 4,000,000,000 switchings per second! And each such switching causes heating of the device.

For this reason, when overclocking the processor (overclocking), the heating process is particularly pronounced.

To remove heat to the surface of the processor, a radiator with a fan is used. The fan blows the radiator fins with cold air and removes the heat generated by the processor. This approach is the easiest to use, which is why it has become widespread.

The development of electronics has led to the fact that every year the speed of processors and the number of transistors rapidly increased, and the size of the processor invariably remained at the same level. Compare the Intel 486 processor at 33 MHz and the modern Intel I7 at 3.8 GHz. The size is the same, the speed is much higher, and, therefore, higher power consumption and heat release.

It is necessary to note the fact that for the transistor to work correctly, its temperature must remain low, otherwise it begins to conduct electric current even when it is not required from it. It turns out that the faster the processor, the more it heats up, and the higher the chance that the transistors inside it will not work correctly. This effect is observed, for example, when overclocking and is expressed in the form of the famous “blue screen of death”. When the processor detects a failure in its own work, the OS stops its operation, and the user is shown a blue screen with information about the current state. If you continue to operate in this mode, there is a high probability that at least one of several billion transistors will break. This will lead to regular failures and the inability to use such a processor in the future.

That is why it is so important to use good cooling systems and operate the electronics in a given temperature mode. The pursuit of speed can lead first to random hangs, and then to permanent ones, with a further breakdown of the processor.

This principle applies, first of all, to modern CPUs - and especially GPUs. Because of the difference in the architecture of these two computing devices, the heating of the GPU is more powerful - simply because during operation almost all the transistors inside are used. The average power of the top-end CPU is 90 watts, and the GPU is 200 watts. Therefore, the radiators of modern video cards are much larger than the radiators of central processing units.

When cooling large computing power, additional difficulties arise. The power of server hardware located on one square meter is extremely high, and amounts to tens of kW. In addition, it is necessary to maintain a constant microclimate, without fluctuations in temperature and humidity. Consider carefully the definition of the word "humidity": the concentration of water molecules per unit volume of air; under certain circumstances, moisture can condense and turn into water, which conducts electrical current very well - which is very dangerous for electronics. The server also has another enemy - dust, which clogs the radiators and significantly reduces the cooling efficiency.

Traditional and alternative cooling systems

Even in spite of all these difficulties, manufacturers of modern server equipment still continue to use air for heat removal. Almost all modern server are designed for air cooling, with the division into cold and hot corridors. To ensure the climatic conditions throughout the year establish a powerful climate control, which include air conditioning. Such plants themselves consume a lot of electricity and, paradoxically, they themselves emit a lot of heat. And this decision, unfortunately, is widespread.

Alternative air cooling technology is free cooling. The air enters from the outside and blows the server, freely going outside. With this approach, equipment costs are reduced, but this solution is not suitable for hot countries. In addition, the air remains dusty, and its humidity corresponds to the humidity outside, which is accompanied by fluctuations in both humidity and temperature inside the object.

Immersion cooling system

Relatively recently, immersion cooling technology became popular. Developments on this topic have been conducted for a long time, since the technology itself is no longer new, but now its demand is growing at an extraordinary pace.

The word "immersion" means "submersible." This means that all electronics, all server boards, processor, video cards, power supplies and hard drives are completely immersed in liquid. Naturally, this fluid is a dielectric and does not conduct current - otherwise the operation of electronics would be impossible. Upon further analysis of the proposed solutions, it becomes clear that immersion cooling is different, with and without phase transition. These types of cooling differ not only in their physical principle, but also have significant differences in operation.

Thus, mineral oil was used to cool power transformers at substations for a very long time. This substance is characterized by the absence of electrical conductivity and sufficient heat capacity. You can also note its low cost.

3M Novec liquid for two-phase cooling, as opposed to mineral oil, is used relatively recently. She also does not conduct electric current and has a low heat capacity. Surprisingly, the cooling effect with its help is achieved by boiling. For a more detailed analysis of this phenomenon, we need to remember the laws of physics.
Heating of the fluid occurs due to the transfer of energy from a warmer object to a colder one. The amount of energy, or the amount of heat, is measured in joules. One Joule is the equivalent of heating a body with 1 W for one second.

Thus, a video card emits 200 W * 1 s = 200 J of heat, if it worked for only one second. For a minute the card will allocate 200 W * 60 s = 12 kJ of heat. The second question that arises is the temperature. How much will the temperature of a video card change with such heating? The temperature change will depend on the heat capacity of the object that we heat, and its mass. It is quite obvious that a glass of water in a kettle boils much faster than a full kettle.

Imagine that we are trying to heat one liter of water with a single video card. The weight of 1 liter of water is about 1 kg. The heat capacity of water is approximately 3800 J / kg / K. This means that it takes 3800 J of energy to heat water weighing 1 kg per 1 degree Celsius. Let's compare this with the power of our video card and get 12000/3800 = 3.15 degrees Celsius. And this is just a minute! By simple calculations, it can be established that in 10 minutes the water will heat up to 31 ° C. Naturally, this process will not last forever. So, if we neglect the thermal conductivity of materials, the water will heat up to 85–90 degrees, after which the video card will overheat and freeze.

If we modify our experiment and after 10 minutes replace the heated water with the cold one, the heating process will start anew. In this case, the card will not overheat. Of course, changing the water every 10 minutes is uncomfortable, and the thought comes to stretch the pipes through which cold water will flow, and the heated water will flow out. Such liquid cooling systems exist and are sold in many computer stores.

Let's go back to immersion cooling with mineral oil. To do this, in our calculations, we need to change the heat capacity and mass of the substance. The weight of 1 liter of oil is slightly less than a liter of water and amounts to 0.85 kg. The heat capacity is 1800 J / kg / K. So, to heat a liter of oil, 0.85 kg * 1C * 1800 J / kg / K = 1.5 kJ of energy are needed. This means that the video card heats up oil at 12,000 / 1500 = 8 ° C in 1 minute. It is much more than 3.15 ° C. However, this method has a big advantage - it does not need pipes for supplying and discharging fluid to each video card. You can simply put a few video cards in one bath and pour them with mineral oil.

The problem of overheating of the oil itself in our case also does not go anywhere. As soon as the oil reaches the temperature of the video card, it will no longer take heat and the equipment will overheat. We'll have to somehow again serve cold oil and pick up hot.

A simple solution could be used: a large capacity of cold oil and a container for storing an already heated oil. Naturally, the installation of such huge tanks is not economically viable, so we will have to let the oil in a closed loop and cool it outside the immersion bath. This will require additional installation of the radiator from the outside, where it will be blown with colder air. Providing street air blowing is quite natural: there is a lot of cold (compared to the oil temperature) air outside, and hot air is carried away by the wind.

But for how much the radiator will cool 1 m3 of air at the same 8 degrees? After all, having made the calculations, we find that to cool one video card it takes about 1 liter of cold (cooled) oil per minute; while the oil will have time to warm up by 8 degrees. That is, the radiator must be such as to cool the oil with cold air just by 8 degrees.

Another important element that remains outside of our calculations is the pump. Requirements for it are made much easier - to ensure the circulation of 1 liter of oil per minute. Special attention should be paid to oil viscosity. It is clear that because of the lower viscosity, 1 liter of water will pass through the pipes and through the radiator is much easier than 1 liter of oil. That is, we need either a powerful pump or large pipes and a radiator.

Let's now imagine that now you have not one, but at least 100 video cards. This is 20 kW of heat and 12,000,000 Joules of energy per minute. Then, under equal conditions, the pump should pump 100 liters of oil per minute. Imagine what difficulties arise when 1000 video cards are in the system ...

Let us turn to two-phase immersion cooling with Novec liquid. It is often heard that it is expensive, very volatile, evaporates easily, etc. Of course it is, because the principle of its action is completely different. When this liquid is heated above 61 ° C, it evaporates. However, at the same time neither heating nor cooling occurs with it. When warming up the entire volume of the liquid after starting it reaches 61 ° C, the temperature simply does not increase. It seems absurd, however, it is. The process of evaporation (boiling) in itself is very energy-intensive. Perhaps this process can be compared with the sensations of a person who bathes in summer and comes out of the water. It is warm in the water, and it becomes cold in the wind. The reason for this phenomenon is the evaporation of water from the surface of the body.

Similarly, Novec evaporates from the hot surfaces of the microcircuits and takes some of the heat with it. For evaporation of 1 g of Novec, you need about 120 J. This means that a 200 W video card will evaporate about 2 g of liquid in 1 second. And per minute - only 120 g. The efficiency of heat removal due to boiling is extremely high, and the size of a radiator for removal of 200 W of heat can be only 3-4 cm2. In fact, the radiator is almost not needed.
Like mineral oil, this system requires fluid cooling. To do this, you can pour 120 g of liquid from a huge tank every minute. On the other hand, at a fluid cost of about $ 100 per liter, such a tank is a very, very expensive solution. Therefore, it is completely natural to go the other way - to organize a closed cycle with an external cooler.

Condensation of liquid vapors

Evaporation is the transition of a substance from a liquid to a gaseous state. That is, it turns out that Novec after boiling does not disappear without a trace, but becomes a gas that accumulates above the surface of the liquid in the form of a fog. What if this fog is cool? We will install several pipes inside and let cold water over them. The gas will begin to condense on them and form droplets, which will unite in streams of liquid and flow back. Thus, Novec will not go anywhere and will remain inside the system without any losses.

The process of condensation is the complete opposite of evaporation, here everything happens in reverse order. As for energy, it must be taken from the gas in exactly the same volume as during evaporation. Naturally, the cold pipes will heat up, and the heating power will be exactly the one that was released by the video cards when boiling.

To effectively operate a two-phase immersion cooling system, we need to constantly cool the pipes. The simplest and most effective solution is to run water over them, which will be cooled by a radiator located on the street. Compared to mineral oil, it looks much easier. After all, the heat capacity of water is twice as much, and the volume of circulation per minute is less. Plus, the viscosity of the water is significantly lower, and, therefore, the pump can easily pump more water per unit of time. These factors will make it possible to use a smaller radiator on the street, thinner pipes and a less powerful pump - compared to the same mineral oil.

Advantages of the immersion cooling system

Both cooling systems — with both mineral oil and Novec immersion liquid — have an important advantage over air cooling. There is no need for expensive air conditioners that consume electricity. In addition, there is no problem of dust and humidity, as in free-cooling systems.

One of the important parameters of data centers is the coefficient PUE (energy efficiency), equal to the ratio of the total power consumption of the data center to the power consumption of computing devices. For immersion systems, this ratio is close to 1, for air systems with air conditioning - about 1.5. The differences are very significant, especially when you consider the difference in the cost of equipment.

Benefits of Novec liquid cooling over mineral oil

At this stage, both systems seem to be the same in terms of characteristics. But in the described calculations we did not take into account that the liquids in the immersion bath do not mix by themselves. Imagine that we plunged a video card into mineral oil and turned it on. The oil layer directly next to the radiator of the video card warms up, and those volumes of oil that are at a certain distance are not. In this case, there will be a local overheating in the area of ​​the video card. Then it becomes obvious the need to ensure effective mixing of the oil inside the bath, either using video card fans or in some other way. This complicates the design and requires special technical solutions.

In two-phase immersion cooling systems with Novec liquid, this problem is absent. It constantly boils and mixes itself - especially in those places where heating occurs. The bubbles detach from the radiator, and in their place comes a new liquid.

The second important difference from Novec oil is flammability. Novec never burns, it is even used to extinguish fires in libraries. Oil is easily burned by nature, and, plus, it can not be extinguished with water. This is indicated in the specifications of any oil. The temperature of the onset of combustion is about 200-400 degrees.

We conducted a series of experiments to verify the conclusions described below. When the oil warmed up to 150 ° C, it began to smoke, after which a flame appeared, which confidently flared up with every second. Further, due to burning, the temperature of the oil began to rise by 2 degrees per second, and the flames grew higher and higher. The flame was already difficult to shoot down, and the temperature, meanwhile, continued to grow.

A similar experiment with Novec showed that the liquid actively evaporated, but it was not possible to heat it above 61 ° C. Burning a pair of Novec, to achieve their burning also failed. As written in the specification for Novec - the material does not burn, does not light up.

So, is it possible to use mineral oil and its analogues for effective cooling of electronics - of course, yes. Will it be risky given the extremely high cost of equipment? Of course. Fire can occur for many reasons, and the presence of mineral oil in large quantities can make the elimination of such a fire extremely difficult, and the consequences - catastrophic.

Novec fluid: weak points

What are the most serious drawbacks of Novec fluid? The high price, increased demands on the tightness of the immersion bath associated with the high volatility of Novec fluid, and the complexity of the design of the latter. In addition, we can note the need for careful monitoring of the parameters of the cooling process in order to avoid boiling away of the liquid. Plus, the use of Novec fluid becomes economically justified only when using specialized graphics cards with high installation density.

When designing a cooling system, about 20 different characteristics must be considered. Full calculations must necessarily take into account the thermal resistance of radiators and thermal interfaces, as well as the material properties and surfaces of heat exchangers and radiators of the equipment used.

Progress does not stand still, it is already obvious that the future of the industry lies in effective immersion cooling, and not in air. It remains only to make a choice between a difficult and safe decision, or more accessible, but risky.