Full immersion: how do we cool ASIC chips in Bitfury
Let's tell how the technology of immersion cooling equipment in data centers works. Let's talk about our solution, developed in conjunction with the subsidiary Allied Control, and give examples of other "liquid" ways to "bring down the temperature" of the chips.
/ Bitfury
As they write in the journal Nature, all data centers in the world consume about 200 TW · h of electricity annually. This amount is comparable to the electricity consumption of some individual countries , such as Australia. Also, data centers are one of the main environmental pollutants. They account for 0.3% of global CO2 emissions. The figure seems small, but as much carbon dioxide is “ generated ” by the planes of all airlines in the world.
Almost half of the electricity consumed data centers spend on cooling server hardware. For this reason, many companies are developing technologies that should reduce the "consumption" of electricity and improve the efficiency of air conditioning systems.
Bitfury deals with this issue. Together with our subsidiary Allied Control, we created a two-phase immersion cooling system . It is designed to work in highly loaded data centers that consume large amounts of electricity during the processing of large amounts of data (for example, mining farms).
The system consumes less electricity and takes up less space compared to traditional air conditioners for air cooling. This is due to the fact that printed circuit boards designed for cooling in an immersion manner do not have radiators. Components on them can be placed more closely.
Since the servers (in our case, ASIC-chips) are immersed in liquid, the data center becomes almost silent. Such cooling systems help data centers provide employees with more comfortable working conditions. Immersion cooling is implemented in our 45 MW data center in Georgia. This is one of the largest implementations of immersion cooling at present. We have prepared a short video in which we talk about the capabilities of the cooling system:
Server hardware is placed in large tanks, similar to aquariums. They are filled with a special liquid 3M Novec 7100 . It does not conduct electricity, does not burn, does not emit toxins and does not harm servers. For this reason, 3M brand fluids are also called “dry water”.
Equipment cooling takes place in two stages. Microcircuits heat up and generate heat, Novec fluid boils and turns into steam. Evaporation rises to the condenser coil, which has the form of a cylindrical spring.
Then the second phase begins - the coil (it itself is water cooled) turns the vapor into liquid, and it flows back into the reservoir. The condensation process takes very little time, since Novec 7100 only boils at 61 ° C. This approach saves up to 95% of energy with minimal fluid loss.
The new cooling system allows to reduce the PUE of the data center to 1.02 and to increase the density of equipment placement - no additional areas need to be allocated for cooling systems. The “refrigeration system” itself plays the role of a rack and effectively copes with 40 megawatts of computing power (252 kW per rack) without excessive consumption of electricity.
As we have said, the development of systems for immersion cooling are engaged in many global companies. One of the popular technologies is the immersion of equipment in mineral oil, which removes excess heat. For example, this solution is used in Green Revolution Cooling. As a heat-conducting fluid, they use a transparent mixture of mineral oils ElectroSafe . According to the authors of the project, this approach is a thousand times more efficient than air cooling, and the system capacity reaches 100 kW per rack.
But there are also approaches when the data center is flooded entirely, plunging into the ocean. In this case, water enters the radiators on the chips through special pipes and removes excess heat. Waste fluid flows back into the reservoir. However, it does not cause significant harm to the ecosystem - the water around the immersed data center heats up by only a few thousandths of degrees Celsius.
The first prototype of such a data center was “ flooded ” in the Pacific Ocean five years ago off the coast of California. He worked without interruption for 105 days. This year, the company sent “to the bottom” a second experimental data center. The dive took place near the Orkney Islands , located 16 km from Scotland. According to the plan, the data center should work for five years. If everything goes well, the “combat” data center, which is used to solve real problems, will go under the water.
The main disadvantage of the approach is the difficulty of repairing and updating the infrastructure. Also, such a data center is subject to corrosion and attacks of marine life. But the company is working to eliminate these problems.
We are convinced that due to rising electricity prices and the need to introduce new "green" technologies in the future, more and more companies will begin to implement immersion cooling systems. The trend has already begun, and over time it will develop more and more.
PS Our first habrapos in English: How Exonum Shifted from Iron to Actix-web .
/ Bitfury
Why do we need new cooling technologies
As they write in the journal Nature, all data centers in the world consume about 200 TW · h of electricity annually. This amount is comparable to the electricity consumption of some individual countries , such as Australia. Also, data centers are one of the main environmental pollutants. They account for 0.3% of global CO2 emissions. The figure seems small, but as much carbon dioxide is “ generated ” by the planes of all airlines in the world.
Almost half of the electricity consumed data centers spend on cooling server hardware. For this reason, many companies are developing technologies that should reduce the "consumption" of electricity and improve the efficiency of air conditioning systems.
Bitfury deals with this issue. Together with our subsidiary Allied Control, we created a two-phase immersion cooling system . It is designed to work in highly loaded data centers that consume large amounts of electricity during the processing of large amounts of data (for example, mining farms).
The system consumes less electricity and takes up less space compared to traditional air conditioners for air cooling. This is due to the fact that printed circuit boards designed for cooling in an immersion manner do not have radiators. Components on them can be placed more closely.
Since the servers (in our case, ASIC-chips) are immersed in liquid, the data center becomes almost silent. Such cooling systems help data centers provide employees with more comfortable working conditions. Immersion cooling is implemented in our 45 MW data center in Georgia. This is one of the largest implementations of immersion cooling at present. We have prepared a short video in which we talk about the capabilities of the cooling system:
How the solution works
Server hardware is placed in large tanks, similar to aquariums. They are filled with a special liquid 3M Novec 7100 . It does not conduct electricity, does not burn, does not emit toxins and does not harm servers. For this reason, 3M brand fluids are also called “dry water”.
Equipment cooling takes place in two stages. Microcircuits heat up and generate heat, Novec fluid boils and turns into steam. Evaporation rises to the condenser coil, which has the form of a cylindrical spring.
Then the second phase begins - the coil (it itself is water cooled) turns the vapor into liquid, and it flows back into the reservoir. The condensation process takes very little time, since Novec 7100 only boils at 61 ° C. This approach saves up to 95% of energy with minimal fluid loss.
The new cooling system allows to reduce the PUE of the data center to 1.02 and to increase the density of equipment placement - no additional areas need to be allocated for cooling systems. The “refrigeration system” itself plays the role of a rack and effectively copes with 40 megawatts of computing power (252 kW per rack) without excessive consumption of electricity.
Alternative cooling methods
As we have said, the development of systems for immersion cooling are engaged in many global companies. One of the popular technologies is the immersion of equipment in mineral oil, which removes excess heat. For example, this solution is used in Green Revolution Cooling. As a heat-conducting fluid, they use a transparent mixture of mineral oils ElectroSafe . According to the authors of the project, this approach is a thousand times more efficient than air cooling, and the system capacity reaches 100 kW per rack.
The principle of operation of oil installations is similar to the one that we use in Bitfury, only for the removal of heat, additional pumps are required here to transfer the refrigerant (oil is more viscous than water). In our system, the circulation of "dry water" occurs passively, due to the heat generated during the operation of ASIC chips.
But there are also approaches when the data center is flooded entirely, plunging into the ocean. In this case, water enters the radiators on the chips through special pipes and removes excess heat. Waste fluid flows back into the reservoir. However, it does not cause significant harm to the ecosystem - the water around the immersed data center heats up by only a few thousandths of degrees Celsius.
The first prototype of such a data center was “ flooded ” in the Pacific Ocean five years ago off the coast of California. He worked without interruption for 105 days. This year, the company sent “to the bottom” a second experimental data center. The dive took place near the Orkney Islands , located 16 km from Scotland. According to the plan, the data center should work for five years. If everything goes well, the “combat” data center, which is used to solve real problems, will go under the water.
The main disadvantage of the approach is the difficulty of repairing and updating the infrastructure. Also, such a data center is subject to corrosion and attacks of marine life. But the company is working to eliminate these problems.
We are convinced that due to rising electricity prices and the need to introduce new "green" technologies in the future, more and more companies will begin to implement immersion cooling systems. The trend has already begun, and over time it will develop more and more.
PS Our first habrapos in English: How Exonum Shifted from Iron to Actix-web .
Source: https://habr.com/ru/post/439654/