Tetris games, or what is capacity management in the data center

Hall of the NORD-4 data center in 2016.

Here is an empty machine room in the data center. Think rack can be put in any place? If only it were that simple!

My name is Alexey, I do capacity management in DataLine data centers. Today I will tell you that it is an entire science, and sometimes art, to properly arrange the stands in the hall.

What is capacity management?

The machine room in the data center has three main resources: space, power supply and cold.
All three of these resources should be spent evenly and end at the same time. Actually, this is followed by a person engaged in capacity management.

If the equipment is placed at random, we may end up in the following unpleasant situations:

• electricity in the hall is fully selected, and half of the hall is empty;
• the hall is packed with racks to the eyeballs, barely enough electricity or cold.

Why is everything so difficult? The data center is designed for racks of a certain capacity, size and cooling schemes. If we place in the halls of the rack with the design parameters, then there is no problem. But in life it does not happen, especially in a commercial data center. The equipment comes in very different ways: some customers want to place a pair of racks of 20 kW each, others 100 for 5 kW, some third 20 wide racks 800 mm each, and for the fourth, the equipment throws warm air from the side. So each request for placement turns into a small optimization problem with three variables: location, electricity and cold. About them and talk further.

About the place

This is perhaps the most inflexible parameter, so every centimeter counts. When I say this, it means that I measure all the new halls with a tape measure in order to verify the reality with the drawings. He began to do so when somehow when installing the racks it turned out that the distance between the supporting pillars turned out to be 1 centimeter less than on the plan. Then, in order to squeeze, I had to disassemble the pillar trim.

On the plans of the halls, I additionally designate all objects under the raised floor, on the walls and network trays. This information is useful if the client wants to stay with a fence that goes from the base floor to the ceiling itself.

On the plans, I also note the service areas of engineering equipment. This is a reserved space near the air conditioner, switchboard and other equipment for its repair. Racks should not climb and, especially, be completely in the service area.

Usually the service area is registered on the equipment passport, but we take as a guideline the size of the object itself. For example, for an air conditioner 900 mm deep, the service area in front of it will be at least 900 mm. So if necessary, we can pull this equipment out of the hall. And so really had to do .


The rack, marked in red, is already superfluous in a row and climbs onto the air conditioner service area. Disorder.

About electricity

The most important thing in this part is not to overload the available capacities. To track the load, you need to know what we have on the project, and keep records of installed racks. This is in theory, in practice, you will not always have all the input. Here are some of these situations:

The hall is empty, there is a design capacity of the data center / hall / rack on hand, and we push off from it. We fill the hall on the basis of these data, keep records of occupied and free capacity, do not allow their overload.

If you already got a partially filled hall and there is no information on the occupied capacities, we first deal with the installed racks. We find out how much they consume at the moment and how much they are allowed to eat. For example, the rack can now consume 2 kW, but when installed under it reserved 5 kW, for growth. Next, we check with the project documentation: we look at how much power we have in principle. After some simple calculations, we understand whether we can put the racks on; if so, how much and how much power.

It’s quite bad when there are racks with equipment in the hall, there is no documentation, no one takes into account. Here again it is better to start with an inventory of installed racks. Next, we need to understand what we have on the project. We go to the UPS, we look at its power according to the passport and what feeds this UPS. For simplicity in the example I will consider 1 UPS.

Suppose its power is 450 kW. Subtract from this power 10% of the reserve and work on with the figure of 405 kW. On the switchboard comes 405 kW. There are 4 automata in the shield, each automaton is 1 row with 19 racks. On a row - 101.25 kW. Additionally, we check the settings on the input machines of the row and the switchboard of the UPS. The settings may be down, and instead of 101.25 kW there may be 90 kW. If the settings did not do anything, it turns out that the rack has 5.3 kW. This does not mean that now we can put racks on 5.3 kW only. It is possible more or less, the main thing every time to recalculate the remaining power.

Important: 5.3 kW is the power of one beam. The same to the rack is suitable from the second. Each beam should be loaded only half the rated power. If one beam falls off, the second will be able to take on a double load.

Together with the pair load on the rays must be observed and the pair load on the phases in switchboards. When one or two phases are overloaded, the so-called phase distortion occurs. This will, at a minimum, mean that the available power is not optimally used. More about this, colleagues wrote in the articles about the device and power supply monitoring in the data center.

About cold

Cold is closely related to electricity. If the rack consumes 5 kW, then, roughly, it emits 5 kW of heat. We usually believe that if the hall is designed for 500 kW, then the cooling capacity of all air conditioners in this room should also be 500 kW (plus the capacity of one backup air conditioner). That is, stocks of cold and electricity must fight with each other.

Custom racks and equipment

When you need to place a standard rack (width 600 mm, 42 units, 5 kW), you can put it in any place where we have electricity and cold. Nuances arise when high-loaded racks need to be installed in the hall (8–30 kW).

For example, you need to put a rack of 30 kW. We have an empty room, put it at the very beginning of the room, next to the air conditioner.

Now imagine that the air conditioner turned off next to it. Calmly, we have a backup. If it were the usual racks of 5 kW, then working air conditioners would quietly pull away the hot air and the equipment would not notice anything. But a 30 kW rack generates a whole cloud of heat, which simply does not disperse along the hot corridor. It is important that the closest air conditioner works. If it breaks, the hot air begins to fall into the cold corridor, again falls into the equipment. After minutes, the equipment in this rack will become very difficult, monitoring will work, the rack will run to help with the help of drive-through fans. Equipment will save, but the problem with the unsuccessful location of the rack will remain.

Empirically, we found in the hall a zone in which you can put such powerful racks. If some conditioner dies, then the rest will be able to draw hot air from this rack (see diagram).


The red stands are places where you can not put a high-loaded rack. Green - wherever possible. Yellow - permissible, but only if absolutely necessary.

To help the cooling system in racks of 8 kW, we use ventilation doors, air tiles, insulation of cold corridors. They do not provide additional cold and will not save if the cold power is already selected, but help to control air flow and improve the efficiency of the air conditioning system.

Ventdveri are installed on the rear wall of the racks 8-11 kW. They do not release hot air into the hot corridor and direct it upwards with the help of fans.


Vertical door

In racks from 11 kW, active ventilation tiles (perforated tile with a fan) are added to the ventilation doors. They additionally direct the flow of cold air to the equipment. Ventilation tiles are smart: read the indicators from the sensors on the rack and adjust the fan speed.


This is a ventilation tile. This even has a small display

If a client has two rows of powerful racks, for example, 15 kW each, it is reasonable to isolate it.


Isolated cold corridor

Often customers bring three phase racks. Not that this is highly non-standard, we just designed our first data centers mainly for single-phase racks with a small margin for three-phase load (10–15% of the total capacity). If the client needed a three-phase rack, then simply put a three-phase automatic switch in the backup section of the switchboard. As the name suggests, a three-phase automatic machine is three times as large as a single-phase one, takes up more space, and you will not put a lot of these. Sometimes the client had so many three-phase racks that the reserve in the switchboard was chosen, and it was necessary to install additional boards.

In our newest data center NORD-4, we have foreseen this moment. There, the power supply is distributed through the busbar under the raised floor, and this allows you to put a rack of any phase.


Both single-phase and three-phase circuit breakers can be installed on the busbar

Usually the equipment takes the air from the front, from the cold corridor, and throws it from behind, into the hot corridor. But it is different. For example, the equipment sucks in cold air from the front and rear, and the hot air issues upwards. For such a rack with the help of isolation they organized a branch of the cold corridor in the hot. This is not exactly feng shui, but it works.


For “non-standard” equipment in racks 030 and 040, they made insulation in the middle of a hot corridor

There is equipment that takes air from one side and throws hot air into the other side. We installed this equipment on a 800 mm rack so that the cold and hot air could have a place to circulate. On the side of the cold corridor in front of the counter, a side air blower unit was placed: it takes the cold air from the front and leads it to the side and up. The space on the sides of the rack is insulated so that there is no parasitic air exchange.


The lower unit occupies the same block of the organization of the side blowing


From this side, cold air enters the equipment. In the background is visible insulation from the hot corridor.


To the left of the equipment is visible insulation, which does not release the waste hot air into the cold corridor.

Fences

Together with racks, customers often order fences for racks, especially banks. They need to be positioned so that they do not eat up a lot of usable space, and therefore the racks must be positioned compactly, so that in the end something will turn out even remotely resembling a rectangle. But sometimes it turns out, as in the diagram below).



Usually, a fence is built from the main floor (i.e., there is also a fence under the raised floor) and up to the ceiling, closing the network trays.


To get to the fence, you need to attach a pass to the access control sensor or a finger to the biometric scanner


Fence times


Fence two


Additional surveillance cameras and sometimes volume sensors are placed inside the fence.


But the fence with integrated insulation of the cold corridor, 2 in 1 so to speak

Capacity management tools

To monitor the consumption of space, electricity in the machine rooms it is not necessary to use expensive and trendy DCIM (Data Center Infrastructure Management) systems. Statistics on the occupancy of the hall can be maintained in the usual excel. Collect there information on the current state of the hall (the location of the racks, their power, size) and discipline make there information on the changes: which racks were added, which moved.

The output will be quite a working tool that will help you understand where, how many and what racks you can install in this room.

This concludes my brief excursion into capacity management. Ask questions if something you are interested in is left out of the post.

Finally, here is a pictorial picture of how the room was filled in the OST-3 data center from the moment it was commissioned in 2014 until today.