The infrastructure of the communication center from the kettle, or the next construction of the server - part 1

Hello! Having read the publications on the topic of building a data center for a small hosting provider, as well as “how to make a server room with my own hands”, from people who apparently did not know how to say why they should order their services, I decided to describe my experience in building a small communication center in this year, for the placement, with subsequent registration in Roskomnadzor, of such objects as OTMUS, TZUS, SPD.

This year I had to search and apply so much information that I felt my limit in remembering useful information. Therefore, it’s difficult for me to immediately remember where I got the information from (I’m talking about the orders of the Ministry of Communications, the Ministry of Communications, Roskomnadzor, SNIP, VSN ...).

Under the cut, I will try to superficially go through all the aspects that need to be decided and implemented in your hardware.

Room infrastructure

If you start with the choice of premises, then we must take into account that the infrastructure of this room should have at least the following set of elements and structures:

• Estimated floor load - you need to look at the building design. Remember that 1m ² can be taken as 700kg and a battery enclosure. If the area allows - better of course the rack
• Cable Management: Trays under the ceiling and / or in the floor (both a cutout gutter in a concrete floor and under a raised floor.
• Power supply and distribution system
• A cabinet / rack to accommodate battery circuit assemblies.
• Gas extinguishing system.
• SKD.
• CCTV.
• A freestanding construct with tubular profiles for attaching 10 pair skirtings, if you have a need to organize a variety of copper SLs (PRI for example) between equipment or from closely located partners / customers.
• Inter-rack copper and optical cross-countries.
• Of course, cabinets / racks.
• Air conditioning (climate) with reserve
• Fire resistance of the building (II or III degree).
• Above the premises of the car room (we will call our server room) there should not be bathrooms and other utilities related to the use of water.
• The floors in the car hall must be fireproof, on which clean floors are laid in the form of antistatic non-combustible linoleum or raised floor tiles.
• If there are windows in the room, at least they need to be tinted with film or covered with plywood, since sunlight is not allowed on transformers and batteries. In other things, the sun's rays will not do anything good for other equipment. Ideally, there should be no windows at all, but this is unlikely to be a decisive factor if all other conditions can be fulfilled.
• The walls of the bus room should not be made of combustible materials. This is not only a fire-fighting cause, but also the need to build an isolated chamber in case fire extinguishing systems were practiced.
• Gas exhaust system, in case of fire extinguishing system.
• Lighting should be good. In some sort of BCH for lighting, I saw a figure of 200 lux. But in principle, I think there is no such problem, so just take care of good lighting, preferably with dust protection.
• Огнестойкая дверь с уплотнителем по периметру (для герметичности).
• Все металлические конструкции (шкафы, лотки, стеллажи, дверь) должны быть заземлены.
• Все пространство в стенах автозала вокруг кабельных и вентиляционных вводов должны быть заделаны асбестом или специальной противопожарной пеной (оранжевая или красная такая).
• Батареи и другие металлические конструкции, которые могут служить устройством заземления, необходимо отгородить не токопроводящими материалами, что бы исключить возможность замыкание стоек или корпусов оборудования на эти конструкции через тело человека (при одновременном прикосновении).
• And a little on labor protection and electrical safety: room category labels and hazard levels, with the name of the person responsible; designations on sockets and grounding devices, posters on electrical safety, fire extinguishing and first aid (usually the themes of posters are chosen closer to the described situations to their industry); carbon dioxide fire extinguishers and first aid kit; rubber dielectric mats in front and behind cabinets and racks

Power supply

Depending on what your plans are for your car dealership, you can choose a certain level of protection against power outages.

You can open any of the parts of the "Rules for the use of equipment for transit, terminal-transit and terminal communication nodes" (there are 14 parts for different types of communication nodes) and in all there will be a reference to


And also on:


... which in turn has lost force in connection with a bunch of changes ... and in general, now we must be guided


But at the same time, there is a link to 32 order, but in 97 it is not written which communication center belongs to which category of electric receivers. In this regard, it is not clear which category is required to be provided. If suddenly someone in the subject - please share information.

But let's say you have decided which category of power receiver you have and you are going to provide it. For the safety of your business, in any case, you must have input from a city substation, a UPS and a 2-5 kW generator (approximately this will be necessary for a small equipment room with two racks clogged with various equipment). So it is necessary to organize an automatic transfer switch (automatic reserve input) in the input-distribution device (ASU) with generator start-up functions (it can be done manually if there is an on-duty shift at the facility).

Perhaps someone will have a question, how to implement all this, in what form, how to place it?

• Firstly, this is a question for electricians. If you do not have such staff, contact the organization that will install the ASU with the ABP. The product must be accompanied by a passport.
• Secondly, - you can try to assemble yourself, using a pair of power contactors with mechanical and electrical interlocking, a voltage monitoring relay with NC / NO contacts. I had to collect this for the purpose of providing a certain infrastructure of the object as a whole (video surveillance, fire extinguishing system, ACS, ventilation, emergency light). And the bus station is powered from the input, where the ABP system is separately organized with two feeders from different substations and the launch of a diesel generator.

A simple ATS for working with two inputs:



So, you have an electrical installation inside your future bus terminal. Next, you will need to divert lines from there to the load. Due to the fact that equipment racks, cabinets for batteries and other metal structures (including a door) are usually located around the perimeter of the room, it would be nice to make a grounding tape around the perimeter, to which you can hook everything that you want to put along the walls and will need to be grounded .

Due to the fact that your equipment must be connected via UPS, the first obvious line from your switchgear will go exactly to the UPS. In my case, this is a 3-phase Symmetra APC, which I would not recommend buying, due to its current cost. You can find UPS more modest manufacturers. The main ones I would single out the following criteria for myself:

• 3-phase, if all the power is supplied through one UPS or 1 phase, if you put the UPS in each rack to ensure an individual (on the scale of the car park) reserve.
• output quality
• presence of bypass
• hot-swappable battery module (possibly a power module)
• monitoring
• power supply to the load through C19 sockets or terminal contacts
• connecting external battery assembly

And already from the UPS to divert lines to the outlet blocks in the rack (s), depending on the location of your UPS (general or individually in each cabinet).

Here in the power supply circuit there is a bottleneck - UPS. Therefore, the design of the UPS should be either one in which there is nothing to break (APC Symmetra, for example, without power and battery modules is just copper busbars (practically)).

If you use the UPS individually on the cabinet, then make sure that the cabinets have lines from different UPSs, and the power supplies of the equipment are connected to different UPSs.

Or there is a third option: your UPS allows you to take the power line from it, drag it back into your ATS, and then assemble a circuit in which you can supply power to the line either through the UPS, or directly from the city input or generator. Such a scheme will allow removing the UPS from the line for routine maintenance and repair.

With a current inverter, circuits are similar to circuits with a UPS, if you have it serious enough that the lines go to more or less serious terminals to the load. And if you have small inverters, then, as a rule, they are not sharpened to take power to the load with a large cross section. Such are placed inside the rack where they will be used. And if there is a need to connect a lot of equipment (and not one or two pieces of iron), then a line is sent from the inverter terminal block to the switchgear in the cabinet itself and there, through buses or AB 2P (or through AB for minus and bus for plus), there are already lines load.

For the purpose of placing the UPS and inverters, I have installed a separate frame rack with a large number in the value of the bearing mass, to which the power lines from the ASU with the ATS come. And then from which the power lines go to the equipment cabinets.

Here I would like to note a couple of points related to the choice of materials and components:

• Choose circuit breakers in the ASU taking into account selectivity, not only at face value, but also at response time (for this you need to choose a line of products from manufacturers, and, preferably, use the products of the same brand).
• Try to use the most fireproof materials inside your car room, so I chose the VVGng-RFLS cable + corrugated HDPE pipe (black and orange).
• For the same reasons, we chose outlet blocks - non-flammable plastic, fire tests, without a switch and always with a terminal block, for connecting a cable from the UPS to it. Maybe I have one such problem, but I spent several days looking for the outlet blocks I needed. All that is offered in the search is frankly not high-quality "China" with an integrated cable, usually a small cross-section, or with a C13 connector, from which the cord will drop out sooner or later, not to mention that the finished cable with 2.5 mm ² conductorsI didn’t find it, I also wanted to use vvgng-rfls, but it’s not good to plug it into the C14 plug. As a result, I found the blocks of outlets made of fireproof plastic with a terminal block inside. By the way, they are produced in Yekaterinburg and the product as a whole is excellent, but I had to refine it - drill a hole for the input, look for cable fixing brackets ... bought 8 pieces - used 4 so far and not a single identical product :). Now I would recommend either the same method, or socket blocks with a built-in cable with an industrial connector. Moreover, if you have a large car park, and you will install racks as needed, then the option with outlet blocks with industrial connectors looks more convenient, since you just connect a new cabinet to the failed line. And it will be a reliable connection.
• I made the power lines for the DC switchgear in each cabinet from the PUVG cable (blue and red) in the corrugation. If you google ShchRZ, you will see what it is and that there is even a functional for monitoring the status of the line from AB to equipment.

Calculating the autonomy of your electrical installation and choosing a battery

In order to understand how much you need to buy batteries (namely, this is a question, because it requires a lot of money), then there is nothing difficult if you know what kind of load on your power system you are counting in CNN (the hour of maximum load) . For built-in battery packs, information on autonomy can be found in the materials for the UPS, and to calculate the required capacity of the external assembly, you must multiply the assembly voltage by the capacity and divide by the load (t = V * Ah / Wt). But since the discharge of the battery is not a linear process, it is necessary to look at the discharge schedules in the manuals for the batteries and, depending on the time you have, apply the coefficient found in the passport. Then you will get more or less real battery life from your battery system. Based on this, you will understand what capacity of your battery assembly you need to provide.

In the article about building your own small data center for a small hosting provider, car batteries were used, due to their low cost relative to unattended batteries developed for use on communication networks (special operating modes - constant charge, rare charge, long-lasting). If you want to change them in 1-2 years (depending on the discharge and the depth of this discharge), then of course you can use this option. Just keep in mind that these batteries are serviced, which means that if something happens (overheating, for example), they can leak or evaporate through the covers - which means you need to put them in a separate ventilated room and all that ... If you use sealed, unattended batteries, you you can place them even in equipment racks (at the bottom of the cabinet or on cantilever shelves),

Alternative opinion

I wrote above about outlet blocks ... but in fact, I believe that it is more correct to connect equipment from the AV to the switchboard inside the rack. So the connection is reliable and secure. But there are some inconveniences. Basically, they are connected with the fact that the equipment has a completely specific outlet C14, the plug for which (C13) can not always be found to the proper quality. As a result, she can stagger there and the weak spot will already be there. In addition, we must take into account that not all engineers at least know what a PUE is, and they need to be able to safely turn on the equipment (let’s omit the point that, according to safety precautions, any engineer should have at least 3 groups in the electrical safety crust + accompanying the person in charge with the 4th group - we all understand the reality of being).

Cable Track Infrastructure

In order to lay the cable from different corners of your car room (from the inputs to the equipment room itself or from the cabinet with batteries) you will need cable trays. This can be a system of cable trays under the raised floor, or trays under the ceiling, passing around the perimeter of the room and above the racks ... In my case, the optimal decision was made: cable elbows go under the ceiling, and a groove running under the telecommunication cabinets, a cabinet with battery was cut out on the floor and a branch to the ASU on the wall. The gutter is covered with iron (and grounded), and on top is covered with 4mm steel sheets, which, like the rest of the floor, are covered with antistatic non-combustible linoleum. We need to put in new racks - remove the hatches from the right places, put up the racks, drag the DC and AC power lines through the gutter and that's it.

Despite the fact that the equipment has not yet been installed in this car park, cable elbows are already engaged in bundles of copper lines and optical cables. Namely, cross-country cross-country racks were made. I have two cross structures: a single-frame rack and a structure with tubular mounting profiles for 10 pair skirtings. From the telecommunication racks, UTP bundles from each rack come to the single-frame rack, which are embroidered on both sides of the patch panel. The optical cross over is organized in the same way. Thus, any crossovers between equipment in different racks are performed by short patch cords inside their racks. And the incoming single copper lines to the car park (SCS from the office to 8 lines and copper connections with partners also come to the frame with crosses, after which they are already transferred to the necessary equipment.

And I need a hefty frame for skirtings because of the need to organize a large amount of PRI port connections between the station and the transport side. The SDH transport card or media gateway is embroidered immediately to the desired side of the construct, then it is crossed, if necessary, with a conventional crossover cord.

Due to the fact that my power cables go through the gutter in the floor, and the weak current goes into the cabinets from above, they practically do not intersect each other (I have equipment whose power is turned on at the front), even taking into account the fact that the crosses are located on the rear mounting profiles.

Telecommunication cabinets and racks

In my opinion, if you are building a communication center or server room where your main or one of the main locations will be located, then the minimum cabinet sizes should be 800x1000 (WxD), standard height 42U, with a base or 10cm raised above the floor, and the same openings for the entry of cable infrastructure, both through the base and through the ceiling. Although I prefer 47U, because when everything is stuffed in the closet, units become scarce and nothing can be done about it.

And I didn’t just write “cabinet dimensions”. I believe that racks (just a double frame frame) can only be used for testing stands and crosses. After some time of operating a number of racks, you will find that the patch cords were thrown by vines between the racks on all sides, even if you yourself like order, someone else will certainly throw such a nozzle to speed up the process, and then forget to redo it before how will the customer service be launched there. That is why there should be cabinets and they must have closed side panels and lockable doors.

In order for the doors to always close, it is necessary to install the mounting profiles at least 10cm in depth, both front and rear. If they are placed close to the door, there may be inconvenience with equipment that protrudes forward with its front panel behind the mounting profiles, + the SFP module sticks out + the optical patch cord in it. As a result, the door will bend the patch cords, if at all able to close. Just recently, because of this lack of thoughtfulness, I had to go to the site several times where we installed cwdm in a rented rack (by the way, he himself did not bulge out of the plane of the mounting profiles). The door could be closed differently and she pressed patch cords differently, dropping the links. It’s good that they understood what was happening at the testing stage and quickly decided on a solution. And in order to be able to drown mounting profiles, and a cabinet depth of 1000 mm is needed - in case of purchasing a satisfied server (for example 800), you cannot fix the slide on the profiles placed closer than 800 mm, and the server itself will not be able to completely drop into the rack. (at least Intel platforms have such a slide design).

The doors themselves can be glass or perforated (someone buys entirely steel without perforation). Here is the choice of tasks and ways of providing climate. If you have cold air flowing into the corridor, then there is no point in taking a glass door (or a door without perforation). Take the perforated doors and all your equipment will be cooled taking air from the front (especially the server), expelling the heated back. If you bring sleeves with cold under a rack, then you need to take racks with blank doors.

The cabinet must have vertical organizers on the front mounting profile and, if necessary, on the rear. I have vertical organizers on both sides, because the front organizes the ascent and descent of copper and optics, and the back supplies power to the equipment and UTP from the servers (again, the copper with the data goes on one side, and the power cords on the other). And when supplying power cables to the rack, figure out on which side you will have power supplies - on the one and draw the line. Then you will have another one for laying copper from servers and katalists (for example) with boards in the rear slots ...

Climate

It seems to be considered not difficult. If you don’t know what your needs will be then, then you can quickly figure out and go to 2 options:

• immediately industrial or semi-industrial air conditioners with a capacity sufficient to cover the maximum heat generation (calculated on the basis of the maximum possible energy consumption per rack, with a coefficient of 0.9 for confidence; and for accuracy - you need to look at the equipment passport).
• put a household split ... then another one ... then another ... Just make sure that you can connect some control module with sensors to these splits in the future, so that all this is a little smart.

In any case, there should be a reserve of the system for maintaining the temperature from 18 to 24 ° C and humidity in the region of 50-60% in your car room. For example, we have a large air conditioner powered by a different electrical network than the car terminal. And now, if someone can repay it, the split system will automatically start, which will also cope with this task without problems, but they will take power from the mains with a diesel engine, but without a precious stand-alone UPS resource.

You can also do automation and monitoring, depending on what the air conditioners are capable of ... For example, we have a control unit for a split system that monitors temperatures at several points in the bus station and sets the operation mode of several air conditioners.

But in general, for me this topic is quite far away and all we needed to do on this issue was the contractor. The plans are only to fasten to this system an alarm from systems such as netping or something similar, so that additional air conditioners would start not only when the big one shuts down.

Security systems

• Fire extinguishing system. Gas must be used to extinguish the fire. It is dangerous for people, but safe for your infrastructure. Of course, when triggered, the system will yell for the specified number of seconds and only after that will it open the gas cylinder. Therefore, personnel should be instructed, not deaf or blind, to see light-signaling information about the state of the fire system. As a rule, fire extinguishing systems have indication and control panels - personnel must be trained in advance to understand and use these devices.
  
  
• Security alarm. Here it is better to use the system operation regulations issued by the security organization. Nothing to say here. If someone is constantly present at your facility, and the premises are located on the territory of the building’s security guards, you may not need it.
  
  
• SKD. According to the regulatory and legal documentation from the Ministry of Communications (Order No. 1 of January 9, 2008), only persons who install, operate and maintain equipment should have access to the premises of communication centers. Here the issue is not only the security of access to equipment (and after all, electronic tags or passwords can be stolen from less trained people - for example, bookkeeping or a manager), but also on safety measures. It is unlikely that the same manager has an electrical safety certificate and he was instructed. And from the moment you pulled the power supply across the cabinets - your entire room is one electrical installation.
  
  
• CCTV. Like any other security system, its power must be reserved and access to its controls must be limited. Cameras should be chosen with PoE power, so that there would be no dependence on any wall outlet. In the car room, cameras should be placed not only to record security incidents, but also to analyze flights with technical personnel. The fact is that an engineer can do something by accident, and then not admit, fearing reprisal. For example, if you still use the socket blocks with a plug-in cable in the C14 connector, sooner or later the C13 plug will drop out, especially if someone decides to climb and loosen the cable there. Similarly, from the equipment side - if the cord does not fit well in the equipment’s power supply connector, find a way to improve its fit. Sometimes it’s just a bad cheap wire - I immediately throw it away. And sometimes the connector in the equipment is already deformed (the plastic has dried over time, for example). Or an engineer may accidentally take out the wrong patch cord. I myself had a case in the first year of work in telecom, when the SFP module was stuck in the board, and it was very necessary to urgently replace it. As I just did not try to take it out of there. And it was the Juniper MX5-T ... it was then that I found out that the interface boards did not screw onto the case - the entire 20x1Gb board crashed with working links ... We really didn’t have cameras, but I did not hide the circumstances of what happened. I myself had a case in the first year of work in telecom, when the SFP module was stuck in the board, and it was very necessary to urgently replace it. As I just did not try to take it out of there. And it was the Juniper MX5-T ... it was then that I found out that the interface boards did not screw onto the case - the entire 20x1Gb board crashed with working links ... We really didn’t have cameras, but I did not hide the circumstances of what happened. I myself had a case in the first year of work in telecom, when the SFP module was stuck in the board, and it was very necessary to urgently replace it. As I just did not try to take it out of there. And it was the Juniper MX5-T ... it was then that I found out that the interface boards did not screw onto the case - the entire 20x1Gb board crashed with working links ... We really didn’t have cameras, but I did not hide the circumstances of what happened.

Getting ready for commissioning

Due to the fact that small telecom operators have the right to register nodes according to a simplified scheme, recently there has been no need to prepare projects for all engineering systems of the node (power supply, air conditioning, lighting, fire extinguishing, and it seems that there was a cable route project before). This means that when submitting documents to Roskomnadzor for registration of a communication center, you have the right not to provide this package of documents. But everything should be done for you, because no one has canceled the inspection commissions of technical supervision and they themselves can come and conduct an inspection of your dangerous (or especially dangerous) object. Therefore, you should not deviate from the implementation of all this just because your management found out that none of this is necessary for registering the site. Sooner or later, these systems will work out and save from losses or death.

After you decide that you have finished arranging your hardware room and now it’s all just about installing and turning on the equipment, make sure that you have a certain set of documentation and regulations that is stored in close proximity to the equipment being serviced:

• power distribution schemes
• lighting schemes
• air conditioning routing schemes
• regulations and dates of commissioning of batteries, air conditioners, generator, ATS, grounding devices.
• cross over circuit
• regulations and a vacuum cleaner for cleaning the premises: you should get a fairly airtight room, then practically no dust will be taken from there over time, and the garbage left by the engineers will appear regularly - caps from optic patch cords and SFP modules, twisted pair strands, rolled nuts and bolts - with all this excellent construction inexpensive vacuum cleaner (kercher for example)
• schemes for fiber-optic lines arriving at the object with reflectograms
• literally everything must be signed and marked - any element of the distribution of power supply, cabinets and doors with signs for the purpose and hazard class, responsible, evacuation schemes on a glowing in the dark basis; if you have a ground bus running around the perimeter, then it should not be painted over and marked as a grounding device. At one enterprise, a friend of mine required technical supervision to paint the edge of the tire in red and green stripes.

Why didn’t I write anything against noise?

It makes no sense to deal with noise with unprecedented racks with soundproofing panels ... There can be only a tishunu in three ways:

• unplug anything that makes noise
• adhere to the rules for choosing a room for a car room described in this article and the noise problem will disappear, provided that jobs are found outside this room
• to some extent, it will help enable quiet mode in the server BIOS.

---

Of course, this material can hardly be called useful at 146%. I am not a designer, and therefore I can’t give you exhaustive links to materials on which you need to prepare yourself for the tasks described in the article. I suspect that even the designers do not keep in mind all the SNIP, VSN, NTP for the building and premises, PUE, and regulatory documentation from the Ministry of Communications / Information Communications, Roskomnadzor, Ros / Technical Supervision. I know almost nothing about labor protection - this is generally a separate issue. But I tried to talk about where to stop and start to get a grasp, so as not to be mistaken.

You can start with this document and try to understand what you need to adhere to. NTP 112-2000 RD 45.120-2000 - documents with a bunch of links to everything that you need to rely on when building a communication center. From it, the truth is useful to everyone, probably, only a few points:

• ВСН 45.122-77 - design of artificial lighting for a communications enterprise
• VSN 116-93 - it may be useful, but a rather old instruction on the design of linear-cable communication structures
• VSN 332-93 - Instructions for the design of electrical installations of enterprises and telecommunication facilities

And the message that I just could not find where to stick:

• When buying any cable, electrical product, cabinets, cross equipment - choose with the CCC series certificates or the declaration of conformity in the field of communications - the closer to the connection, the better. For many things, of course, certificates / declarations in the field of communications are not and are not required, but then at least compliance with GOST. On an electric cable, for example, there is GOST, there is TU and a product of economy in production ... Nothing but GOST is worth taking - you will find yourself on underestimated characteristics and compositions of the material.
• Just quality materials. For example, baseboards do not require certification or compliance in the field of communications. But take the krone skirting board and another “china”, in a couple of years you will see that krone still hangs on the mounting tubes, and the “china” is not dried up and friction enough to stay hanging where it was stuck. A little careless attitude, and you will know the difference in the strength and elasticity of the plastic. I myself came across all this only this year, when I had to disassemble the old cross with both noname manufacturer and krone.

Probably many will say that they don’t need all this in server rooms, especially in small business. You're right, but you have to strive for something? Choose the limits that you are ready to reach and achieve. In any case, the norm from the field of communications seems to me the most adequate and reasonable.

In the next part I will try to explain the principles of equipment selection and placement, as well as how to connect to low-voltage networks and power supply.

All comments on spelling, spelling and typos please in private messages.

I will also be glad if you share what solutions you used when arranging the infrastructure.

PS: please forgive me for the lack of tabulation - it is in the editor, but it is not transferred to the post by the parser. I could not solve this problem.



Part 2 published.