Weekdays of the data center: unobvious trifles for 7 years of operation. And the continuation about the rat
I must say right away: that rat in the brought server, which we drank tea a couple of years ago after an electric shock, most likely escaped. Because we somehow saw her friend on the circuit. And immediately decided to put ultrasonic repellers.
Now there is cursed land around the data center: no birds sit on the building, and probably all the moles and worms escaped. Worried that the sound could cause the HDD to fail, but we checked that the frequencies were not the same.
The following story is much more fun. We somehow got a piece of iron for a couple of million rubles in a box with tilt, vibration, and humidity sensors. All is whole. Carefully removed the packaging, and the piece of iron is bent. Mystic.
The body is straight in an arc. Beautifully so.
We would not attach importance to this, because the metal body was curved was almost a design way. Beautifully so, without chips. And if it weren’t for other similar glands nearby, we would not even have thought at unpacking that something was wrong. But nearby were the same, only more regular geometric shapes.
Fortunately, the unpacking of such iron is filmed on video (I recommend everyone to have such a habit), so I was able to prove to the manufacturer that she arrived so much. The whole package and neatly bent body - this is not a blow from the movers. Most likely, she was damaged before leaving for Russia.
The vendor says: “Uh, guys, let’s change it for you right away under warranty.” And then an epic ambush awaited us.
The fact is that customs allows us to import such equipment according to documents without the right to export. That is, you can bring it, but you cannot resell it to someone outside of Russia. When we send back a burnt-out power supply, for example, everything is clear. This is a spare part, power supply.
And then I needed to send everything back:
- Guys, look, we are sending the piece of iron back to the manufacturer.
- The whole equipment?
- Is the model such and such?
- Can work?
- We don’t know, they didn’t.
“So this is the whole equipment.”
- Well, she doesn’t work.
- Well, look, the whole equipment of such a model. Without the right to re-export. Do not let it go.
In general, there were a lot of squats before we figured out that we did not export it, but give it back. As a result, I managed to do everything.
There were also shoe stores
At first, many years ago, we had the first automatic, admin’s dream. You load the shoe covers in a pack, she unpacks them, opens and puts them in a position where you just have to step on. Chop-chpok and done.
After about six months, she chewed about a hundred packs of shoe covers and choked. It turned out that there are so many moving parts that it is either necessary to repair it once a month at our load (we have a lot of customer engineers walking around the site because we are a commercial data center), or we need to buy a new one.
The second problem is that then, at one of the regular cleanings, we somehow found a “little blue rag” hanging on the grill of one of our testing stands. The forensic scientist in the person of an engineer from the X-team identified a fragment of the shoe cover. It turned out that this is convenient in the clinic with shoe covers: half an hour was like that - and that’s it. And some engineers can work with iron all day. Shuffling feet. Shuffling a lot. And shoe covers are rubbed into these little shreds that fly around the gym.
Almost immediately, we bought a new shoe store. We took a thermo-boot: this is the machine into which the film is charged, and it carefully shrinks this film over the boot. Beautiful, effective, durable. Less scatters. For a long time she stood with us, but it was necessary to change the shrink film about once every 1-2 hours, because the sole had the property of falling off itself.
At first we thought that we were unlucky, but people somehow solved this problem. But no. They asked Western colleagues - the same story. As a result, they began to think how to do it normally. To return from the gym for new shoe covers is an idea, frankly, so-so. We found industrial cleaners for construction sites and industries. This is something like the tracks along which a shift enters the workshop. The tracks with a bunch of rollers clean everything, and are made so that you want it or not - it will catch and clean. They cost half a million to one million rubles. We poked around and found the same for 200 thousand, but we must poke our feet into it ourselves. It’s similar in size to a shoe polisher. You approach, shove a leg there, she chews it and gives it back clean. Put at the entrance to the data center.
It works fine except for two nuances. The first - it quickly became clear that this was normal for us engineers. But in practice, a variety of people come to see the data center, among which are the tops of large companies. With leather shoes from the ass of the dragon. And they even have a brush that is even more expensive to apply cream on shoes than my training sneakers, they pick up a special pile there. They refused to shove their legs into our miracle device. The second problem arose in the winter: when the shoes are right in the mud, they cannot get everything out of a deep tread. Then you walk around the hall, leaving traces of ectoplasm.
We decided simply. They put a roll shoe rack near. All the same, we need to duplicate everything according to the standard.
A new problem has arisen. Observing the behavior of customer engineers, we saw the following picture: they first poked their feet into the cleaning machine, and then clung to the shoe covers from a rolled shoe rack. Now they hung a sign that either one or the other is better to be cleaned, but if life principles forbid you to clean shoes, then wear shoe covers. It seems that the ticket, which was for a couple of days, but stretched out for a long time, was closed. Here is the device:
Two times “ku”
According to PCI DSS requirements, you need to be able to visually distinguish the roles of people in the data center. Not looking at the pass and reading something there, but directly visually how the military distinguishes each other in epaulettes, only even brighter. We decided not to show off and used the good old Chatlan method - color differentiation of pants. Specifically, they began to make badge tapes of different colors. Green as your favorite immediately took our admins.
It sounds simple, but it caused three unexpected effects:
- Retractors were needed to automatically retract these passes when worn (these are the things that themselves regulate the length of the tape). We wrote a statement of work, which took into account all the wishes of all departments. This was a strategic mistake. Color, format, material, non-plastic retractor, metal fishing line to put the logo so that it is sewn into the tape. The pieces were so expensive that I later had to cut the requirements and change the format.
- When the pants differentiation worked, it became very convenient. Contractors have some tapes, external administrators have different tapes, and our administrators have third. It can be seen who has what role. In electrics - only gray, in air conditioning - blue. And then ribbons for drivers were needed (this is a separate role, they can enter the unloading zone, but they cannot leave it except outside). At the same time, drivers do not need a pass. At first we gave them ribbons without passes. Then the security officers decided that it was very strange and degraded the human dignity of the drivers. They have their own military logic, so now drivers come to get a pass right with the tape, but this pass does not let anywhere. From the point of view of security, it turns out a marker that the security of this person has checked.
- One of our engineers suggested making green sweaters instead of ribbons. And sent a rationalization proposal. They did it in half: they left a badge with a ribbon, plus green uniform sweaters really sewed. Now we have the admin uniform. The security guards supported the joke and introduced it into the regulations. Now she is obligatory (pants, shirt, sweater, but the sweater can be removed).
Also, our customers often complained about curved routes in maps before entering our Compressor data center. You clog the address, and the road is shown incorrectly. Visitors ended up taxiing the wrong way, because there is a railway there, and there is a traffic jam behind it, and there is no turning around. At first we wanted to hang signs over the road. There is such a service in the city - under the usual signs place yellow additional ones, they are considered advertising. And the price for them is like an advertisement: on the Enthusiasts highway, one sign costs a million rubles a year. In parallel, we wrote to Yandex, and they even suddenly answered. And they stopped messing up. You can even specify the gate diodes: entry through some, exit through others.
Google, if you read us, then you know: you still have a problem, and we don’t know who to tell about this so that we can be heard.
In the invitation letters, they inserted links not just to the address, but to the address with the route being laid from the user's geo-location. As a result, there was less oversight.
Gobo projectors and other little things
Do you know what gobo projectors are? We didn’t know either. Somehow they thought how to mark rows of racks. The racks themselves, of course, are marked with special quick-detachable markings, but they can be seen from a distance of 1-2 meters. The hall itself is 500 square meters, so there is where to get lost. Therefore, nevertheless approached the marking of the rows. Brainstorm has begun. How to label, what and where? On the floor, on the wall, signs on the ceiling, etc. And then our colleague saw that in Ikea there were abrasive stickers on the floor before, and then light arrows appeared. Well, we decided to reverse the simple way: go to the "Ikea" and turn out one of the projectors to see. It was not possible to get it: while we wore chairs, the seller asked what we were doing. And he immediately helped, saying that it was a gobo. It turns out that this is not the projector itself, but a plate or lens for a color image. This filter is the gobo. One projector costs from 40 thousand rubles (there is a powerful lamp for daytime use), and we have 14 rows in each of the four halls. Therefore, stickers were pasted.
We also have schemes on the walls fading over the years. We exchanged them for laminated, with "sewn in" special pockets for revisorro. In our case, Revisionorro is the chief engineer, whose duties include checking for the relevance of all the schemes located in the data center. So, all schemes should be checked annually and signed by such a revisorro. And the presence of a special small magazine in the pocket of the circuit makes this procedure easier and does not require replacing the circuit itself every three years. Profit!
They did rotary cleaning of the raised floor from the outside. We have regular cleanings, there are techniques and cleaning times. But from the wheels of heavy racks traces remain. They did the cleaning. Now we are nervous: it didn’t become much more accurate in appearance, but glare appeared under certain angles in certain people, well, who have their own felt-tip pens to their taste. Now we’re thinking and looking for some chemistry that will whiten the floor and give gloss. So that even the chosen questions do not have.
Have you seen console racks? These are traveling buffet tables, but instead of drinks there is a terminal for connecting to the rack. So at these cantilever racks the wheels fall off and wedge, like the trolleys in the supermarket. Tired of us incredibly. As a result, the only possible way to reanimate is to buy a new wheel. And it was no longer possible to get the wheels specifically to our models, we interviewed all the contractors. As a result, they themselves designed the rack for themselves, focusing on the convenience of movement around the hall and maintainability. It turned out very well.
There was a story with synthetic socks. There is such a thing - antistatic bracelets. This is when you approach the rack, connect the bracelet to the ground on the rack, and it communicates with the potential equalization system. So, the rack is grounded, but it may turn out that the engineer is not grounded. Colleagues from past workplaces told how they saw sparks on video surveillance a couple of times, and we decided to obligate everyone to use them right under the rules.
From a serious one, there was a situation when all the chillers were cut off at once. Our chillers are not protected by UPS, because we believe in physics, and as a reserve for temperature we have a pool with chilled water. If something turns off, you do not need batteries to power the chillers that cool the water, but just the cold water itself, already ready. Convenient and simple, but there is a nuance. The chillers are equipped with protective automation, which disables them with dangerous parameters of the electrical network. If the input is turned off, we turn on the DGU, and then the chillers are powered by them. Everything would be fine if we did not live in Russia. We have had drawdowns many times, but everything was fine. But once the jump was sharp, first down, then sharply up, then down again - in a few seconds the input parameters changed about 4 times. The chillers shut off naturally. We first tried to turn them on remotely, but they protected themselves very reliably, in an emergency. I had to change the legs to go to the roof and turn them on manually. What is important, according to the TierIII standard, such a situation is a legitimate reason for stopping the data center. We didn’t have a stop, because people on the ground with their heads, and there is a drill with exercises. For this, the UI scammed us just regularly to be sure about the TIII Operational. If anything, we passed the UI recertification at TIII Gold - Operational Sustainability. There is nothing cooler on the Russian commercial market of data centers, except our one with the same achievment For this, the UI scammed us just regularly to be sure about the TIII Operational. If anything, we passed the UI recertification at TIII Gold - Operational Sustainability. There is nothing cooler on the Russian commercial market of data centers, except our one with the same achievment For this, the UI scammed us just regularly to be sure about the TIII Operational. If anything, we passed the UI recertification at TIII Gold - Operational Sustainability. There is nothing cooler on the Russian commercial market of data centers, except our one with the same achievmentData center . I note that it is more difficult to recertify than to obtain a certificate from scratch, since they check the previous period as if they were not in themselves, and much more evidence is required.
There was an interesting incident with cameras. We decided to recalculate the blind spots just in case, drew intersections, built diagonals of viewing angles on the plan, and suddenly found a dead zone of about 30 centimeters by 15 meters right in the center of one of the halls. Narrow and long. In the next room there is none. It turned out that the PTZ camera slowly moved over the years so that it began to show about one and a half degrees to the left than necessary in the extreme position.
Another big case was in a post about the repair replacement of DDIBP .