Work as an “IT cosmonaut”: how we engineers go on a visit to bears and seals



    "Heck. Launch the astronaut ”- the phrase with which our most difficult trips begin. I remember last Christmas about this way we went to repair the ABHM of one state institution. The car froze, and the data center warmed up from this. It depends on whether we fix it or not, as older people across the country spend the New Year (with or without a pension). Saved, but the work was still the same. Then it was calculated and charged. Retirement.

    Since then, we have been called astronauts, because we work in an autonomous area. At least half of the trips is a maximum of an hour for picking and a week somewhere where there are only bears instead of Habr. And even money costs nothing.

    About astronauts


    It so happened that now the whole CROC knows us as a detachment of astronauts. We have two “special forces” visiting teams for particularly difficult tasks - “X-team” colleagues on heavy servers, and “cosmonauts” for us. The X-team travels more around Moscow to the center of the country - nevertheless, heavy iron is poked here unmatched more than beyond the Arctic Circle. Yes, and remote control almost always decides, unless firewalls are configured. Networkers somehow somehow managed to coordinate the operation to connect about 50 points outside the Urals at enterprises by local engineers. And there the specifics are this - it was necessary to make a gap at the entrance to the Internet channel and connect a new piece of iron. As soon as the gap appeared - even telephony does not work. Therefore, if you did not write the instructions in plain language, you will go to visit the bears. Then it turned out.

    And we often go to preventive maintenance, various repairs, server, data center engineers and various related infrastructure things. Chillers and UPSs cannot be remotely worked out, so everyone happens in at least 5 cities in a month. We know the country like the back of our hand the features of airports, dining rooms for staff, all kinds of hobbies, the location of charging sockets in different public buildings - too.

    About my work


    I'll start with myself and how the work is generally arranged. For colleagues, it is arranged in the same way, maybe the details are different.

    My name is Roman Karpov. My specialized education is an electrician in trade and refrigeration equipment, the second higher - a leading engineer in ventilation and air conditioning systems. The team is 5 years old. He started as a technician, before that he worked in the protection of Rosatom (more precisely in the private security company). He served in the army as the commander of a fire department, and planned to go to the military school of the Ministry of Emergencies. At that time, there was no education at all, only a very loud obscenity and combat deployment. By the way, I still fit into the norm: with an attack staircase - to the 4th floor from 35 meters in 42 seconds.

    I had a trial period of 2 months — I didn’t understand anything, I sat, read the documentation for the equipment, and traveled with senior comrades to trips around Moscow. The first combat independent trip was to St. Petersburg - it was necessary to make the data center service, there were difficulties with air conditioning. The thermostatic valve failed, replacement was required, plus there was a complex of works on putting the air conditioning system into operation. Day four is gone. Everything was clear, understandable, the main thing - to take and do with your hands, agree with people to help.

    This was the last exit without surprises. The next 5 years, surprises were every day. There are many departures. For example, in 20 days I visited Rostov-on-Don, Chita, Ulan-Ude, Surgut, Naberezhnye Chelny. There was a couple of weekends between Chita and Ulan-Ude, we went to Baikal with my colleagues, I have long wanted to. Then again arbeiten.


    Pavel Stepanov and I recently drove this to Ulan-Ude. It is beautiful there even on the road.

    The specifics of the team - you need to be able to jump and run at any time. Constantly someone on duty. There are service contracts and accidents. Service contracts are when you need to go round N objects in a limited period of time and do maintenance, repairs, maintenance, and so on. We are able to do this, probably the best in the country. Why and what is needed for this - a little later, when we talk about Goryachiy Klyuch and the delivery of materials there. In the meantime, I’ll talk about applications.

    So, I, as a leading engineer in the field, draw up a schedule of visits, bring it to the team and plan my time. For example, 20 days of the month - prevention in the country (November 2–7 - Chita and Ulan-Ude; November 9–13 - Surgut; 16–17 - Rostov-on-Don; 19–21 - Naberezhnye Chelny; 23–27 worked quietly in Moscow). I come home almost every weekend: business trips are short, 2-3 days usually (if there are planes within a radius of 400 kilometers from the serviced object). The colleagues have a similar schedule, the only moment, for example, if the spouse is pregnant, then we try not to send a person beyond the Moscow Region, and so on. Or when a person just returned from Perm, then according to a new application, the one who was in Moscow yesterday will go, and not he from plane to plane. Although, of course, sometimes it is necessary to do so when accidents happen simultaneously somewhere in different places.

    Day


    In the shift shift, we are engaged in ongoing office work and are waiting for applications. We study the documentation, mess with the reports, disassemble, collect interesting nodes, solder something killed rare, train people. Statistically, accidents should be distributed evenly on a schedule, but in practice this is far from the case. Typically, breakdowns occur in pairs, and at once in two different parts of the country. And not on Monday morning, no. Application Monday morning is a holiday. Usually the alarm starts at 23:00 on Friday.

    Next - an attempt to remotely understand what happened, whether to go. The X-team and networkers have a common problem - a power outage or problems in the trunk, so they leave on business in about a third of cases. We have 90% of accidents - the question is directly to us, and that's it.

    “Past” looks like this - on January 3 at 9 a.m. Pasha drove to the object. He asks where the malfunction is. Security shows in the monitor: they say that it turned off after an alarm and does not turn on. Pasha moved his mouse - the monitor turned on. The guard continues: what’s the password then? .. In general, it was not clear to them who to call, everyone just mumbled — so they put us a ticket.

    So, the application. A responsible person is assigned to the application. For example me. If there is an SLA per day (most often there is), you need to look at what's with the ticket. For the most distant and important flights, sometimes we keep tickets already bought with an open date, but more often you just need to order. If the office works, the resource allocation team can help, if not, the order is usually made already in a taxi to the airport. Another warehouse - urgent in our building, and a large one with rare things - in Khimki, specials from there can bring you what you need to the airport. But these are isolated cases, most often - 12 kilograms of tools, a laptop - and forward.

    If on-site operators do not run in circles shouting "everything is lost" (but only in one word) - we will live out the equipment. We, unlike, again, from the X-team, you can do anything. In the server park, we replace only the modules, and we solder, assemble one of the other (really - often we disassemble non-critical systems and non-traditionally use their components for repairs in cities where parts are quickly missing). Virtually none of the vendors (with two exceptions, in particular, when the customer bought the spare parts from the vendor) can dictate the conditions for repair and support - the systems are not removed from the guarantee for our actions. If we were waiting for specialist vendors in engineering, factories in the country would stand.


    Preserved photo from my first object

    My schedule is usually known (approximately, if there are no major accidents) a week in advance - there is no point in planning further. It is possible, but, as experience shows, it is not necessary. You plan so a couple of weeks - then something will happen. It’s almost a sign, no one plans to go on a trip from the weekend. And it’s better to take return tickets on the last day - from the same opera.

    The year is not scheduled, the quarter too. Heck. Right now, while I am writing, the application has arrived. In Nizhny Novgorod, the cooling system does not start. Thank you all, we went to work.

    ...

    That's something like that. Exit work - home, family, I manage to do it normally. There are no children yet. Will - I will go to the office, travel around Moscow and St. Petersburg (there are such options).

    Departures


    Usually long trips are about a week. In Khabarovsk, I was with a colleague for two weeks. Hard. Hung in the balance of a data center of a large cooling bank (in summer). There was a great risk that the processes would get up. In place - very difficult freon leaks from the circuit, it was necessary to find and fix them. The story is this: the chiller was a one-piece unit with an air-cooled condenser and a free-cooling water cooling evaporator. 4 compressors, 2 circuits, in both 407th multicomponent. And this is such a freon that according to the norms it is refilled only in its entirety, because otherwise the proportions of the components will be unbalanced. Process: the machine stops, freon is evacuated from the system to the tank (it is impossible to the atmosphere). Then, using the dry nitrogen method under excessive pressure, the system is filled to the recommended values ​​and work is underway to find leak points. We go we look at the pipes - we carry out a detailed inspection of the nodes using special tools. Leaks are detected. Then nitrogen is evacuated from the system. The pipes are soldered. Now you need to check how the leaks were closed, that is, the tightness according to the norms. Again check with nitrogen under pressure. Maintains the terms of the standard and there is no pressure drop - it means it is tight. An act is drawn up and endorsed. Again we bleed nitrogen. Then the system is evacuated and refilling with freon and compressor oil is done again in case of volume loss. Then the machines come back into operation. sealed. An act is drawn up and endorsed. Again we bleed nitrogen. Then the system is evacuated and refilling with freon and compressor oil is done again in case of volume loss. Then the machines come back into operation. sealed. An act is drawn up and endorsed. Again we bleed nitrogen. Then the system is evacuated and refilling with freon and compressor oil is done again in case of volume loss. Then the machines come back into operation.

    In general, Seryoga, our guru, somehow saved all the courts, but you can’t talk about it yet. He also had a long trip to the fire alarm of a 22-story building, it was necessary to integrate it with all the systems. For example, with elevators, evacuation, alerts and about a dozen subsystems. And to make friends with each other these subsystems. He then had three weeks of separation from the family and the dining room. He says the second was especially hard.

    Communication when how. Somewhere normal, somewhere not. For example, in Magadan, Wi-Fi was through the ticket office of the hotel - they printed the password on tape for 1 hour or two. Moreover, such a picture in all hotels of the city - that is, in both. There is still local 3G, but it still jumps through the satellite, so everywhere we try through Wi-Fi. Although in the same Surgut it was easier to distribute from a mobile - faster and more productive than from a local network. Sims are corporate, paid by the company. But not everywhere our operators catch, so sometimes we take our own backup ones. For example, in Noyabrsk there was no hotel network, and corporate MTS was bad. Locals say that Mega works well - I bought Simka on the spot and perfectly the whole business trip for 200 rubles surf.

    If we have a laptop with us, then on the whole we are able to connect to local closed networks with technical traffic, if it’s really bad with the connection.

    Stripping foreign jambs


    Sergey appeared at our doorstep in 2006. He graduated from the Voronezh State Polytechnic in Radio Engineering. He worked in the regional clinical hospital, served CT and MRI. Then he was engaged in security in Voronezh. It was necessary for family reasons to move to Moscow, he began to look at what companies on the market are according to his profile. I suddenly found out that a lot of people are required for intelligent buildings (this is just an engineer at the junction with IT). He came to us, went through an interview, got into our team in 2007.

    Seryoga often gets objects where he needs to be “cleaned up” by another team. Usually how - the contractor works, promises the golden mountains and commissioning right on time. Maybe we have such specifics, but “right on time”, it seems to me, is a professional myth. A couple of weeks before the delivery (or 2-3 days in general), the customer realizes that if there is no infrastructure physically, then they probably will not be able to return it. And their name is us. Here all the words are remembered, and about the "astronauts", and about the "special forces", and why only you will not learn about yourself good. In 80% of cases, we decide, but it also happens that the customer is surprised:
    - Three days have passed! Why not ready?
    And we are sitting and thinking - really, why didn't the three of us build the missing buildings?

    Or here's another example, we completed there after the official launch. There was somehow work on commissioning systems - there 10 people after 9 months did not give birth on time. They handed over the object, because when the fire inspector arrived, the customer proved that everything was smooth, showing an army savvy. They put the fighter on the roof, he watched with binoculars what the inspector was doing, and he told the “Soviet automatic equipment” to the other fighters in the switchboard what to turn on by radio. As a result, they turned on one of the pumps, but forgot to turn it off. Water filled the roof, and then began to spill onto the street. There is a center, cafes, all things - the people began to panic and google about the rainy season.

    Far East


    Pavel came to us in 2007 after the army. Then we just recruited interns, and he got a job. He says, I remember, they gave a manual of some kind, like, words familiar, my topic is professional, but what kind of equipment is it - for 5 years of the university no one even showed me this closely. After the trial period, the first “combat” business trip — we went with him to Rostov to launch a store. Two flights were combined at the airport, and on the spot it turned out that they forgot to load our luggage (everyone who flew the second plane). And we had a tool there. The customer says - most importantly, we arrived, we’ll give everything. And they gave. Baggage was taken only in the evening, but it was not very useful - they worked out what they have.

    The most remembered business trips are Magadan and Khabarovsk. On Friday we arrived at the office, on Monday we flew to Khabarovsk. The difference is 8 and 7 hours. On flights - 9 hours there. Then another 8 hours. The work is the same - air conditioning, UPS, diesel. It is necessary to prevent for the data center and server.

    When you fly from the Far East, the day increases by 6-8 hours. He got up at 5 in the morning, and flew to Moscow at 5 in the morning. But this is still good, usually the flight is like that - at 3 o’clock in the afternoon it flew out, in Moscow at 3 o’clock in the afternoon too. If you are not lucky - an accident, and you need to go to the office. Or vice versa, when you fly from Moscow in the evening, then in the morning on the spot. I didn’t sleep normally, and already morning, I have to work again.

    Pasha celebrated his anniversary in Novokuznetsk at the facility. It was necessary to change the equipment there, the customer said that he was finally delivered (hooray!), But it turned out that he was guided by the report of the transport company that it “will be tomorrow”. Waited another week. All night he was woken up with congratulations.

    By the way, about the frost. I somehow got the most frosty day in Kemerovo, when it was either -42 or -43. I knew it would be cold, but I had no idea what it really was FROST. After this incident, we agreed with the resource allocation service, we bought new Bavaria jackets, warm. It’s easier with them, before that it was still necessary to pry off a “lint”.

    Took it - do it until the result


    Pasha (the other one) firmly joined the team after he proved that “There are no impossible tasks”, and he will solve the problem until victory. This character trait is very important for the engineer.

    For example, at my place of departure it was like this — literally last month, the opening of a new large store in Surgut. It is necessary to climb 10 meters to put some communications. But there is no ladder. They took and rented a picnic - a lifting mechanism. At this time, the outside temperature has already reached -20. They arrive, they start to warm the diesel engine, they say - "climb in." Me:
    - Guys, you’ll drive the hydraulics, it’s frozen, I guess.
    - No, everything is fine, let's work.
    - You run, raise and lower, we'll see.
    Exactly, it comes to the middle - the hose comes off. Everything, tragedy, a repairman to wait. They had to fix the picnic on the spot and work in the shopping center. Pasha’s stories are not so bright, but the idea, I think, is clear.

    In general, Pasha worked as an assembly and welding production technologist at one of the defense enterprises. Dealt with large welding and not only welding machines and assemblies. When I first got there, the worldview turned upside down what a workplace might look like. After a huge workshop at this enterprise, so half-dark where 3 pensioners worked (not the best years were, the end of the 90s), a normal engineering team with modern tools - it was like a toy store for him.

    Before us, I was looking for work with difficulty. "We need young cadres, but go get experience somewhere."

    Pasha also noted that he was driving to an object where there was no electricity, and all the power supplies were dead. It was necessary to carry batteries, but he had just removed the cast. He worked on a par with the locals, repaired everything.

    And on the first business trip, he went for Lake Baikal, to Aldan, but it’s already better to tell in bikes.

    Here's who I talked about:

    Our team: Seryoga Zolotarev, Pavel Alexandrov, Pavel Stepanov, I (Roman Karpov), Anton Timoshenko and our leader Alexey Vorobyov. He used to leave, now he has almost stopped. In comparison with him, from experience we are still young.

    About Anton and Alexei did not tell, they are better themselves. They will write about our stories.

    Also popular now: