Karmic incompatibility and other vicissitudes of the fate of a service engineer

We recently told a couple of instructive stories about how things can go awry, no matter how much the straw grows. We recalled another incident, at the same time funny and extremely dramatic, especially for the direct participants. Although over time, they already remember those nightmare hours with a laugh.

We have one wonderful customer, let's call him Customer. And engineer N. worked for our company. An intelligent engineer, competent, skilled in many things, successfully worked with a wide variety of clients. And according to the decisions of one of the network vendors, he had quite rare expertise at all - it is not easy to find such specialists in the market, even for impressive money. But when N executed the Customer’s applications, everything invariably ended in long and painful downtimes. Moreover, each time it was not a lack of knowledge of engineer N, but some omissions, random errors, oversights. As a result, the Customer asked us to replace this engineer. We repeat: the engineer was good, and there were no problems with any other clients. But as soon as it came to the Customer ... The devil knows - it is clear that they had some kind of karmic incompatibility.

I had to close N access to all Customer applications and appoint another specialist to this site.

Thus, at that time we had two engineers who were engaged in the systems of this company. The piquancy of the situation was that once, when it was necessary to work on the next request of the Customer, one of them was on vacation, and the second fell ill. As an exception, we suggested that Engineer N, already familiar with the organization’s systems, do everything. The customer agreed. There is a problem - it is necessary to eliminate. Restored N access, he completed the application. Everyone was glad that everything went smoothly. But literally two hours later there was an angry call: Engineer N did not control the segment of mobile numbers, which caused him to disconnect ... Again, N was denied access to the infrastructure of this client.

Time passedstorms rebellious impulse / Scattered previous dreams Customer's infrastructure grew at a very high pace - about twice in a year. He became cramped at his local data center. The equipment could no longer fit, and the load on the network was extreme. We decided to move to a new spacious data center. The migration project has been entrusted to us. For six months we have successfully worked it out, provided for everything, prepared everything.

By the approved date of the move, we have provided a list of employees who will perform the work. The engineer N. also appeared there. The customer looked and doubted: “Again there will be problems.” However, he agreed, because the amount of work was such that a group of engineers had to work tirelessly at the facility continuously for two days. We agreed that N will participate only in the initial installation of equipment, and other specialists will take up and configure the main systems.

It should be noted here that the launch dates for the new data center were very tight. For every hour of downtime after the deadline, we were facing seven-figure fines. In general, we had no right to make a mistake or delay.

We arrived at the old data center, dismantled the equipment in two hours. They brought the first batch to the new data center and began to mount. To raise critical business systems, they quickly switched equipment to a virtualization system, assembled, launched, checked, and waited for the links to appear outside so that the business could start using the appropriate applications. That is, everything is ready, we are waiting for the completion of the installation of telephone gateways. It remains two hours before the deadline, after which huge fines begin. Everyone is very tense.

A sufficiently large IBM X-Series server was allocated for the quick transfer of the virtualization system. This was a necessary measure, because the customer did not have a spare data center during the move, and he asked us to come up with something. In about a week, we were able to find the only server suitable for the number of cores to run the business system. There was a lot of RAM on the server, we still scraped through the gimbal and increased the amount of memory. By a strange coincidence, this server in the new data center was in the same rack where the telephone gateways were mounted. And engineer N. was engaged in the installation.

And then, all of a sudden, from a height of 38 units (i.e., more than 1.5 meters), he drops a 10-kilogram telephone gateway clearly onto the virtualization server, on which the necessary software has already been lifted and configured in anticipation of external links .

In the hall - deathly silence. Next to the counter was a cart with a monitor for setting up a server. The monitor has gone out. A scattering of emergency lights came on, and the server went into reboot.

Silence exploded with mats, Engineer N was removed from the engine room and sent to his ancestors for a walk. We started checking the server. It did not start from the first, second or third time. The alarm indication on all processors and memory bars flashes. They suspected that the motherboard had cracked, because from the blow from the server a cover flashed off, which is fastened to the latches. They were afraid that the server would not be able to start at all. And then, if you use the whole team, it will take about 20 hours to assemble, switch and verify the new server. And every such hour will cost the company a fine of millions of rubles. It was just painful to look at the project manager.

In about an hour, the guys from the service center went through the server by components and using the trial and error method found the failed components. Several memory slots were damaged. They were replaced, the rest were reinstalled, all processors were reinstalled and mezzanines were reinstalled. We checked all the connectors, twitched everything, moved it. The server has started.

We started to raise business applications again. Initially, the server resources were close to them, so I had to artificially limit performance. By the time the applications were uploaded, external links had already been raised, and the data center was put into operation two minutes before the deadline.

Have time.

“Do you have many such engineers?” You ask. No, N was one of a kind, and therefore was remembered for years. But we still made conclusions: 1) not every professional will suit a particular customer 2) karmic incompatibility still exists :). By the way, N today works in another company and he is doing well.

Jet Infosystems Remote Monitoring and Administration Department