As I drove the robot, I almost turned gray and filled the server room with blood
This is the story of one of the most memorable cases in my engineering practice. For obvious reasons, I changed the names, places and some recognizable details so that it was impossible to pinpoint the customer and other participants in the story.
Here is the tape storage (ours was smaller) and the library robot (ours is the same). Chinese not included.
I remember the last days of November were. Already thinking about the end of the working day, I was planning my evening, when suddenly they informed me that in the glorious Siberian city of N our tape library had broken. The spare part was immediately sent by the transport company, but now it has been 3 days since it was still on its way. The transport company inexplicably explained and grunted on the phone, and the customer became seriously joked. The forecasts were uncertain, so it was decided to carry another spare part on our own on an airplane. A warehouse employee handed me a ten-pound box, pasted over with barcodes and stickers, and happily slapped me on the shoulder with the words: “Just don’t think to drop it in your luggage - remember.”
The box, of course, bothered me, but a package of powdered milk that I was handed to me by the project manager at the last moment inspired me no less concern. “They have some troubles with milk there now ... because of bad weather or something ... local guys asked them to bring 2 kg. Isn’t it difficult for you? ”He said. From the expression on his face and the characteristic gesture of his palm, as if covering my mouth, it was clear how he now wanted me to be accommodating or even dumb.
This story happened several years ago, when I was an engineer, when I flew around the country a lot, and, waking up in another hotel, sometimes I could not figure out what city I was in right now. 4-5 business trips per month left their mark on lifestyle. And the personification of this way was my backpack, always ready for the next trip.
I got out of a taxi, and suddenly a police officer stopped me at the terminal building. “Your documents!” He spat out, and for some reason a feeling of deepest satisfaction spread over his face. An evil dank wind blew, preventing me from sorting out my pockets and bags. I, as I could, extracted my documents and handed them to the guard of law and order. In theory, a young man who travels only with a laptop, a large box and a bag of white powder should not cause any suspicion, so I portrayed sincere bewilderment on my face. “What's in your box?” He asked indifferently. Apparently, that in the package was nothing more than milk powder, he already determined his professional instinct, and therefore the package was no longer of interest to him. But the box has not yet succumbed to his insight. “This is a robot,” I said.
At that moment, I had the feeling that his indifferent face had been replaced, for it was lit from within with genuine interest, as if in the bowels of his hat with a cockade an additional module with the function of human curiosity had been connected. I understood that I was in a hurry with the answer and somehow it was necessary to somehow correct the situation. “This is a robot ... for the library,” I added even less confidently, continuing to observe the connection of the remaining modules of maliciousness, gloating and greed. “Speak, a robot for a library ... a librarian robot or something?”
I advanced to the landing zone, remembering how I had just finished with the cockade. “It worked out,” I thought, “but I could have missed the plane too.”
“We remove the baggage on the tape, jackets, belts and shoes, we spread the phones ...” - a woman in uniform commanded imperiously and soullessly. Placing my belongings on the tape, I myself, without sound and squeak, went through the frame of the metal detector. It was evident that the box with the robot does not give rest to an employee of the airport security service, who drove her back and forth through an x-ray. “What is this?” She finally said when all the comparative images in her head had run dry. I wrinkled a bag of milk in my hands, thinking over my tactics, so as not to repeat the cockade episode now. But suddenly her colleague, sitting next to her, glanced quickly at the monitor and said: “Yes, this is a robot for the L200 library. This week they’re being driven for the third time, the season of breakdowns has begun. Come on in, young man! ” There was no limit to my amazement
Rusty "Volga" with a creak rushed me from the airport to the hotel. Of the thirty people of dubious type of taxi drivers who met a night flight in front of the local airport, this seemed to me the most restrained. In all his stoop and smoky appearance, some kind of decency and even nobility were felt. However, the appearance of a taxi driver is often misleading, and he wondered why I live in Moscow all the way, but I still don’t know how things are with Alla Borisovna Pugacheva. “Do you live with her in the same city, and don’t know ?!” - my taxi driver didn’t stop. He did not like me for ignorance, but at the same time he did not want to face the front of the capital’s guest. We drove famously to the door of the hotel with a squeal of brakes, completing a steep pirouette, unacceptable for a motor unit with such a venerable age. The car door seemed to swing open itself, or simply fall off.
I entered the building, looking at the tastelessly decorated walls and columns, all with some kind of gray coating, like dust. The building was an old building, possibly a former manor. At the reception, the girl blankly painted her nails. Confident of my rightness as a client, I resolutely took out my passport and slammed it on the counter. Madame reluctantly distracted from her nails, scornfully looked at my passport and grunted: "We are not giving in debt." “What ...?” - in my head there was an interjection and other signs of extreme indignation. “I have a reservation here,” I tried to be polite. “This is a casino! The entrance to the hotel is from the other side of the courtyard. ” Who knew that both institutions have the same name? And then I managed to think that all this time the taxi driver was carrying me, apparently thinking
The hotel room met me with chic curtains to the floor and a musty smell of dampness. I sat on the edge of the bed located in the wall niche, gently lowered the treasured box with the robot in front of me and greeted the spider in the corner in surprise. Work at the customer was scheduled for 9 in the morning, the clock was 6 local. The engines of the IL-62 and other delights of a many-hour flight were still buzzing in my head. I mentally figured out the prospect of sleeping a couple of hours and waking up fresh, and made the right decision just not to go to bed. Physiologically, my case tended to late dinner, even night kefir, so I adjusted our signature green tape on the box and went into the shower.
When by 9 am I arrived with a robot and milk powder at the customer, a representative of the IT department met me at the checkpoint and said that his name was allegedly Semyon. Semyon spoke incessantly about everything around him, now and then wiping his mouth with his hand. By his appearance and rustic manners, it was clear that in the department the role of the person he met was given to him more often than others, as the most illiterate specialist, but a very sociable person. Without a doubt, Semyon evoked trust among people, and, as it later turned out, had a wonderful, almost indomitable appetite.
There was fuss on the floor, and it smelled of paint. Gesturing violently, Semyon explained that this is a new building and now a second platform is equipped here. And then my cell phone rang. This was a project manager. “Have you already reached the customer?” Listen, help them there. We are now setting up a cluster for them. Our Lesha has been with them for a week. Something does not work. Isn’t it difficult for you? ”
Accompanied by Semyon, I entered the gym. Boxes of various sizes with the inscriptions Dell, Cisco, EMC were placed everywhere. Many of them have already been opened, wires, rails for mounting equipment in racks, piles of documentation and electricians lay around. Electricians didn’t just lie, they fumbled with wires under the raised floor and quietly quarreled. At the center of this creative chaos stood the senior specialist of the department, Nikolai. He held a workshop. Hiding from the meeting, our Alexei languished behind a pile of boxes, squinting at the monitor of his laptop and secretly having breakfast with a bun. Semyon also wanted a bun, but he needed to participate in the meeting, so he just puffed.
From the first words of Nicholas, I realized that the time to start the complex was running out, less than a day. With the advent of the second site, management has set the task of reorganizing one of the most important information systems. The database - the core of this information system - was supposed to be clustered. Now it was stored on two storage systems, between which synchronous data replication was carried out. And cluster software monitored the state of the entire complex and could automatically respond to a failure if something bad happened on the main site. The preparatory work was almost finished, but a lot of time was wasted searching for and eliminating broken optical links between the sites that the contractor was laying out. And still had to finish the installation of servers, update the operating system and microcodes on them, as well as test everything. The launch of the complex into commercial operation was scheduled for the night - the only time when you can stop critical financial systems.
I don’t remember how that day went. I only remember that my colleagues immediately told me that the tape library in the light of the circumstances would wait, and now my hands are more important in another matter.
All day we mounted servers, twisted nuts, laid cables, connected adapters. During the breaks, we kind of even had lunch and went for a walk in the forest. Then I installed the operating systems, rolled patches, conjured with microcodes. Sometimes microcodes conjured with me, and in order not to doze off, I recalled how the flight attendant expertly juggled a kettle on a plane, complaining of turbulence, and distributed tea to feared passengers with boiling water. Night fell, it was clear from the window that a large orange moon had risen above the forest.
“Hour X has arrived,” said Nikolay cheerfully, yawning, briefly conferring with someone on the local telephone. “It has arrived, it has arrived,” they repeated to him from the pipe, which he had not yet had time to put down, and giggled. These were the “applicants” who were also on duty that night and upon completion of our work had to make sure that everything worked.
On the clock, the hands showed midnight. We came to the most crucial moment when we had to switch data replication to another storage system, wait for the synchronization to complete, and raise the cluster on the second site. This process was fully commanded by Alexei, a well-known specialist in storage systems in our circles. Some even knew about his irrational habits ... He smacked his lips and copied the volume identifiers to the script, referring to the list that the customer had provided him. Half an hour later, when this painstaking work was completed, Lesha touched the enter key for fidelity, and making sure that it was real, he pressed it. Everyone waited anxiously for something, and general silence was broken only by the crackling of fluorescent lamps. I was bent under the raised floor and in reality I had a dream, but I don’t remember what it was about.
I woke up because the synchronization was over, but for some reason everyone was not happy. Cluster services did not rise. I was called from a dreamland to look in the logs. Through the fog in my mind, I began to realize that the problem was with the disk subsystem. The logical volume group was imported with an error and the data was not available. The reasons were not yet clear, but the prospect of a rollback to the old configuration began to emerge, i.e. Plan b". At seven in the morning the system was supposed to be operational, so there was no time to think. Then I could not even imagine that the tragic events of that night were just beginning to unfold.
I meditated for another five minutes and offered to roll back to the old system. Reluctantly, my colleagues agreed, dejectedly thinking that they would report to their superiors in the morning. We switched the replication direction and gave the command to import the disk group on the old server. I had already calmed down and dreamed of getting to the hotel bed with a sense of accomplishment. And suddenly I heard an exclamation: "And here it also does not start." And inside, everything broke off ...
The fog completely vanished in my head. They say that in an extreme situation, the human body is mobilized. I furiously figured out what could be done. On the storage side, everything was in order. Having convinced myself of this, my colleague Alexei waved his hand, signed his impotence, and went to watch the movie on his laptop. I already spoke about his irrational actions. But formally, there was nothing to complain about.
The cause of the problems loomed quite clearly. The customer previously added additional volumes to the disk group, but they forgot to add them to the list of disks that are replicated. When the cluster started, the disk group was imported, but metadata was updated only on part of the disks, and the new ones remained in read-only mode. Now the configuration of the disk group was completely inconsistent, and the data is not available from either the new storage system or the old one.
When it comes to financial data of a huge organization, their loss is tantamount to disaster. I immediately remembered a similar story when at the dawn of my engineering career I was renovated by one customer - a serious structure that was legalized in the 90s. They discussed that their admin inadvertently deleted something. “And what happened to him?” - by the way, I asked then. “Well, while he is sitting here in the basement, the third day already. We are waiting for the general, he will decide what to do with him, ”the shaven-headed employee with a heavy chain on a bull’s neck answered me in all seriousness.
Have you ever felt hair moving on your head, about to turn gray? Pretending that we had lost the database, I didn’t even want to think what would happen in the morning.
In situations such as ours, you should immediately call the tech support of the manufacturer. She, at such an hour, is understandably English-speaking. I scribbled some keywords for myself and dialed a number. Two beeps and the English baritone answered me. I immediately told who we are and what happened to us. Baritone patiently listened to my monologue and asked: "What is first name?" I was dumbfounded, but, having gathered, with thoughts, I answered. Then questions followed about last name, customer name, city, location, serial number and so on ad infinitum. Time was running inexorably. After completing the survey, the baritone asked me to write down the reference number and told me to wait for a specialist call. The specialist did not call me, after 5 minutes he wrote me an e-mail. He wrote that he accepted my application for work, but his shift was over, so I need to call the dispatcher again and ask to assign an application to another specialist. It is not clear why some people think that they do not like to work in Russia. This seems to be a fallacy.
And suddenly a miracle happened. Another specialist called me back, but he was from India. The second global technical support center was there and they themselves accepted my application. I was glad of him as a father, but my joy did not last long: we did not understand each other. It was clear that he seemed to speak English, but it wasn’t easier. Sometimes some of the sounds of his speech led me to a distant thought of what he meant. But in the end, our communication came down to correspondence. I strained my memory, recalling all the best moments from Indian cinema and trying to let him feel the depth of our grief. As time went. India, as expected, sang and cried, but could not help us either.
At 4 o’clock in the morning, we realized that there were no other options but to recover from the backup. All eyes were on the tape library. Nikolai even crawled in front of her, wiping the floor with a tie. “How much will the base from the ribbon unwind?” I asked the local baker. “About three hours, probably,” he answered uncertainly, “but we never checked.” We urgently needed to replace the robot in the library, because for three hours we had a big stretch. Never before have I opened the packaging so fast and screwed on screws. The robot, covered in blood, was over, he fell into place. I poured blood over him, injuring my finger because of the rush, but there was no time for bandaging. The toggle switch flashed and the library was illuminated from within with light, the robot began to move and began calibration. We crowded around the library and looked through the window, mentally helping him move and read barcodes from cassettes. He did it slowly, with pleasure, and we, clinging to the glass, gritted our teeth. “Initialized!” Said the bootman, referring to the library. And he added: “I am starting recovery.” The robot thought for a second, then briskly darted to the desired row of cartridges, confidently grabbed the desired one, turned around the steering axle and drove towards the drive.
At this time, Alex in the corner behind the boxes admired the film and another bun. The film went without sound, but the excitement on the face of my unlucky colleague was genuine. Engineers often carry with them on long business trips a selection of favorite films that are good even without sound. For real art, such as films with Charlie Chaplin, can be watched this way.
Suddenly, a dull thump was heard from the library. The robot brought the cartridge to the drive, but could not determine where the hole is in the drive, and began to beat against its wall, trying to insert it somewhere. It was a complete fiasco, and despair gripped you. Unfortunately, mechanical spare parts are extremely sensitive to transportation, so there are times when a new spare part is immediately classified as DOA (dead on arrival).
Cold sweat rolled hail down my forehead and clouded my eyes. I slowly pressed the keys, rechecking each one. We had less than an hour before the start of the working day, and all our attempts already seemed in vain, when suddenly a thought dawned on me. “Have you collected diagnostics from the old server for the last time?” I asked my colleagues. “Last night, what?” Answered Semyon, who was hungry and hungry. “There may be a conclusion from the team that reflects the configuration of the disk group before the changes, and we can try to recreate it by hand using this conclusion.”
And now I was engaged in the fact that with my hands I ran commands that re-partitioned the disks into the service area and data area. " “Have you ever done this before?” Asked me, creeping out from somewhere Nicholas. “No, I only read the article, this is shamanism, but we must try it,” I replied. Another time, my hands would be cut off right away if they knew that I had just slammed the entire disk group. But now it was our only chance, so there were no objections. All of us in this room were both sentenced and conspirators. Hope in man never dies.
I finished with the markup and typed a command that, according to the documentation, was supposed to re-create the disk group, but not to initialize the space, i.e. Do not overwrite existing data. “Doing it - do not be afraid, afraid - do not do it,” I recalled Confucius and launched the team. She, to my surprise and joy, was fulfilled and did not give a single error. I deduced the configuration, all logical volumes were in their places, with normal status. It remains to check the availability of data. In our stock was another 15 minutes. A telephone rang in the room for about 40 minutes, but no one paid attention to it. Whoever it is, they have so far had nothing to answer. “Let's start the cluster,” I suggested, and I was surprised at my voice.
In the library, the robot raged wildly about the drive, apparently hoping to starve it sooner or later. Alex stroked his laptop, impressed by the ending of the film. Semyon bored patiently, thinking about whether he had chosen the right profession, and in the interruption dreamed of a chocolate bar. Someone else was snoring behind the bars, jerking every now and then in a dream. The remaining characters focused on looking at the monitor, on which, without embellishment, our whole life was reflected. “It started!” Gasped Nicholas and grabbed the ringing pipe. These were the "applicants", they already washed the ropes and called to say goodbye. “Check the butt!” Nikolai shouted and brushed off his tie.
I sat on the box and smiled happily, thinking how now people from all apartment buildings of this beautiful city are streaming in pea-jackets, earflaps and felt boots. They gather in line to ATMs, and they also joyfully and reliably serve them. And in this whole picture there was some kind of right human happiness.
Everything worked for us, and 2 minutes before the start of the working day, the system was started. My colleagues silently thanked me, shaking my hand and shaking my head. Everything that happened that night, we were going to carry away with us, not giving out to anyone.
The Indian continued to write to me, still offering some crazy ideas. At this time, a door opened and a sleepy physiognomy popped into the room. It was a security guard with a checkpoint. He said: “There the courier arrived. He says he brought some robot. It’s like a librarian robot ... ”
A great new day began.
Here is the tape storage (ours was smaller) and the library robot (ours is the same). Chinese not included.
Part 1. Box
I remember the last days of November were. Already thinking about the end of the working day, I was planning my evening, when suddenly they informed me that in the glorious Siberian city of N our tape library had broken. The spare part was immediately sent by the transport company, but now it has been 3 days since it was still on its way. The transport company inexplicably explained and grunted on the phone, and the customer became seriously joked. The forecasts were uncertain, so it was decided to carry another spare part on our own on an airplane. A warehouse employee handed me a ten-pound box, pasted over with barcodes and stickers, and happily slapped me on the shoulder with the words: “Just don’t think to drop it in your luggage - remember.”
The box, of course, bothered me, but a package of powdered milk that I was handed to me by the project manager at the last moment inspired me no less concern. “They have some troubles with milk there now ... because of bad weather or something ... local guys asked them to bring 2 kg. Isn’t it difficult for you? ”He said. From the expression on his face and the characteristic gesture of his palm, as if covering my mouth, it was clear how he now wanted me to be accommodating or even dumb.
Important retreat
This story happened several years ago, when I was an engineer, when I flew around the country a lot, and, waking up in another hotel, sometimes I could not figure out what city I was in right now. 4-5 business trips per month left their mark on lifestyle. And the personification of this way was my backpack, always ready for the next trip.
Part 2. Librarian
I got out of a taxi, and suddenly a police officer stopped me at the terminal building. “Your documents!” He spat out, and for some reason a feeling of deepest satisfaction spread over his face. An evil dank wind blew, preventing me from sorting out my pockets and bags. I, as I could, extracted my documents and handed them to the guard of law and order. In theory, a young man who travels only with a laptop, a large box and a bag of white powder should not cause any suspicion, so I portrayed sincere bewilderment on my face. “What's in your box?” He asked indifferently. Apparently, that in the package was nothing more than milk powder, he already determined his professional instinct, and therefore the package was no longer of interest to him. But the box has not yet succumbed to his insight. “This is a robot,” I said.
At that moment, I had the feeling that his indifferent face had been replaced, for it was lit from within with genuine interest, as if in the bowels of his hat with a cockade an additional module with the function of human curiosity had been connected. I understood that I was in a hurry with the answer and somehow it was necessary to somehow correct the situation. “This is a robot ... for the library,” I added even less confidently, continuing to observe the connection of the remaining modules of maliciousness, gloating and greed. “Speak, a robot for a library ... a librarian robot or something?”
Part 3. Search
I advanced to the landing zone, remembering how I had just finished with the cockade. “It worked out,” I thought, “but I could have missed the plane too.”
“We remove the baggage on the tape, jackets, belts and shoes, we spread the phones ...” - a woman in uniform commanded imperiously and soullessly. Placing my belongings on the tape, I myself, without sound and squeak, went through the frame of the metal detector. It was evident that the box with the robot does not give rest to an employee of the airport security service, who drove her back and forth through an x-ray. “What is this?” She finally said when all the comparative images in her head had run dry. I wrinkled a bag of milk in my hands, thinking over my tactics, so as not to repeat the cockade episode now. But suddenly her colleague, sitting next to her, glanced quickly at the monitor and said: “Yes, this is a robot for the L200 library. This week they’re being driven for the third time, the season of breakdowns has begun. Come on in, young man! ” There was no limit to my amazement
Part 4. Manor
Rusty "Volga" with a creak rushed me from the airport to the hotel. Of the thirty people of dubious type of taxi drivers who met a night flight in front of the local airport, this seemed to me the most restrained. In all his stoop and smoky appearance, some kind of decency and even nobility were felt. However, the appearance of a taxi driver is often misleading, and he wondered why I live in Moscow all the way, but I still don’t know how things are with Alla Borisovna Pugacheva. “Do you live with her in the same city, and don’t know ?!” - my taxi driver didn’t stop. He did not like me for ignorance, but at the same time he did not want to face the front of the capital’s guest. We drove famously to the door of the hotel with a squeal of brakes, completing a steep pirouette, unacceptable for a motor unit with such a venerable age. The car door seemed to swing open itself, or simply fall off.
I entered the building, looking at the tastelessly decorated walls and columns, all with some kind of gray coating, like dust. The building was an old building, possibly a former manor. At the reception, the girl blankly painted her nails. Confident of my rightness as a client, I resolutely took out my passport and slammed it on the counter. Madame reluctantly distracted from her nails, scornfully looked at my passport and grunted: "We are not giving in debt." “What ...?” - in my head there was an interjection and other signs of extreme indignation. “I have a reservation here,” I tried to be polite. “This is a casino! The entrance to the hotel is from the other side of the courtyard. ” Who knew that both institutions have the same name? And then I managed to think that all this time the taxi driver was carrying me, apparently thinking
Part 5. Sleep or not sleep?
The hotel room met me with chic curtains to the floor and a musty smell of dampness. I sat on the edge of the bed located in the wall niche, gently lowered the treasured box with the robot in front of me and greeted the spider in the corner in surprise. Work at the customer was scheduled for 9 in the morning, the clock was 6 local. The engines of the IL-62 and other delights of a many-hour flight were still buzzing in my head. I mentally figured out the prospect of sleeping a couple of hours and waking up fresh, and made the right decision just not to go to bed. Physiologically, my case tended to late dinner, even night kefir, so I adjusted our signature green tape on the box and went into the shower.
Part 6. Semen
When by 9 am I arrived with a robot and milk powder at the customer, a representative of the IT department met me at the checkpoint and said that his name was allegedly Semyon. Semyon spoke incessantly about everything around him, now and then wiping his mouth with his hand. By his appearance and rustic manners, it was clear that in the department the role of the person he met was given to him more often than others, as the most illiterate specialist, but a very sociable person. Without a doubt, Semyon evoked trust among people, and, as it later turned out, had a wonderful, almost indomitable appetite.
There was fuss on the floor, and it smelled of paint. Gesturing violently, Semyon explained that this is a new building and now a second platform is equipped here. And then my cell phone rang. This was a project manager. “Have you already reached the customer?” Listen, help them there. We are now setting up a cluster for them. Our Lesha has been with them for a week. Something does not work. Isn’t it difficult for you? ”
Part 7. Cluster
Accompanied by Semyon, I entered the gym. Boxes of various sizes with the inscriptions Dell, Cisco, EMC were placed everywhere. Many of them have already been opened, wires, rails for mounting equipment in racks, piles of documentation and electricians lay around. Electricians didn’t just lie, they fumbled with wires under the raised floor and quietly quarreled. At the center of this creative chaos stood the senior specialist of the department, Nikolai. He held a workshop. Hiding from the meeting, our Alexei languished behind a pile of boxes, squinting at the monitor of his laptop and secretly having breakfast with a bun. Semyon also wanted a bun, but he needed to participate in the meeting, so he just puffed.
From the first words of Nicholas, I realized that the time to start the complex was running out, less than a day. With the advent of the second site, management has set the task of reorganizing one of the most important information systems. The database - the core of this information system - was supposed to be clustered. Now it was stored on two storage systems, between which synchronous data replication was carried out. And cluster software monitored the state of the entire complex and could automatically respond to a failure if something bad happened on the main site. The preparatory work was almost finished, but a lot of time was wasted searching for and eliminating broken optical links between the sites that the contractor was laying out. And still had to finish the installation of servers, update the operating system and microcodes on them, as well as test everything. The launch of the complex into commercial operation was scheduled for the night - the only time when you can stop critical financial systems.
Part 8. From dawn to dusk
I don’t remember how that day went. I only remember that my colleagues immediately told me that the tape library in the light of the circumstances would wait, and now my hands are more important in another matter.
All day we mounted servers, twisted nuts, laid cables, connected adapters. During the breaks, we kind of even had lunch and went for a walk in the forest. Then I installed the operating systems, rolled patches, conjured with microcodes. Sometimes microcodes conjured with me, and in order not to doze off, I recalled how the flight attendant expertly juggled a kettle on a plane, complaining of turbulence, and distributed tea to feared passengers with boiling water. Night fell, it was clear from the window that a large orange moon had risen above the forest.
Part 9. Hour X
“Hour X has arrived,” said Nikolay cheerfully, yawning, briefly conferring with someone on the local telephone. “It has arrived, it has arrived,” they repeated to him from the pipe, which he had not yet had time to put down, and giggled. These were the “applicants” who were also on duty that night and upon completion of our work had to make sure that everything worked.
On the clock, the hands showed midnight. We came to the most crucial moment when we had to switch data replication to another storage system, wait for the synchronization to complete, and raise the cluster on the second site. This process was fully commanded by Alexei, a well-known specialist in storage systems in our circles. Some even knew about his irrational habits ... He smacked his lips and copied the volume identifiers to the script, referring to the list that the customer had provided him. Half an hour later, when this painstaking work was completed, Lesha touched the enter key for fidelity, and making sure that it was real, he pressed it. Everyone waited anxiously for something, and general silence was broken only by the crackling of fluorescent lamps. I was bent under the raised floor and in reality I had a dream, but I don’t remember what it was about.
Part 10. Untitled
I woke up because the synchronization was over, but for some reason everyone was not happy. Cluster services did not rise. I was called from a dreamland to look in the logs. Through the fog in my mind, I began to realize that the problem was with the disk subsystem. The logical volume group was imported with an error and the data was not available. The reasons were not yet clear, but the prospect of a rollback to the old configuration began to emerge, i.e. Plan b". At seven in the morning the system was supposed to be operational, so there was no time to think. Then I could not even imagine that the tragic events of that night were just beginning to unfold.
I meditated for another five minutes and offered to roll back to the old system. Reluctantly, my colleagues agreed, dejectedly thinking that they would report to their superiors in the morning. We switched the replication direction and gave the command to import the disk group on the old server. I had already calmed down and dreamed of getting to the hotel bed with a sense of accomplishment. And suddenly I heard an exclamation: "And here it also does not start." And inside, everything broke off ...
Part 11. Akella missed
The fog completely vanished in my head. They say that in an extreme situation, the human body is mobilized. I furiously figured out what could be done. On the storage side, everything was in order. Having convinced myself of this, my colleague Alexei waved his hand, signed his impotence, and went to watch the movie on his laptop. I already spoke about his irrational actions. But formally, there was nothing to complain about.
The cause of the problems loomed quite clearly. The customer previously added additional volumes to the disk group, but they forgot to add them to the list of disks that are replicated. When the cluster started, the disk group was imported, but metadata was updated only on part of the disks, and the new ones remained in read-only mode. Now the configuration of the disk group was completely inconsistent, and the data is not available from either the new storage system or the old one.
When it comes to financial data of a huge organization, their loss is tantamount to disaster. I immediately remembered a similar story when at the dawn of my engineering career I was renovated by one customer - a serious structure that was legalized in the 90s. They discussed that their admin inadvertently deleted something. “And what happened to him?” - by the way, I asked then. “Well, while he is sitting here in the basement, the third day already. We are waiting for the general, he will decide what to do with him, ”the shaven-headed employee with a heavy chain on a bull’s neck answered me in all seriousness.
Part 12. Hindi
Have you ever felt hair moving on your head, about to turn gray? Pretending that we had lost the database, I didn’t even want to think what would happen in the morning.
In situations such as ours, you should immediately call the tech support of the manufacturer. She, at such an hour, is understandably English-speaking. I scribbled some keywords for myself and dialed a number. Two beeps and the English baritone answered me. I immediately told who we are and what happened to us. Baritone patiently listened to my monologue and asked: "What is first name?" I was dumbfounded, but, having gathered, with thoughts, I answered. Then questions followed about last name, customer name, city, location, serial number and so on ad infinitum. Time was running inexorably. After completing the survey, the baritone asked me to write down the reference number and told me to wait for a specialist call. The specialist did not call me, after 5 minutes he wrote me an e-mail. He wrote that he accepted my application for work, but his shift was over, so I need to call the dispatcher again and ask to assign an application to another specialist. It is not clear why some people think that they do not like to work in Russia. This seems to be a fallacy.
And suddenly a miracle happened. Another specialist called me back, but he was from India. The second global technical support center was there and they themselves accepted my application. I was glad of him as a father, but my joy did not last long: we did not understand each other. It was clear that he seemed to speak English, but it wasn’t easier. Sometimes some of the sounds of his speech led me to a distant thought of what he meant. But in the end, our communication came down to correspondence. I strained my memory, recalling all the best moments from Indian cinema and trying to let him feel the depth of our grief. As time went. India, as expected, sang and cried, but could not help us either.
Part 13. Execution at Dawn
At 4 o’clock in the morning, we realized that there were no other options but to recover from the backup. All eyes were on the tape library. Nikolai even crawled in front of her, wiping the floor with a tie. “How much will the base from the ribbon unwind?” I asked the local baker. “About three hours, probably,” he answered uncertainly, “but we never checked.” We urgently needed to replace the robot in the library, because for three hours we had a big stretch. Never before have I opened the packaging so fast and screwed on screws. The robot, covered in blood, was over, he fell into place. I poured blood over him, injuring my finger because of the rush, but there was no time for bandaging. The toggle switch flashed and the library was illuminated from within with light, the robot began to move and began calibration. We crowded around the library and looked through the window, mentally helping him move and read barcodes from cassettes. He did it slowly, with pleasure, and we, clinging to the glass, gritted our teeth. “Initialized!” Said the bootman, referring to the library. And he added: “I am starting recovery.” The robot thought for a second, then briskly darted to the desired row of cartridges, confidently grabbed the desired one, turned around the steering axle and drove towards the drive.
At this time, Alex in the corner behind the boxes admired the film and another bun. The film went without sound, but the excitement on the face of my unlucky colleague was genuine. Engineers often carry with them on long business trips a selection of favorite films that are good even without sound. For real art, such as films with Charlie Chaplin, can be watched this way.
Suddenly, a dull thump was heard from the library. The robot brought the cartridge to the drive, but could not determine where the hole is in the drive, and began to beat against its wall, trying to insert it somewhere. It was a complete fiasco, and despair gripped you. Unfortunately, mechanical spare parts are extremely sensitive to transportation, so there are times when a new spare part is immediately classified as DOA (dead on arrival).
Part 14. Hope
Cold sweat rolled hail down my forehead and clouded my eyes. I slowly pressed the keys, rechecking each one. We had less than an hour before the start of the working day, and all our attempts already seemed in vain, when suddenly a thought dawned on me. “Have you collected diagnostics from the old server for the last time?” I asked my colleagues. “Last night, what?” Answered Semyon, who was hungry and hungry. “There may be a conclusion from the team that reflects the configuration of the disk group before the changes, and we can try to recreate it by hand using this conclusion.”
And now I was engaged in the fact that with my hands I ran commands that re-partitioned the disks into the service area and data area. " “Have you ever done this before?” Asked me, creeping out from somewhere Nicholas. “No, I only read the article, this is shamanism, but we must try it,” I replied. Another time, my hands would be cut off right away if they knew that I had just slammed the entire disk group. But now it was our only chance, so there were no objections. All of us in this room were both sentenced and conspirators. Hope in man never dies.
I finished with the markup and typed a command that, according to the documentation, was supposed to re-create the disk group, but not to initialize the space, i.e. Do not overwrite existing data. “Doing it - do not be afraid, afraid - do not do it,” I recalled Confucius and launched the team. She, to my surprise and joy, was fulfilled and did not give a single error. I deduced the configuration, all logical volumes were in their places, with normal status. It remains to check the availability of data. In our stock was another 15 minutes. A telephone rang in the room for about 40 minutes, but no one paid attention to it. Whoever it is, they have so far had nothing to answer. “Let's start the cluster,” I suggested, and I was surprised at my voice.
In the library, the robot raged wildly about the drive, apparently hoping to starve it sooner or later. Alex stroked his laptop, impressed by the ending of the film. Semyon bored patiently, thinking about whether he had chosen the right profession, and in the interruption dreamed of a chocolate bar. Someone else was snoring behind the bars, jerking every now and then in a dream. The remaining characters focused on looking at the monitor, on which, without embellishment, our whole life was reflected. “It started!” Gasped Nicholas and grabbed the ringing pipe. These were the "applicants", they already washed the ropes and called to say goodbye. “Check the butt!” Nikolai shouted and brushed off his tie.
Part 15. Box
I sat on the box and smiled happily, thinking how now people from all apartment buildings of this beautiful city are streaming in pea-jackets, earflaps and felt boots. They gather in line to ATMs, and they also joyfully and reliably serve them. And in this whole picture there was some kind of right human happiness.
Everything worked for us, and 2 minutes before the start of the working day, the system was started. My colleagues silently thanked me, shaking my hand and shaking my head. Everything that happened that night, we were going to carry away with us, not giving out to anyone.
The Indian continued to write to me, still offering some crazy ideas. At this time, a door opened and a sleepy physiognomy popped into the room. It was a security guard with a checkpoint. He said: “There the courier arrived. He says he brought some robot. It’s like a librarian robot ... ”
A great new day began.