Incredibly truthful tech support stories
Today we continue the series of posts about the work of the technical support team, which started at the time with Loxmatiymamont in the article on Veeam support .
What would seem to be Technical Support? Sit yourself, solve technical problems, you are the smartest, the most knowledgeable, you are the very Engineer to whom the Frightened Users come. They bring their fears, pains, malfunctions, and you decide, help, advise, and, ultimately, the User leaves you not Frightened, but Inspired.
Have you already felt the significance of this work, the deep philosophical and pedagogical principles contained in it?
So, everything is a little different. Technical support is, first of all, Support, and then Technical, and therefore all work is 99% percent about people and communication with them, so of course we have something to do with textured misanthropes and pathological introverts, but it will be difficult - this time, and two - people are not always predictable, and therefore, working in Technical Support, you can learn, see and hear a lot of interesting and unusual things. Under the cut, I will share with readers a couple of such stories.
The background is as follows: the NTP server (the one that is responsible for synchronizing time on the network) got stuck with the client, and time jumped on unpredictably different hosts. Of course, this does not apply to Veeam, and the client, as a technically competent specialist, decided everything himself, but: the client using Veeam backed up its MariaDB servers, and with a separate script did both a database dump and backup of binary logs. Everyday.
Having dealt with NTP, the client checked the backups made by our software and saw a terrible thing: 12 days of backup of binary logs disappeared somewhere, and the task report showed success. Everyday.
It was to solve this mystery that they called us.
A quick investigation led us to the main culprit, which turned out to be the same NTP server. How?
And here's how: during time jumps, the NTP server generously put on the server the first days of September of the peasant uprising in Normandy, the subordination of Kashmir by the Afghans and the beginning of the construction of the Tobolo-Ishim fortified line in the Russian Empire - that is, the year 1752 from the birth of Christ. By a strange whim of history, it was in September of this year that Great Britain and its North American colonies decided to switch to the Gregorian calendar, and therefore the month looks something like this:
So, both base dumps and backup of binary logs were really done every day, only these days in 1752 turned out to be exactly 12 less than always, and the logs for September 3 were dated to the fourteenth, the fourth to the fifteenth, and so on.
The same problem also manifested itself when users use the Hijra calendar - in this case, backups went without a hitch, but when restoring data, the service stubbornly went crazy trying to work with a database created in about 1400.
But not always problems over time - these are software problems: once our engineer had to prove to the client two weeks that the second day of the week was Tuesday, but not Thursday, which the client, a completely respectable European bourgeois, refused to believe at all.
A story that itself provokes if not a smile, then a perplexed twitch with one eyebrow for sure.
The client for a long time suffered from problems in all his backup tasks: they died over and over again, as did the main service that monitors their work. The engineer, who dealt with the problem, quickly found out: before each episode, the time in our logs jumped for one hour, now into the past, then into the future. The reason was found out simply by asking the client.
Believe in the explanation and suppress the question "why?" it turned out to be much more complicated.
Imagine: a ship is going somewhere in the Atlantic Ocean. Our software is installed on it, and a small script is spinning next, tracking the GPS coordinates of the ship. Each time the ship crosses the next time zone, the script changes the time on all servers and services, including ours, and moreover, to be sure that everyone works with the correct time, it also reloads some of the services. A good example of a creative solution in the style of “we solved one problem at the cost of two others”).
PS In the process of working on this article, I was reminded of the Dovecot mail server and its method of handling similar situations:
Fatal: Time just moved backwards by 7 seconds. This might cause a lot of problems, so I'll just kill myself now. (A catastrophic error: a time transfer occurred 7 seconds ago. This can lead to a lot of problems, so I’m just clearing myself.)
A friend from the USA who called us suffered for a long time from the hangs of his Veeam service and also did not want to try the reboot offered for mystical reasons (machine uptime by that time was estimated for years), until he finally gave up and explained the reason for his resistance:
“You see, on this not only Veeam, but also the controller of the Smart House is spinning with a Windows 7 machine: all cameras, sensors, lighting, alarm and in general everything. The last time we rebooted it, a boiler exploded. ”
We have Surebackup technology, which allows you to run backups in an isolated environment and check how successful they are, not only in the report, but also in reality (and if they turned into the pumpkin with the last percent of the job completed). Good technology, quite reliable, built on the use of several test scenarios, including checking the availability over the network (for example, checking the port of an application).
In order to work with an isolated network, we run a separate application (a small virtual machine), which lives as a gateway and as the main working tool of all tests - and here something went wrong.
Appliance was launched successfully, but we couldn’t connect to it ourselves - after a while the network port turned out to be unavailable. A cyclic scan of the IP address ports quickly showed that sometimes our port exists, and sometimes it does not, but TCP 544 port appears from somewhere, which should not even exist in theory. We try other addresses - the script repeats, we check arp - the mac addresses are different.
In complete confusion, open the address with a web browser and in surprise see the login of some Chinese web camera. Once again, we change the IP address for the application, and we get exactly the same picture - for some reason, all the IP addresses on the network will redirect to this camera, which neither we nor the client understood.
The riddle remained unresolved.
Do you think these are isolated examples? Any engineer accidentally taken can tell you about a dozen of such stories, without particularly straining and never repeating themselves.
This is exactly what gives our work a taste and drive. Such stories help to remember that not all of our work comes down to a dry solution to technical problems, it has comedy and drama, a riddle and a challenge. This is our resource. This is what helps us to meet every call not with the tired “well, what do you want again?”, But with joyful anticipation ”and with what can you please me today?”
I brag about the words of one of the vice-presidents of our company, said about support:
But this is a completely different story.
What about you, dear readers? Have something to share in the comments?
Yes, if you recognized yourself in the previous paragraph, look at careers.veeam.ru/departments/support , maybe there is a vacancy for you.
What would seem to be Technical Support? Sit yourself, solve technical problems, you are the smartest, the most knowledgeable, you are the very Engineer to whom the Frightened Users come. They bring their fears, pains, malfunctions, and you decide, help, advise, and, ultimately, the User leaves you not Frightened, but Inspired.
Have you already felt the significance of this work, the deep philosophical and pedagogical principles contained in it?
So, everything is a little different. Technical support is, first of all, Support, and then Technical, and therefore all work is 99% percent about people and communication with them, so of course we have something to do with textured misanthropes and pathological introverts, but it will be difficult - this time, and two - people are not always predictable, and therefore, working in Technical Support, you can learn, see and hear a lot of interesting and unusual things. Under the cut, I will share with readers a couple of such stories.
The first, detective story: Veeam Support and a dozen lost days
The background is as follows: the NTP server (the one that is responsible for synchronizing time on the network) got stuck with the client, and time jumped on unpredictably different hosts. Of course, this does not apply to Veeam, and the client, as a technically competent specialist, decided everything himself, but: the client using Veeam backed up its MariaDB servers, and with a separate script did both a database dump and backup of binary logs. Everyday.
Having dealt with NTP, the client checked the backups made by our software and saw a terrible thing: 12 days of backup of binary logs disappeared somewhere, and the task report showed success. Everyday.
It was to solve this mystery that they called us.
A quick investigation led us to the main culprit, which turned out to be the same NTP server. How?
And here's how: during time jumps, the NTP server generously put on the server the first days of September of the peasant uprising in Normandy, the subordination of Kashmir by the Afghans and the beginning of the construction of the Tobolo-Ishim fortified line in the Russian Empire - that is, the year 1752 from the birth of Christ. By a strange whim of history, it was in September of this year that Great Britain and its North American colonies decided to switch to the Gregorian calendar, and therefore the month looks something like this:
So, both base dumps and backup of binary logs were really done every day, only these days in 1752 turned out to be exactly 12 less than always, and the logs for September 3 were dated to the fourteenth, the fourth to the fifteenth, and so on.
The same problem also manifested itself when users use the Hijra calendar - in this case, backups went without a hitch, but when restoring data, the service stubbornly went crazy trying to work with a database created in about 1400.
But not always problems over time - these are software problems: once our engineer had to prove to the client two weeks that the second day of the week was Tuesday, but not Thursday, which the client, a completely respectable European bourgeois, refused to believe at all.
The second story - a comedy of a geographical location: Veeam Support and 20 thousand leagues above the water
A story that itself provokes if not a smile, then a perplexed twitch with one eyebrow for sure.
The client for a long time suffered from problems in all his backup tasks: they died over and over again, as did the main service that monitors their work. The engineer, who dealt with the problem, quickly found out: before each episode, the time in our logs jumped for one hour, now into the past, then into the future. The reason was found out simply by asking the client.
Believe in the explanation and suppress the question "why?" it turned out to be much more complicated.
Imagine: a ship is going somewhere in the Atlantic Ocean. Our software is installed on it, and a small script is spinning next, tracking the GPS coordinates of the ship. Each time the ship crosses the next time zone, the script changes the time on all servers and services, including ours, and moreover, to be sure that everyone works with the correct time, it also reloads some of the services. A good example of a creative solution in the style of “we solved one problem at the cost of two others”).
PS In the process of working on this article, I was reminded of the Dovecot mail server and its method of handling similar situations:
Fatal: Time just moved backwards by 7 seconds. This might cause a lot of problems, so I'll just kill myself now. (A catastrophic error: a time transfer occurred 7 seconds ago. This can lead to a lot of problems, so I’m just clearing myself.)
Story Three - “Horror”: Veeam Support and Exploding Boiler
A friend from the USA who called us suffered for a long time from the hangs of his Veeam service and also did not want to try the reboot offered for mystical reasons (machine uptime by that time was estimated for years), until he finally gave up and explained the reason for his resistance:
“You see, on this not only Veeam, but also the controller of the Smart House is spinning with a Windows 7 machine: all cameras, sensors, lighting, alarm and in general everything. The last time we rebooted it, a boiler exploded. ”
The Fourth Story, Mystical: Veeam Support and the Eye of Heaven
We have Surebackup technology, which allows you to run backups in an isolated environment and check how successful they are, not only in the report, but also in reality (and if they turned into the pumpkin with the last percent of the job completed). Good technology, quite reliable, built on the use of several test scenarios, including checking the availability over the network (for example, checking the port of an application).
In order to work with an isolated network, we run a separate application (a small virtual machine), which lives as a gateway and as the main working tool of all tests - and here something went wrong.
Appliance was launched successfully, but we couldn’t connect to it ourselves - after a while the network port turned out to be unavailable. A cyclic scan of the IP address ports quickly showed that sometimes our port exists, and sometimes it does not, but TCP 544 port appears from somewhere, which should not even exist in theory. We try other addresses - the script repeats, we check arp - the mac addresses are different.
In complete confusion, open the address with a web browser and in surprise see the login of some Chinese web camera. Once again, we change the IP address for the application, and we get exactly the same picture - for some reason, all the IP addresses on the network will redirect to this camera, which neither we nor the client understood.
The riddle remained unresolved.
***
Do you think these are isolated examples? Any engineer accidentally taken can tell you about a dozen of such stories, without particularly straining and never repeating themselves.
This is exactly what gives our work a taste and drive. Such stories help to remember that not all of our work comes down to a dry solution to technical problems, it has comedy and drama, a riddle and a challenge. This is our resource. This is what helps us to meet every call not with the tired “well, what do you want again?”, But with joyful anticipation ”and with what can you please me today?”
I brag about the words of one of the vice-presidents of our company, said about support:
“ Technical Support - they are monster! They know not only how to solve technical issues, but also how to talk to customers. ” (Technical support is some unreal guys! They not only know how to solve technical problems, but also know how to properly conduct a dialogue with a client. ”)And we really can, moreover, we teach this: a person with a good language (especially second or third, in addition to English), who knows how to communicate with customers and understands why this is necessary, we introduce IT in general and our product in particular (would you know how many talented graduates of language universities work for us - but they started with almost zero knowledge!). And we can also tighten up a good technical specialist and help soft-skills develop.
But this is a completely different story.
***
What about you, dear readers? Have something to share in the comments?
Yes, if you recognized yourself in the previous paragraph, look at careers.veeam.ru/departments/support , maybe there is a vacancy for you.