A day in the life of technical support

Is it good to work in technical support? Well it depends on what needs to be supported! Today we will talk about what tasks the support team at Virtuozzo has to solve, and they will share their secrets - why they came to work in these positions.


So, technical support is a specific work related to constant communication with customers. No matter how hard we try to make the program code better, there are always enough problems. They can be associated with the next release, and with an atypical scenario for using software, and with unforeseen situations in the personal server client. In our case, all these troubles have to be solved by a team of support workers, which employs 10 people - and their number is planned to be increased.



Leading Virtuozzo Technical Support Specialists Maria Antonova and Anna Vinokurova

- I have been working in technical support for a long time, and I really like this work in terms of language self-improvement. English is the language of international communication, for many of our customers it is not native, and this cannot but leave its mark on the manner of their speech and perception of what you say in response. For those who love communication and linguistics in general, but see themselves in a technical profile, it seems to me that working in such an environment is very useful for pumping out language skills and career growth, says Anna Vinokurova.

The practice of Virtuozzo technical support service has established for itself several rules that must be followed when communicating with customers from different regions. In this case, it turns out to be easier to work, and the results are achieved faster. Here is some of them:

• Clients from the Asian region should not write too long and detailed letters; from the entire letter, most likely only the first couple of paragraphs will be read.
• When verbally communicating with American clients, you need to give confirmation several times that you are understood correctly.
• Germans and Austrians can be given quite complex instructions, consisting of many points, and they will apply them exactly as it was written

In general, the specifics of the work of the supporters is a very high degree of spontaneity. You will never guess at what moment something will go wrong and the engineers should be, as they say, “always ready”. Clients may need urgent help for various reasons, but basically any urgency is associated with obligations to end users - after all, most of Virtuozzo's clients are hosters, and idle virtual machines or containers are someone’s inaccessible sites or databases.

However, there are predictable load waves; they are usually associated with the release of new products, the release of major updates, or the discovery of another Linux vulnerability.

“When the next security advisory is published on the Internet, excited customers start coming to support who want to know which versions are affected by the vulnerability, and how soon we will release an update,” says Anna Vinokurova.

- I really like to work in technical support, because this work combines the tasks of an expert in the field of software, programmer and detective. Sometimes a problem is so disguised that its solution is completely unobvious. And every time you manage to solve the riddle “why nothing works”, a feeling of deep satisfaction and even triumph comes! - says Maria Antonova.

What does Virtuozzo support do?

For those who are considering whether to go to work in the support team, we will tell you a few stories from the real practice of our technical support.

1. No memory is added - CPU error

One night, a client reports in support of an error on an attempt to increase RAM for one of the virtual machines. However, the error somehow "swears" on the CPU, not RAM.

At first, such a situation surprised everyone very much, but after a while we figured out that the error returned was not directly related to the action being performed - it only exposed the failure, which was much deeper inside the system.

The reason turned out to be very serious: the CPU socket was “knocked out” on the physical server, and it ceased to be detected, and with it all the virtual kernels within the socket went offline. Naturally, when changing the memory limits for the virtual machine, the validation of the future configuration did not pass simply as a whole, because the number of cores began to exceed the number of available cores on the server node.

The fact is that during the validation of the configuration file, NUMA arrays are processed, and the CPUs that belong to them. On a physically “healthy” server, not a single NUMA should be empty, and the situation arose because no one had previously assumed that a server that was damaged in this way would be able to work for a long time.

Decision

On the client server, a correction was made to the python component of the product, which allowed it to continue to work correctly, despite the existence of NUMA nodes consisting entirely of offline kernels. After that, we were able to set the number of CPUs within the real “surviving” cores for the virtual machine, which means that further configuration changes became possible.

However, more global improvements followed: in the product code, the determination of CPU statuses was constantly improved: an empty array with virtual CPUs belonging to the same NUMA group no longer causes errors in the code and is considered a legal situation.

“In fact, the client was very lucky that his server remained available for quite some time, despite such a serious hardware failure. This allowed virtual machines to migrate to other hosts while they were still available, ”said Maria Antonova.

2. Loss of network connection in virtual machines

The client noticed a periodic loss of network access in virtual machines. However, if such a virtual machine is migrated to another physical server or restarted on the current one, the network works again.

To investigate the problem, it was necessary to get a virtual machine at the time the problem was reproduced, but the services inside the virtual machines should have worked without interruption, and our online research would have caused problems for a large number of end users. Therefore, the client provided us with a copy of the virtual machine on which we were able to reproduce the problem and examine it in our environment without inconvenience for users. The bug was found in the memory allocation mechanism for the socket buffer of the virtio-net virtual device - part of the kernel module.

Decision

Alas, there was simply no quick solution to the problem without changing the kernel modules; it turned out that the client simply did not upgrade to the latest version of Virtuozzo Linux. At the time of the appeal, an update had already been released, moreover in ReadyKernel format. This allowed us to apply the update and solve the problem in a few minutes without even restarting the services.

3. The user deleted all containers

One client decided to directly change the path to the private container area, but due to an error in the shell script prepared by him, the path for all containers began to point to the same place - the common storage of all containers. Further, as expected, one of the end users decided to delete his own container. Removal led to the disappearance of all containers in the repository - after all, the updated configuration pointed to this path.

Solution

In order to prevent further data loss, the client control panel was temporarily disabled for end users so that they could not stop the containers, as this would lead to further data loss. There was nothing to be done with completely deleted containers, so the task was to restore metadata for those containers that remained running.

Since there were about 400 pieces of damaged containers, the task was quite creative - you had to write a script that would do similar, but not the same type of manipulations for all the victims. As a deeper inspection showed, for 200 containers, the data was completely lost, since the containers were in a stopped state at that moment. But for running containers, the deletion did not complete completely, since their virtual disks were locked. In such containers, all data was lost, except for the virtual disks themselves - metadata, configuration files, and other service files.

Physically, the virtual disk files were moved to a temporary directory to delete the files, and stopping such a container would lead to the final deletion of data.

Since metadata varies from container to container, they must be created individually from scratch. Configuration files were restored on the basis of billing information and end-user tariff plans - this was done by the client himself, since our support service does not have access to the billing information of the client. But we, in turn, provided something like a "skeleton" of the configuration in which you had to add IP, host name, resource limits. To restore disk metadata, Virtuozzo supporters created a script that generated the correct metadata based on information about mounted containers from the kernel, and also calculated virtual disk parameters, such as the number of cylinders, heads and sectors.

By the way, after this call, a check was included in the container removal mechanism to check if the directory to be deleted is a container - this will avoid similar problems in the future, even if the user runs the script with an error.

And a detective, a linguist, and a programmer.

Thus, we can confidently say that our support team is a very interesting team, ready to solve multifaceted problems and even engage in product development. And if you want to try yourself in such work, you can work at night and are not afraid of Asian English, we may just have new vacancies!