Cyber ​​quest from the Veeam technical support team

    This winter, or rather, one of the days between Catholic Christmas and New Year, Veeam technical support engineers were busy with unusual tasks: they hunted for a group of hackers called Veeamonymous.

    About how the guys themselves came up with and carried out a real quest at work, in reality, with tasks “close to fighting,” said Kirill Stetsko , Escalation Engineer .

    “Why are you doing this?”

    - In much the same way that people came up with Linux - just for fun, for their own pleasure.

    We wanted to move, and at the same time we wanted to do something useful, something interesting. Plus it was necessary to give some emotional relief to the engineers from their everyday work.

    “Who suggested that?” Whose idea was it?

    - The idea was our manager Katya Egorova, and then the concept and all further ideas were born together. Originally thought to make a hackathon. But during the development of the concept, the idea turned into a quest, after all, a technical support engineer is a different kind of activity than programming.

    So, we called friends, acquaintances, acquaintances, different people helped us with the concept - one person with T2 (second support line - ed. ), One person with T3, a couple of people from the SWAT team (quick response team for especially urgent cases - Ed. ). They all came together, sat down and tried to come up with tasks for our quest.

    - It was very unexpected to find out everything about it, because, as far as I know, usually the script mechanics work out the quest mechanics, that is, you not only did such a difficult thing, but also in relation to your work, to your professional field of activity.

    - Yes, we wanted to do not just entertainment, but to “pump over” the technical skills of engineers. One of the tasks in our department is the exchange of knowledge and training, but such a quest is a great opportunity to let people “touch” some new live techniques.

    - How did you come up with tasks?

    - Have a brainstorming session. We had an understanding that we should do some technical tests, such that they were interesting and at the same time carry new knowledge.
    For example, we thought that people should be given the opportunity to try to sniff traffic, use hex editors, do something for Linux, and some slightly deeper things related to our products (Veeam Backup & Replication and others).

    Also an important part was the concept. We decided to build on the topic of hackers, anonymous access and an atmosphere of secrecy. Guy Fawkes mask was made a symbol, and the name came by itself - Veeamonymous.

    “In the beginning was the word”

    To stir up interest, we decided before the start of the event to arrange a PR-company in the theme of the quest: they hung posters with an announcement at our office. And a few days later, secretly from everyone, they painted them with spray cans and launched a “duck”, they say, some attackers ruined the posters, even attached a photo with a proof ...

    - So you did it yourself, that is, the team of organizers ?!

    - Yes, on Friday, at 9 o’clock, when everyone had already left, we went and drew a green “V” letter from the balloons.) Many participants in the quest didn’t guess who did it - people came up to us and asked who ruined the posters ? Someone took this issue very seriously and launched an entire investigation on this subject.

    For the quest, we wrote and audio files, “tearing” sounds: for example, when an engineer logs into our [production CRM] system, then there is a robot answering machine that says all kinds of phrases, numbers ... Here we are from those words that he has recorded, composed more or less meaningful phrases, well, maybe a little curves - for example, we got “No friends to help you” in an audio file.

    For example, we represented the IP address in binary code, all, again, with the help of these numbers [pronounced by the robot], all sorts of frightening sounds were added. They shot the video themselves: in the video we have a man sitting in a black hood and wearing a Guy Fawkes mask, but in fact there are not one person, but three, because two are standing behind him and holding the “background” from the blanket :).

    - Well, you got confused, to be honest.

    - Yes, we caught fire. In general, we first came up with our technical specifications, and then composed a literary and game canvas on the topic of what allegedly happened. According to the script, the participants hunted for a group of hackers called “Veeamonymous”. The idea was also that we were “breaking the 4th wall”, that is, we are transferring events to reality - for example, we painted from a spray can.

    With literary text processing, we were helped by one of the native speakers of English from our department.

    “Wait, why the native speaker?” You also did everything in English ?!

    - Yes, we did it for the St. Petersburg and Bucharest offices, so everything was in English.

    For the first experiment, we tried to make everything just work, so the script was linear and fairly simple. Added more surroundings: secret texts, ciphers, pictures.

    We also used memes: there were a lot of pictures on the topics of investigations, UFOs, some popular horror stories - some teams were distracted by it, tried to find some hidden messages there, apply their knowledge of steganography and other things ... but, of course, there’s nothing like that It was.

    About thorns

    However, in the process of preparation, we were faced with unexpected tasks for ourselves.

    They fought a lot over them and solved all sorts of suddenly arising questions, but somewhere around a week before the quest they generally thought that everything was gone.

    Probably worth a little talk about the technical basis of the quest.

    Everything was done on our internal ESXi lab. We had 6 teams, which means we had to allocate 6 resource pools. So, for each team we deployed a separate pool with the necessary virtual machines (the same IP). But since all this was on servers that are on the same network, the current configuration of our VLANs did not allow isolating machines in different pools. And, for example, during a test run, we received situations when a machine from one pool connected to a machine from another.

    - How could you fix the situation?

    - At first we thought for a long time, tested all sorts of options with permissions, separate vLANs for machines. As a result, we did this - each team only sees the Veeam Backup server through which all further work takes place, but does not see the hidden sub-pool, in which there are:

    • several windows machines
    • Windows core server
    • Linux machine
    • VTL pair (Virtual Tape Library)

    All pools are assigned a separate group of ports on the vDS switch and their Private VLAN. Such double isolation is just needed to completely eliminate the possibility of network interaction.

    About the brave

    - Anyone could take part in the quest? How were teams formed?

    - This was our first experience in holding such an event, and the capabilities of our laboratory were limited to 6 teams.

    First, we, as I said, held a PR company: using posters and newsletters, we informed that the quest would be carried out. We even had some hints - phrases in binary code were encrypted on the posters themselves. Thus, we interested people, and people already agreed among themselves with friends, with friends, and cooperated. As a result, more volunteers responded than we had pools, so we had to conduct a selection: we came up with a simple test task and sent it to everyone who responded. It was a logical task, it had to be solved for speed.

    The team allowed up to 5 people. The captain was not needed there, the idea was in cooperation, in communication with each other. Someone is strong, for example, in Linux, someone is strong in teips (backups on tapes), and everyone, seeing the task, could put his efforts into the general solution. Everyone talked to each other, found a solution.

    - And at what point did this event start? Did you have some kind of "hour X"?

    - Yes, we had a strictly scheduled day, we chose it so that there was less load in the department. Naturally, the team leaders were notified in advance that such and such teams were invited to participate in the quest, and they needed to be given some relief [regarding the load] that day. It appeared that it should be the end of the year, December 28, Friday. Expected to take about 5 hours, but all the teams coped faster.

    - Everyone was on an equal footing, did everyone have the same tasks based on real cases?

    - Well, yes, each of the compilers took some personal stories from personal experience. We knew about something that this could be in reality, and it would be interesting for a person to “feel” it, look, figure it out. They took some more specific things, for example, data recovery from damaged tapes. Someone with tips, but most of the teams coped on their own.

    Or it was necessary to apply the magic of fast scripts - for example, we had a story that a certain “logical bomb” “torn” a multi-volume archive into random folders on a tree, and it was necessary to collect data. You can do this manually - find and copy [files] one by one, or you can write a script by mask.

    In general, we tried to adhere to the point of view that one problem can be solved in different ways. For example, if you are a little more experienced or want to “get confused,” then you can solve it faster, but there is a direct way to solve it “forehead” - but at the same time you will spend more time on the task. That is, almost every task had several solutions, and it was interesting which paths the teams would choose. So the nonlinearity was precisely in the choice of the solution option.

    By the way, the Linux problem turned out to be the most difficult - only one team solved it independently, without prompts.

    “Could you take the hints?” How is this quest ??

    - Yes, it was possible to take, because we understood that people are different, and those who lacked some knowledge could fall into the same team, so that we won’t lose the passage and competitive interest, we decided that we would hints. For this, each team was watched by a person from the organizers. Well, we made sure that no one cheated.

    About the stars

    - And there were prizes for the winners?

    - Yes, we tried to make the most pleasant prizes both for all participants and for the winners: the winners received designer sweatshirts with the Veeam logo and a phrase encrypted in a hexadecimal code, black). All participants received a Guy Fawkes mask and a company bag with a logo and the same code.

    - That is, you had everything as in a real quest!

    - Well, we wanted to do a cool, adult thing, and it seems to me that we did it.

    - And there is! And what was the reaction of those who participated in this quest? Did you achieve the goals?

    - Yes, many later came up, said that they clearly saw their weaknesses and wanted to pull them up. Someone has ceased to be afraid of certain technologies - for example, dumping blocks from teips and trying to get something out there ... Someone realized that he needed to tighten Linux, and so on. We tried to give a fairly wide range of tasks, but not entirely trivial.

    Winners team

    “He who wants will succeed!”

    - Did those who prepared the quest take a lot of effort?

    - In fact yes. But this was most likely due to the fact that we had no experience in preparing such quests, such infrastructures. (We will make a reservation that this is not our real infrastructure - it just had to perform some kind of game functions.)

    For us it was a very interesting experience. At first I was skeptical, because the idea seemed even too cool to me, I thought it was very difficult to implement. But they began to do, they began to plow, they all started to catch fire, and in the end we did it. And there were even almost no overlays.

    In general, we spent 3 months. For the most part, we came up with a concept, discussed what we can implement. In the process, of course, something changed, because we understood that for some reason we did not have the technical ability to do this. On the go, I had to redo something, but so that the whole canvas, history and logic would not break. We tried not just to give a list of technical tasks, but to make it fit into history, to be coherent and logical. The main work went on during the last month, that is, 3-4 weeks before X.

    - So, in addition to your main activity, did you take the time to prepare?

    - This we did in parallel with the main work, yes.

    “Are you asked to do this again?”

    - Yes, we have many requests to repeat.

    - And you?

    - We have new ideas, new concepts, we want to attract more people and stretch it in time - both the selection process and the game process itself. In general, we are inspired by the Cicada project, it can be google - this is a very cool IT topic, there people from all over the world come together, start branches on reddit forums, they use cipher translations and solve riddles, and all that.

    - The idea was great, just respect for the idea and implementation, because it really is worth a lot. I sincerely wish you not to lose this inspiration, so that all your new projects will also be successful. Thanks!

    - Yes, but it will be possible to look at an example of a task that you definitely won’t reuse?

    - I suspect that we will not reuse one. Therefore, I can tell you about the course of the whole quest.

    Bonus track
    At the very beginning, the players have the name of the virtual machine and credentials from vCenter. Logged into it, they see this car, but it does not start. Here you have to guess that something is wrong with the .vmx file. After downloading it, they see the prompt necessary for the second step. In fact, it says that the database used by Veeam Backup & Replication is encrypted.
    Having removed the prompt, having downloaded the .vmx file back and having successfully turned on the machine, they see that on one of the disks there really lies a base64 encrypted base64. Accordingly, the task is to decrypt it and get a fully functional Veeam server.

    A little bit about the virtual machine on which this all happens. As we remember, in the story the main character of the quest is a rather dark person and is engaged in something that is clearly not too legal. Therefore, his working computer should have quite a hacker appearance, which we had to create, despite the fact that it is Windows. The first thing was added a lot of props like information on major hacks, DDoS attacks and the like. Then they installed all the typical software and laid out everywhere different dumps, files with hashes, etc. Everything is like in a movie. Among other things, there were folders named on the principle of closed-case *** and open-case ***.
    To go further, players need to restore prompts from files in backups.

    Here it must be said that in the beginning the players were given quite a bit of information, and they receive most of the data (such as IP, logins and passwords) during the quest, finding hints in backups or files scattered on the machines. Initially, backup files are located on the Linux repository, but the folder on the server itself is mounted with the noexec flag , so the agent responsible for restoring files cannot start.

    After repairing the repository, participants gain access to all content and finally can restore any information. It remains to understand which one. And for this, they just need to study the files stored on this machine, determine which of them are “broken” and what exactly needs to be restored.

    At this point, the script shifts from general IT knowledge to specific Veeam features.

    In this particular example (when you know the file name but don’t know where to look for it), you need to use the search function in Enterprise Manager, and so on. As a result, after restoring the entire logical chain, the players have one more login / password and nmap output. This leads them to the Windows Core server, and by RDP (so that life does not seem to be honey).

    The main feature of this server: with the help of a simple script and several dictionaries, an absolutely meaningless structure of folders and files was formed there. And when you login, you receive a welcome message of the form "A logical bomb exploded here, so you will have to collect the tips for the next steps in pieces."

    The following clue was divided into a multi-volume archive (pieces 40-50) and randomly arranged in these folders. Our idea was that players should show their talents in writing simple PowerShell scripts in order to collect a multi-volume archive and get the required data using a well-known mask. (But it turned out as in that joke - some of the subjects were unusually physically developed.)

    The archive contained a photo of the cassette (with the inscription “Last Supper - Best Moments”), which hinted at the use of the connected tape library, where there was a cassette with a similar name. Here is only one trouble - it turned out to be inoperable so much that it was not even cataloged. Here probably the most hardcore part of the quest began. We erased the title of the cassette, so in order to recover data from it, you just need to dump the “raw” blocks and view them in the hex editor to find the markers of the beginning of the files.
    We find the marker, look at the offset, multiply the block by its size, add the offset and using the internal tool try to restore the file from a specific block. If everything is done correctly and the math came together, then the players have a .wav file in their hands.

    In it, using a voice generator, among other things, a binary code is dictated, which is revealed in another IP.

    This, it turns out, is a new Windows server, where everything hints at the need to use Wireshark, but it's not there. The main focus is that there are two systems installed on this machine - only the disk from the second is disconnected offline through the device manager, and the logical chain leads to the need for a reboot. After which it turns out that by default a completely different system should be loaded where Wireshark is installed. And all this time we were on the secondary OS.

    There is nothing special to do here, just turn on capture on a single interface. With a relatively careful consideration of the dump, a clearly left packet is sent out from the auxiliary machine at regular intervals, in which there is a link to the youtube video, where players are asked to call a specific number. The first caller listens to the congratulations on the first place, the rest - an invitation to HR (joke)).

    By the way, we have open vacancies for technical support engineers and for interns. Welcome to the team!

    Also popular now: