The Standoff: how it was

Greetings! Having seen enough interest in what is happening at The Standoff in the ranks of defenders at PHDays 9, we decided to talk about how the preparation and the “standoff” took place through the eyes of the Jet CSIRT as part of the Jet Security Team.

Guo Standoff, I created

Something like this the colleagues reported that we are once again participating in The Standoff, and we naturally agreed.

It’s worth mentioning right away that for the defenders this year the format of the competition has changed somewhat. All teams received very similar office infrastructures, and this made it possible for the organizers to enter a rating of defenders in certain parrots. And for the Jet Security Team, this was the first "Confrontation", where the office was defended, and not the industrial infrastructure.

We got access to the infrastructure for preparing for cyber battle in late April. After the audit of the infrastructure, a whole wagon of deficiencies was assembled, here are just some of them. Absolutely throughout the infrastructure there were no actual patches. The passwords of all users could be obtained through Ntds.dit in cleartext. Moreover, some users had passwords from the TOP-500 list or passwords with an easily reversible hash. System hardening was close to nothing or nothing at all. Some hosts in the DMZ had an interface in the server subnet.

Based on the results of the audit, we developed certain security measures, and in turn, the organizers, after prior approval, allowed us to apply the policies we needed and bring along any security tools and tools that can be deployed in a virtual environment. Due to the tight deadlines, some ideas on protective measures fell off at the start. The main settings and profiling of the SPI were carried out on May holidays (hello to everyone who threw pictures from the picnics, we love you too), and some of the protective equipment had to be tuned right before the start right on the site. Also, a number of services were forbidden to patch and strongly reconfigure. For example, one of these was Oracle Weblogic with CVE-2019-2725, which PoC was released in the very early days of May 2019.

Well, the list of what we brought with us:

• Firewall (provided by the organizers has been replaced);
• Antivirus solution;
• WAF;
• EDR
• A couple of deception solutions;
• A pair of vulnerability scanners;
• ELK stack for additional Netflow analysis;
• SIEM.

We should also talk about what was going to SIEM. As sources of events, we had at our disposal the Windows logs, Sysmon, Auditd logs and, as you might guess, events from the SZI itself. If there were no special problems with the first two, and we quickly agreed on changes in the Sysmon policy and configuration, then we had to wrestle with the organizers for the Auditd config.

In parallel with this, we identified the main attack vectors, and based on this we selected and adapted the relevant scenarios and correlation rules - a total of about 160 correlation rules. Plus, a set of motley dashboards was assembled for critical nodes, SZI and what required special attention in the gaming infrastructure.

The standoff

For The Standoff, we decided to adhere to the concept of separating incidents into external and internal, as there was an exact understanding that outside we would actively try to scan and operate the web. Incidents related to scanning and attempts to circumvent WAF were monitored separately, at a lower priority, this allowed us to focus on internal incidents. Dashboards for SPI were distributed between defenders in areas of responsibility, and at least 2 people were assigned to each tool - for the possibility of rotation and rest.

Everything happened, as we expected. The confrontation began at about 10 am, and as soon as the start was made, the SIEM system began to give off a bunch of incidents related to an external scan and attempts by attackers to exploit the web. In some cases, even the group did not save. Along with this, the organizers' checkers started working, checking the status of various services in the office, which forced us to re-conduct profiling to some extent to cut off false positives associated with them.

In the very first hours of the game, the Hack.ERS team managed to find the standard credentials from the administrator (admin / admin) on the CMS of one of the resources and detect a potential LFI vulnerability. These attempts did not go unnoticed, our defenders carried out an operational response, and the attackers in the end could not advance further.

Until the end of the first game day, the methods did not change, WAF still beat off all attempts to upload something interesting to the company's websites, and the same “external addresses”, without ceasing, tried to scan our resources.



In total, for the entire event, 3000 incidents related to scanning attempts were recorded, without taking into account the grouping of events in incidents.



And about 2500 incidents with attempts to circumvent WAF, also without taking into account the grouping of events in incidents.

Toward night, the intensity of all activities decreased — there were several reasons for this. Part of the defenders and attackers could not resist the soundcheck and rehearsal of the concert, which was supposed to be held the next day. Some attacking teams decided to take a break and continue the attacks towards the end of the morning in the hope that the defenders and the monitoring would have fewer monitoring resources and some fatigue.

On the morning of the second day, the attackers changed tactics. Information on a part of its employees was posted on one of the company's websites. Hackers took advantage of this information and began actively collecting user accounts through Exchange (statistics of attempts in the screenshot).



A little later, insecure attempts were made to select a password on the VPN gateway, accounts that were not in our infrastructure participated in the brute. With high probability, the attackers tried to use accounts from the already hacked infrastructure in the hope that the organizers left them the same everywhere. As a result, the whole situation with brute force led us to create a group of dashboards on trends in terms of user authentication. Plus, we tightened monitoring on incidents related to successful brute force, but, fortunately, no such cases were detected.

Approximately an hour before the end of the game, trends saw single successful attempts to authenticate several users, including those on Exchange, operational analysis showed that the sources were user machines directly, most of the events indicated that the organizers logged in from the VMware console Vcenter.

At the same time, we recorded an internal scan from a node that successfully connected via VPN. After an operational analysis of the events associated with the incident, it became clear that the attackers were able to compromise the credentials of several users, and judging by the absence of unsuccessful authentication attempts, it was very likely that the user data was “leaked”.

Information was passed on to the defenders. During the entire response time on compromised users' personal computers, the endpoint solution was put into a preventive protection mode to slow down the ability to gain a foothold in the system. Attacking sessions on the VPN gateway were forcibly dropped, and the attackers were kicked out of the protected perimeter. In compromised UZ, passwords were promptly changed.

At that very moment, the guys from the True0xA3 team took the stage and successfully used OSINT and reported the compromise of the Behealthy office, which is under the protection of another team. Attackers managed to compromise the domain admin. The organizers were notified of our incident and proofs were provided.

The last hour was especially hot due to the sudden OSINT, everyone was waiting for some more preparations from the organizers. The monitoring team, in turn, monitored all suspicious anomalies and incidents, but after an unsuccessful attempt to penetrate new hacking attempts, it did not follow. So the last minutes of playing time passed, and the successful hacking of the office protected by the Jet Security Team did not happen.

And a little final statistics for two gaming days:

• 1200 EPS on average and slightly less than 3000 EPS at the peak;
• About 7000 incidents;
• Over 120 million events.

Factors That Helped Us Win

• Competent distribution of roles. Each set up and was engaged in most of what he does in everyday projects. No one had to study materials from the category of "firewall for dummies."
• Operational profiling of correlation rules. All false positives were processed and made corrections related to the features of the gaming infrastructure.
• Prompt response. Due to the fact that several people were assigned to each type of SZI and systems, we had no problems with the fact that the responsible person had a rest or just walked away for a couple of minutes. All information from monitoring was processed as quickly as possible.
• Experience in hardening and monitoring various infrastructures.

PS Special thanks to everyone who came and asked questions, and we apologize to those who were not able to tell something in the process - the team was waiting for “mishandled Cossacks” from the attackers and could not disclose all the details.

Dmitry Lifanov, expert at the Center for Monitoring and Response to Information Security Incidents Jet CSIRT