May 31, 2017 at 15:41PHDays VII: The Chronicles of the Confrontation
This year, before the Confrontation, we once again assembled a hodgepodge team partly from Solar Security employees, partly from caring friends and SOC builders of Russia. In the article we will try to describe the whole process of participation in the “Confrontation” - what were the “Easter eggs” from the organizers, what were the attacks, how were we defending, what tools worked out, etc.
Last year, our team was engaged in monitoring Telecom (which was not very different in infrastructure from the Office, but not the point). This year, there have been fewer SOCs, Telecom’s infrastructure has lost its domain, workstations and most of its servers, and we decided to work with the Office as well.
Each participant in the confrontation had their own goals and objectives. For us, the main task was the opportunity in “combat conditions” to check the content on working with endpoint (we checked the network part last year), ideas on identifying various methods of delivering malware to users' workstations and fixing Attackers in the infrastructure. The office was ideally suited for this, especially considering the fact that the organizers promised "the enemy inside."
If you take an integrated assessment of the event, everything was quite cheerful and interesting. Both infrastructures that we defended did without successful attacks in the field of protection and at work (GSM Telecom was not in the scope of protection or monitoring). We completely protected the infrastructure from the attackers (although the Telecom team received a yellow card for disputes with the referee). Along the way, they found a couple of Easter eggs from the organizers, which apparently deprived themselves of a fascinating struggle with the internal enemy. Now, first things first.
Monitoring the availability of infrastructure and the relevance of audit settings
From the experience of last year, it was clear that the infrastructure was unstable and could change at arbitrary points in time. But only if last year it grew (we received 6 hosts on the external perimeter within 10 minutes), then this began to rapidly decline. At the time of launch, the number of servers within the network decreased to 11, and the number of workstations - to 15.
In such circumstances, it quickly became clear that, despite the configured audit, reliable information about its state was needed. Indeed, if the virtual machine with any of the hosts is rolled back, then we will be left without patches, and without monitoring.
As a result, a decent part of the time was spent quickly dealing with emerging problems. The hosts periodically hung (the entrance to the console via RDP could take half an hour), they rebooted, and all this seriously spoiled the nerves. One thing is good - accessibility monitoring worked for all 100.
The first incidents - Web.
Expectedly, it all starts with external web services. They scanned constantly, tried to throw shells and deface sites periodically.
This is how attackers sometimes reveal themselves (successful operation of SSTI in photo hosting: defcon.ru/web-security/3840 ):
As a result, under the PHDays conditions, the rule worked quite well:
Now any attempt to gain control over the host and initiate connections to its hosts was detected quite well.
At some point it became boring to look at all this and chmod –x / bin / nc was done on the server.
Result:
Apparently, the people from the side of the Attackers were somewhat offended and the following requests were sent:
General results:
And where are the attacks inside the network?
Over the entire monitoring period, there was not a single penetration of attackers outside the Office perimeter.
Perhaps this is the biggest disappointment in the past PHDays, since working with callbacks, embedded malware and insiders is one of the very serious layers of the life of current information security and SOC. Communicating with colleagues from the Defender teams, we heard from everyone that the main vector is aimed at overcoming the perimeter: they will scan the web, sometimes sites are defunct, and exploit vulnerabilities.
At the same time, we regularly conducted our checks for detects on different hosts, and all of them passed successfully. Everything that we planned to detect, everything worked out.
Unfortunately, due to the absence of attacks, it will not work in detail to show the logic of detection. Therefore, a couple of examples of how such detections look in our infrastructure:
In general, the presence of unprotected infrastructure and a bank in the monitoring mode, rather than blocking, greatly changed the focus of Attackers. Most of the time they focused on hacking and auditing the security of this particular infrastructure, and not on breaking through the defensive redoubts of the Defenders and SOCs. As a result, some of the rich toolkit of the Attackers was not applied to the defenders' infrastructures, and some of the surprises of the defenders and SOCs in detecting and counteracting were not fully tested in combat conditions.
However, do not take this as criticism towards the organizers. They did a tremendous amount of work to launch the event, infrastructures and complex integrations, and with all the difficulties encountered PHDays confirmed its status as one of the key information security conferences. But SOCs and Defenders want more fire and hardcore, and we are ready to help in its invention;)
Last year, our team was engaged in monitoring Telecom (which was not very different in infrastructure from the Office, but not the point). This year, there have been fewer SOCs, Telecom’s infrastructure has lost its domain, workstations and most of its servers, and we decided to work with the Office as well.
Each participant in the confrontation had their own goals and objectives. For us, the main task was the opportunity in “combat conditions” to check the content on working with endpoint (we checked the network part last year), ideas on identifying various methods of delivering malware to users' workstations and fixing Attackers in the infrastructure. The office was ideally suited for this, especially considering the fact that the organizers promised "the enemy inside."
If you take an integrated assessment of the event, everything was quite cheerful and interesting. Both infrastructures that we defended did without successful attacks in the field of protection and at work (GSM Telecom was not in the scope of protection or monitoring). We completely protected the infrastructure from the attackers (although the Telecom team received a yellow card for disputes with the referee). Along the way, they found a couple of Easter eggs from the organizers, which apparently deprived themselves of a fascinating struggle with the internal enemy. Now, first things first.
Stage of preparation, or how the infrastructure looks before the start of the game
- According to the organizers, "in order to maintain balance in the infrastructures of companies there is a constant, with a slight backlash, number of vulnerabilities." In practice, it looks like Windows 2008 R2 without a single update on all 50 hosts. This was the first surprise. At the same time, the possibility of updating the OS was knocked out almost with a fight, because there should be a "balance".
But you need to update in any case, because, for example, Sysmon v6 is no longer installed on Windows 2008 and Windows 7, since the OS cannot verify the digital signature. Solution .
- The second surprise was the number of Web applications in the office infrastructure. Apparently, we decided to collect all the well-known platforms with the largest number of holey software.
- What did the infrastructure look like:
- 25 servers running Windows 2008 R2. The main functionality is the mysql base for each of the external web services.
- 25 Windows 7 workstations.
- 32 external services, mainly Ubuntu with a variety of web content.
SZI, for which the Defenders had enough funds:- CheckPoint FW + IPS on the external network perimeter.
- Kaspersky AV 10 on workstations + Kaspersky for windows server 8.0 on servers.
- Kaspersky for Linux on external web servers.
- ObserveIT on all infrastructure servers.
Monitoring in SOC:- ArcSight - SZI + OS audit.
- Splunk - netflow + Unix Audit. Monitoring Web + monitoring network attacks, the availability of new hosts / services.
- In agreement with the Defenders, in addition to monitoring, we took upon ourselves the response to the Windows infrastructure. Including for the reason that the audit we need is not quite trivial to roll out and we ourselves had to identify the problem areas.
General infrastructure monitoring concept
- External Web:
- At the OS level - auth log + auditd on EXECVE + change of Web server configurations and / tmp / directories.
- Audit of running commands in bash (something like webplay.pro/linux/syslog-log-bash-history-every-user.html , but much improved).
- Hope for CheckPoint IPS.
- Kaspersky AV on Linux. Not that we strongly believed in him, but since he is ...
Tasks for monitoring (they took it to a minimum so that it would not be completely embarrassing):- External scans.
- Password guessing and server logins via SSH are not from our segments.
- Creating scripts in the OS.
- Some custom scenarios that were added on the fly under the change of infrastructure.
- Windows OS:
- Sysmon v6:
- Running processes (many advantages compared to standard Windows auditing).
- Creating files and changing the registry (included immediately on the entire host, no need to configure separately on each directory).
- Downloading libraries (I really wanted to see the use of various malware by the “hackers").
- Addressing the address space of other processes.
- Audit OS Windows (Advanced Audit Configuration).
There is also no magic here, everything is standard. The only point is that the audit of the Filtering Platform Connection category was included without fail, in which it is possible to correlate network events with the processes that initiate these connections. - A couple of libraries that are injected into the cmd.exe and powershell.exe processes at the start and log all the commands entered. Wonderful thing :)
- A few “surprises” for Attackers:
- We adopted the experience of surveys on projects and made ultrasound svcbackup in the domain with a password in the comments. And next to the rule is the detection of any activity under this ultrasound:
- Putting patches is good and right, but unfair to Attackers. Therefore, we decided to help the organizers, and updates on the OS began to look like this:
- We still expected that the attackers would quickly get inside the network, so through the domain LSA Protect was rolled out to almost all workstations to protect against Mimikatz. But this is not for all types of ultrasound, and only Logon Credential
HKLM\SYSYTEM\CurrentControlSet\Control\SecurityProvider\Wdigest /v UseLogonCredential /t REG_DWORD /d 0
Naturally, with monitoring changes in the corresponding parameter in the registry. - In case Attackers try to disconnect us from control when they enter the system (for example, by turning on blocking rules on FW), a script was prepared that runs according to the schedule and changes the policy to the original one if there are FW enable events:Warning spoiler!
$compname = Get-WMIObject Win32_ComputerSystem | Select-Object -ExpandProperty name $outputevent = Get-WinEvent –LogName "Microsoft-Windows-Windows Firewall With Advanced Security/Firewall" | Where-object {$_.ID -eq 2003} | Format-list TimeCreated, Message | Out-File $env:temp\JSOC_TEMP_FW #reg.exp = (?i).(.*value.*yes) $readoutputeventE = type $env:temp\JSOC_TEMP_FW | Select-String -Pattern "Value: Yes" if ($readoutputeventE) { & $env:systemroot\getrulesFW.vbe & $env:systemroot\System32\cmd.exe /c netsh advfirewall import $env:systemroot\JSOC.wfw Get-WinEvent –LogName "Microsoft-Windows-Windows Firewall With Advanced Security/Firewall" | Where-object {$_.ID -eq 2003} | Format-list TimeCreated, Message | Out-File $env:temp\JSOC_FW_LOG & $env:systemroot\System32\wevtutil.exe cl "Microsoft-Windows-Windows Firewall With Advanced Security/Firewall" Write-EventLog –LogName Application –Source "Application" –EntryType Information –EventID 666 –Message "Firewall enabled detected: START AUTORESPONSE!!1111" }
- We adopted the experience of surveys on projects and made ultrasound svcbackup in the domain with a password in the comments. And next to the rule is the detection of any activity under this ultrasound:
- Sysmon v6:
- Windows monitoring content eventually broke into several parts:
- Standard Solar JSOC content that works across all customers.
- Separate rules for detecting abnormal activity and attempts to fix in the OS.
It is not interesting to write about the first part, but let us dwell on the second part in more detail.
The general idea is as follows:- We detect options for fixing in the OS. To describe for a long time, I’ll better give a link to an excellent presentation of colleagues from the LC on PHDays, who have wonderful painted all the detection methods: www.phdays.com/program/231388
- We detect the network activity of system processes. To do this, they collected work profiles and set up exception lists according to what was recognized as legitimate.
- We detect the activity of system processes on the file system and in the registry. To do this, they also collected profiles and compiled lists of allowed activity.
- We detect a change in the registry branches that are responsible for autorun applications and libraries at system startup and / or user login.
- We detect changes to registry branches that are responsible for protecting against conditional mimikatz and its analogs, changes to powershell cmdlets, etc.
In general, this is a topic for individual articles, perhaps a little later we will describe in more detail.
After the entire audit was configured and the rules started, the first results appeared. We started fixing network activity from the svchost.exe process to a certain external server 208.91.197.46:9999, which was not in our profile. In ArcSight, it looked like this:
After a quick analysis, it turned out that the malicious code was used in svchost.exe through a malicious dll: c: \ windows \ FileName.jpg:
In parallel with it, another malware was detected on the same server. That's just the functionality did not have time to learn, removed immediately, as they saw :)c:\program files (x86)\wina\wina.exe (471d39a51a79f342033c5b0636c244dc).
www.trendmicro.com/vinfo/us/threat-encyclopedia/malware/troj_scar.alr + www.virustotal.com/ru/file/1154535130d546eaa33bbc9051a9cb91e2b0e3a3991286c3d5b0a708110c9aa7/analysis
Perhaps it was our biggest mistake of the opposition time, but we are "nailed" this surprise from the organizers before the start of the game. It is likely that he was part of some large-scale plan, but it’s too late to regret anything :)
With these results, we went to sleep the last time before the start of the game.
Confrontation
Monitoring the availability of infrastructure and the relevance of audit settings
From the experience of last year, it was clear that the infrastructure was unstable and could change at arbitrary points in time. But only if last year it grew (we received 6 hosts on the external perimeter within 10 minutes), then this began to rapidly decline. At the time of launch, the number of servers within the network decreased to 11, and the number of workstations - to 15.
In such circumstances, it quickly became clear that, despite the configured audit, reliable information about its state was needed. Indeed, if the virtual machine with any of the hosts is rolled back, then we will be left without patches, and without monitoring.
As a result, a decent part of the time was spent quickly dealing with emerging problems. The hosts periodically hung (the entrance to the console via RDP could take half an hour), they rebooted, and all this seriously spoiled the nerves. One thing is good - accessibility monitoring worked for all 100.
The first incidents - Web.
Expectedly, it all starts with external web services. They scanned constantly, tried to throw shells and deface sites periodically.
This is how attackers sometimes reveal themselves (successful operation of SSTI in photo hosting: defcon.ru/web-security/3840 ):
/var/lib/php5/sess_vhd4sts3mpk78n4qacc4o8knm0:logged|b:1;name|s:98:"{{_self.env.registerUndefinedFilterCallback('exec')}} {{_self.env.getFilter('ping 100.64.199.5')}}";username|s:4:"test";avatar|s:25:"images/default_avatar.jpg";
As a result, under the PHDays conditions, the rule worked quite well:
Now any attempt to gain control over the host and initiate connections to its hosts was detected quite well.
At some point it became boring to look at all this and chmod –x / bin / nc was done on the server.
Result:
[Tue May 23 21:41:36.273170 2017] [:error] [pid 4203] [client 198.18.78.12:23814] PHP Notice: Undefined index: avatar in /var/www/html/models/User.php on line 21, referer: http://203.0.113.155/login.php
sh: 1: /bin/nc: Permission denied
sh: 1: /bin/nc: Permission denied
Apparently, the people from the side of the Attackers were somewhat offended and the following requests were sent:
198.18.78.12 - - [24/May/2017:01:48:33 +0400] "GET /xymfrxtestfilehu****.php?hu***a=%63%61%74%20%2f%74%6d%70%2f%6d%79%70%69%70%65%7c%2f%62%69%6e%2f%64%61%73%68%20%32%3e%26%31%7c%6e%63%20%2d%6c%20%39%39%39%39%20%3e%2f%74%6d%70%2f%6d%79%70%69%70%65 HTTP/1.1" 200 1985 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
General results:
- Connection to the database (I recall, web-servers in the external segment, and their databases inside the network) on all servers did not differ in originality:
define('DB_NAME', 'wordpress'); /** MySQL database username */ define('DB_USER', 'wordpressuser'); /** MySQL database password */ define('DB_PASSWORD', 'secondstrongpassword'); /** MySQL hostname */ define('DB_HOST', 'o-srv05.office.cityf');
Once they even took advantage of it :) - Surprisingly, a large number of attempts to upload something to the server was detected by AV Kaspersky:
Object HEUR:Backdoor.PHP.PhpShell.eg is detected in /var/www/modx/temp/8f18b8d9d5a4cdbbe1f2962fa3868dd4/fonts/s.php Object HEUR:Backdoor.Multi.Mibsun.gen is detected in /tmp/.Xorg/tshd Object HEUR:Backdoor.Linux.Agent.ai is detected in /tmp/.Xorg/tsh/tsh Object HEUR:Exploit.Linux.CVE-2017-7308.a is detected in /tmp/pwn123 Object HEUR:Exploit.Linux.Dirtycow.a is detected in /tmp/cowroot - Web server error log monitoring + application launch audit - partially solve the problem of the lack of WAF.
- Without WAF, it’s very bad. It is simply impossible in some cases to quickly close a hole.
And where are the attacks inside the network?
Over the entire monitoring period, there was not a single penetration of attackers outside the Office perimeter.
Perhaps this is the biggest disappointment in the past PHDays, since working with callbacks, embedded malware and insiders is one of the very serious layers of the life of current information security and SOC. Communicating with colleagues from the Defender teams, we heard from everyone that the main vector is aimed at overcoming the perimeter: they will scan the web, sometimes sites are defunct, and exploit vulnerabilities.
At the same time, we regularly conducted our checks for detects on different hosts, and all of them passed successfully. Everything that we planned to detect, everything worked out.
Unfortunately, due to the absence of attacks, it will not work in detail to show the logic of detection. Therefore, a couple of examples of how such detections look in our infrastructure:
- An example of embedding a keylogger in the mstsc.exe process. Alert to an unsigned library -> We look at calls to the process that downloaded this library -> By PID we get further activity and the original malicious libraries.
- SandBox SIEM This is how ordinary cryptor looks like:
In general, the presence of unprotected infrastructure and a bank in the monitoring mode, rather than blocking, greatly changed the focus of Attackers. Most of the time they focused on hacking and auditing the security of this particular infrastructure, and not on breaking through the defensive redoubts of the Defenders and SOCs. As a result, some of the rich toolkit of the Attackers was not applied to the defenders' infrastructures, and some of the surprises of the defenders and SOCs in detecting and counteracting were not fully tested in combat conditions.
However, do not take this as criticism towards the organizers. They did a tremendous amount of work to launch the event, infrastructures and complex integrations, and with all the difficulties encountered PHDays confirmed its status as one of the key information security conferences. But SOCs and Defenders want more fire and hardcore, and we are ready to help in its invention;)