July 4, 2016 at 10:19

“Confrontation” through the eyes of the defender: A story about the PHDays CTF from the competitor



    This year the main hacking competition PHDays changed its format: we moved away from the usual CTF. Instead, a real battle of hackers and security guards unfolded on the forum. This time, three teams fought: “hackers”, “defenders” and SOC (security operations centers). Events at the competition site were as close as possible to reality. The teams had at their disposal an emulation of a city in which there is a bank, a telecom operator, an office of a large holding, an electric power company, and other facilities.

    Information security engineer for the telecom operator Andrei Dugin , who participated in CityF on the defense side, in his blog described in detail the course of the competition and his impressions. With the permission of the author, we have collected all of his publications in one habratopik.

    Description


    On May 17-18, 2016, at the Positive Hack Days conference in Moscow, the Capture The Flag (CTF) contest, characteristic of major events in practical information security, was held. This year, the organizers slightly modified the competition, giving it the format of “confrontation” between the forces of good and evil in a virtual city, as announced on the conference website PHDays .

    Naturally, there were many who wanted to try their hand at the competition. Participants were divided not into 2, but into 3 functional areas:

    • Hackers - naturally, their goal was to gain unauthorized access to protected resources.
    • Defenders - as the name implies, the main task is to prevent hacking and / or quickly eliminate its causes and consequences.
    • SOC - operational monitoring of suspicious information security events, notification of “defenders”, joint analysis and response to incidents.

    Since I participated in this competition as a member of the team of “defenders” of the infrastructure of a telecommunications operator, I will continue to talk about the event through the eyes of an information protection engineer.

    Advocates and SOC work in tandem to protect their services. Our tandem turned out to be really very productive, as is customary in Russia :)

    Based on the results of two days of “confrontation”, I’ll briefly say that we as partners in SOC got a team of professionals from JSOC with whom we understood each other perfectly (not for the sake of advertising, but statement of fact for). As tight as with JSOC in this contest, we did not always work even with “sister” divisions in our company. It sounds pathetic, but it’s true, you won’t erase words from a song.



    The team of telecom defenders “You Shall Not Pass” from large telecom operators and Solar JSOC, the team “False Positive”

    Hackers, of course, work separately from their opponents in the competition (so that there is no fight), trying to gain unauthorized access to protected resources .

    It turned out pretty interesting, but the details of the contest later. In many cases, I try to understand the initial cause of any transformation towards complexity. It is logical that the conference should keep up with world trends, but here, perhaps, in the wake of impressions from participating in the competition, I will say that Positive Technologies seems to have even stepped forward a half step (deflection counted?) - for how long? (deflection compensation to maintain neutrality).

    So, there are several obvious reasons, some of which are a consequence of others.

    1. To draw the attention of the state and business to:

    • the need to protect the “Internet of things” and other critical infrastructure management systems connected to public networks.
    • the need to comply with information security measures when working with banking systems;
    • financial effect of hacking banking systems;
    • possible consequences for organizations in which information security is developing on a residual basis.

    2. Let specialists feel what it is like to work with a constant threat of breaking into the infrastructure in some approximation to reality (why some - more on that later).

    3. Give the event scale.

    4. Market promotion:

    • information security products and services;
    • testing and popularization among IS services of such a service as SOC (Security Operation Center).

    Which of the above is the root cause, and which is the consequence - I think it’s obvious.
    I do not pretend to the completeness and breadth of analysis, because I described my viewing angle earlier.

    Training


    So, our combined team of telecom advocates in tandem with JSOC protected IP-based public services specific to the telecom operator: personal account, portal, paid messaging services, other VAS, etc. Of course, an integral part of our task was to ensure the safety of network equipment.

    As for the team: we had to work with competing colleagues with whom we had not been acquainted before, and when I first found out about it, I thought that at first we, as in Avengers, would re-exchange with each other, determining who cooler, and then we’ll start doing business. Fortunately, I was wrong, and the reason is simple: we did not have time for this. The unspoken principle was this: if you are cool - handsome, well done, we believe in you, recognize your talent and do not dispute, WORK! And I just reviewed the militants.

    In the “virtual city”, where the above events took place, there were 5 infrastructure facilities, for which there was a struggle:

    • city ​​office;
    • bank 1;
    • bank 2;
    • energy company;
    • telecommunication network.

    Naturally, without normal technical support, neither the city would work, nor means of protection. For operational monitoring of information security events, preventing and repelling hacker attacks, the organizers provided us with:

    • remote access for presetting before the start of the competition;
    • jobs for the full-time phase of the competition;
    • extras users;
    • network map;
    • network and server infrastructure;
    • admin login passwords for already prepared elements (network, server);
    • computing power for the deployment of information security and operational monitoring systems;
    • complete freedom of choice of remedies;
    • assistance in obtaining test temporary licenses from manufacturers of commercial information security tools.

    Of course, we got remote access to our segment a little in advance, for the commissioning of the information security systems we need. Otherwise, we would not have time to do anything really. It is a pity, of course, that part of this time fell on Easter and May. Confrontation with opposition, and potatoes require integration into the soil, and also in a short time. Each set their own priorities.

    Special thanks to our colleague Gennady Shastin , who, in fact, assembled the entire team and competently kicked everyone with the necessary strength and intensity in the necessary organs to the point of no return. I used to know him as just a good qualified specialist, and then organizational skills showed up. Gena, you, in fact, started a steam locomotive alone with the pusher, our team thanks you unrealistically.

    There were no jobs as such for us, only ethernet for connecting laptops and power sockets. But this is normal: we would only have puzzled the organizers to take the PC back. It’s more common to work behind your laptop. Only extras users who needed to portray the activity of the average user had jobs in the form of stationary PCs: read mail, click on malicious links, sit on social networks and complain that they pay little. Interacting with users was prohibited.

    The network map contained exclusively L3 information on the subnets and their topological location. What is in what network - you had to find out yourself. On the one hand - the complexity of the task, on the other - an extra injection of drive. That is, to find and neutralize the servers, we used ARP tables of L3-terminating devices and scanners, deployed infrastructure elements such as vses, etc., updated, reconfigured, integrated ... All this, including network equipment, was on virtual machines. At the installation meeting, Sergey Pavlov from Positive Technologies promised that there would be at least more computing power than he had on his home computer, and did not seem to be deceived. But this equipment was sweating from the load notably, we were not shy in the requests of the necessary virtual machines, and the hackers launched the car of scans at the same time.

    As for the means of information security: I thought at the beginning, when Gena was just starting to kick us, that the “Positive” as organizers would try to impose their products and solutions for their PR, but not only there was complete freedom to choose any means, but active assistance in obtaining temporary licenses from any commercial suppliers. This is also easily explained: by driving the scope of the use of their products, the organizers would limit the defenders, sharply turning the confrontation into public testing and give space for accusations of advertising and an excuse maneuver in case of a resonant hack. Yes, I myself would try to write off some tough jamb in defense in this case to the double decision that it is a sin to conceal. Plus, if you impose for a real confrontation - please train pretty deeply and for free,

    And so: dear defenders and “juices”, here’s a breath of freedom, look, do not choke. Everything will be fine - well, okay, if everything is not very good, you can always say: “But if you had our MaxPatrol scanner or MaxPatrol SIEM ...”, you might not say, depending on the situation. True, I have not heard yet that such a thing was said or even hinted at.

    In general, preparation for the competition was very good. Of course, not without nuances such as interruptions in infrastructure when the network smokes bamboo due to the load on the hypervisor, but these are working moments that are quite normal for the first event of this magnitude. This year we got bumps, now next year with preparation it will be easier, more predictable, and therefore, there will be time for imagination.

    Opinion on resonant breaking of hydroelectric power stations


    Fights without rules. A duel is organized between the fighters Orc and Goblin. The fighters are beginners, but the audience is trump and with the money. An influential lover of such fights, an official who protects the organizers, makes a very big bet that the Goblin will win. And then the Orc deals a good blow to the corps, after which the Goblin "floats". The judge, contrary to the rules, counts this as a blow in the groin, taking away to the side of Ork with a penalty point and giving time to restore the Goblin. In the process, Orc quickly whispers: “You need to lie down. Dad said. " And Orc, realizing that even if he wins and takes the prize, then in the future the fighter’s career in this organization is closed for him, he begins to quickly think, looking at the gradually recovering Goblin, how to substitute for a “knockout” blow, so that it is believable and convincing to the viewer.

    The very day after PHDays VI, the entire Internet was full of news about how a schoolboy hacked a power plant and how hackers, after a successful hack, stopped a hydroelectric power station and flooded the city.

    I participated in the "defenders" of the telecom, so the opinion is based on an analysis of the situation and analogies with my bell tower, but without knowledge of the details of the SCADA hack.

    As I wrote in the first partof its series of reviews of PHDays, one of the key goals for the organizers was to draw the attention of the state and business to the need to protect the “Internet of things” and other critical infrastructure management systems connected to public networks. Although I myself have not worked with SCADA protection, I see from public sources that when any device starts to be controlled by IP, and even more so via the Internet, few people pay attention to its security. As always, technology develops first, and then their safety. As a result, an “Internet of leaky things” is formed. What it can threaten depends on which device to access. This can be conveyed to the public through the news, but in order to attract a magazine, you need something that will be perceived as a sensation. And much more sensational than stopping the power plant and flooding the city with hackers,



    Image: Xakep.ru

    There is a team of "defenders" who have stored control systems. There is an SOC that has connected everything to event monitoring. Details of the preparation are described in the second part of the same name . As far as I know, these are commercial companies in the field of information security, which probably would like to promote at this competition, and therefore will take seriously the protection of industrial facilities. And here, in spite of all the seriousness of the protection described, hydroelectric power plants are broken 2 times ... Is it really so bad with ISSU in Russia?

    My options for the reasons for what happened:

    1. The defenders defended poorly: either SCADA does not exist in the nature of normal protective equipment or neglected the defense;
    2. SOC overlooked: either poorly integrated with the monitoring or clicked;
    3. Restrictions by the organizers on protective equipment;
    4. Dad said.

    With the first two, everything is clear. If the SOC and the defenders really clicked, then there is a way for them. But I have a suspicion that the restrictions imposed by the organizers implied the possibility of hacking the hydroelectric station by hackers. The bet was made by the “dad” that the hydroelectric power station would be disconnected and the city would be flooded.

    In order not to be unfounded with the statement of restrictions on the part of the organizers, I will say as a member of the telecom “defenders” team: there were restrictions on the use of protection, and they were quite strong. Use any additional means of protection, but from the fundamental means we were forbidden to even close the firewall from the world of SSH / RDP to protected resources, only to the network equipment itself, not to mention other unused services. Change passwords, update, patch, reconfigure as much as you like, but the service should be available ... But I generally wanted to set default deny, as is done in real life (details about the degree of proximity to real life later). But the organizers did not allow me.

    And strategically, this is right: otherwise there will be no show, hackers will whine, saying that it’s all dishonest, that the organizers gave a tremendous advantage to the defenders and juices. There will be no injection of the necessary information in the media, the leadership of industrial enterprises will dismiss protection. He sacrificed a pawn (SOC and SCADA defenders) in this game, but received a strategic advantage for several years in advance (future orders from business and the state for security). And most importantly - the world was thinking about how to prevent bad guys from leaving people without light and with an excessive excess of water in the city.



    Image: Xakep.ru

    If this all developed, then most of all I would not want the “defenders” and SOC defending SCADA to be blamed for breaking the hydroelectric power station and flooding the city.

    More attentive colleagues and representatives of the organizer corrected me, clarifying that the defenders of the hydroelectric power station did not really break. After the hacking, the defenders and the SOC very accurately described the hacking process (that is, they saw everything, but did not, because, after all, the show is necessary) and the hackers confirmed this. It was possible to hang the skad only after the defenders weakened, and then completely removed the defense. In the morning of the second day, it turns out, they announced it publicly from the stage. Looks like I, after a sleepless night, was not particularly able to perceive what was being broadcast there.

    But, nevertheless, after several days I see on the Internet information that the hydroelectric power plants were not protected, but that it was hacked. And over time, from this confused phrase, it remains only obvious that the hydroelectric power station can be hacked. Continuing the logic - which means it needs to be protected. That is, the shadow of the good name of the “defenders” and the SOC, if it has fallen, will dissolve very quickly. Probably, PR specialists know their job better than an information protection engineer (there wasn’t enough for it to be the other way around).

    Here it is, the official information from the organizers:

    The forum has clearly demonstrated what happens to unprotected critical infrastructure. Information security specialists are able to provide a very high level of protection without disrupting the process - but they are rarely given the opportunity to deploy like in PHDays. After disconnecting the protective equipment, the attackers entered the technological network of the ACS through the corporate network, attacked the physical equipment of the system, broke into the hydroelectric power station, carried out a water discharge, and disconnected the power lines.

    Limitations


    On the official website of the conference there is a rather pathetic quote:

    This time, instead of the usual CTF competitions, we are arranging real hostilities. Events at the site will be as close to reality as possible : large-scale emulation of urban infrastructure will unfold at the PHDays VI CityF training ground.

    That's about the highlighted in bold and I would like to talk.

    We were practically not limited in the means of information security: whatever you want, put it on, just say the requirements for the virtual machine. You need to kick the vendor for a temporary license - say, let's do it. Well, we were not shy.

    VMware. At the deployment stage, a limitation was voiced: the solutions chosen by us for the tender for the provision and monitoring of information security should be virtualized. Only portable devices could take hardware; the organizers were not responsible for them. In general, this is not such a big problem, because it’s an order of magnitude easier to get a vendor from a vendor, and the global trend is striving for universal virtualization, even entire networks and data centers are virtualized, doing all kinds of NFV, SDN and SDDC there. Some manufacturers managed to bother about virtual machines, and one even refused to provide. I was not ready for such a serious relationship with hacker attacks.

    Default permit. On the other hand, it was forbidden to execute default deny on the border firewall. But in the real world this is done first. It disturbed me to the core, I cried and cursed, but continued to participate in the competition. As I wrote earlier, one of the main goals of the “confrontation” was a show that would not work, I forbid everything unnecessary. Hackers would stumble into secret interfaces, do nothing, and charm the organizers in indulging the bright side of the force. So here the approximation to reality is practically zero. They gave hackers a head start on the very tomatoes. Because of such odds and hydroelectric power stations broke , and the city was flooded .

    No ban for IP-address. It was also forbidden to ban by src IP. Naturally, if it were simply forbidden by words, then I could also “forget”, especially at night, so that I could sleep more calmly, and indeed that I could sleep. But the cunning organizers got everyone into one IP: both hackers and their service-checker. If we extinguished too much, we ran to us and said that ay-ay-ay, that’s impossible. Moreover, the services checked in general any that were originally available on the servers, even in FIG not necessary. Shaw, cho.

    It also turned out to be a curiosity at the very beginning of the confrontation: the SOC discovered that someone was scanning us from an IP address that was not indicated in any way on the provided diagram. Having found the junction, I blocked it, removing the unknown subnet from routing, despite the ban. The junction point was inside the trusted segment, so the logic is simple: there is no on the diagram - I think for the bookmark and the intrigues of the hackers who should be squeezed out of our network. Naturally, at first the organizers came running shouting: “Everything is gone! Our everything does not work! ” Then they realized that they had not reassembled the joint with our segment, and agreed with my actions.

    Access to equipment. Of course, we were granted admin access to both network devices and servers. It’s hard to protect what you cannot configure. Perhaps, but difficult. Active network equipment at the L2-access level was denied access. The reason is simple: the show. When the organizers noticed discouragement on hacker faces and boredom on ours, they raised another vulnerable service somewhere without warning. Well, the SOC with the defenders, respectively, begin: “catch him, crush him!”. So that we would not interfere with the extra security settings and not see the excess, we would not be given access to the switches and, as an object of protection, they would not be handed to us.

    It seems that I have not forgotten anything. If I remember, I’ll add it. In general, although there were restrictions, they were not severe. Of course, the default permit still excites my worried worldview, which I periodically continue to grumble in conversations with the organizers, but the show is a show. It was wrestling, not MMA.

    Battle


    It is worth starting to describe the process of confrontation itself - the battle, so to speak, that lasted 29 hours - from 11:00 May 17, 2016 to 16:00 May 18, 2016.

    Previously, the defense and operational response teams carried out:

    • network research;
    • infrastructure inventory;
    • installation and pre-setting of necessary protective equipment;
    • reconfiguration and patch of leaky servers;
    • Change passwords that you can at least remember.

    Naturally, the first thing that was done shortly before the start was a performance check and an inventory of resources. Some servers, hitherto unknown to us, were immediately discovered, and many of them turned out to be rather full of holes on the fact of the scan. As I have repeatedly written, the organizers wanted the show, and without such tricks it would not be very fun.

    The network diagram (simplified, without a branch office) was approximately as follows: the



    Internet was pseudo-and not real, hackers got into it from a limited segment. If the Internet were real, it would be impossible to understand where the attack of the competitor, and where the hacker is, and there was no guarantee that someone would not freak out and start DDoS.

    First of all, I decided to keep the perimeter secret, that is, hang on the Internet side an ACL that would allow the ports of the DMZ servers detected by the scan, except for the control ports (SSH / TELNET / RDP / SNMP) and databases (MySQL, Oracle), and close the rest by default deny principle. He warned the organizers, and immediately received a refusal: it is impossible, all ports from the pseudo-Internet should be accessible. I can describe my reaction in two words “surprised” and “outraged”, which are expressed in one word together.



    Having told the organizers everything that I think about what our initially design is full of holes, that we are not allowed to properly protect the infrastructure, are forced to turn our asshole to the Internet, that these are all giveaways to hackers, etc., we adopted new rules of the game and started working :

    • patch leaky servers in DMZ;
    • hung ACLs on servers in the subnets Servers1 and Servers2, allowing access to them from DMZ and admins (organizers);
    • moved the security scanner, which was initially deployed in the DMZ, to the Defense_servers subnet - it wasn’t enough for the scanner to be downloaded;
    • Access to servers on the Defense_servers subnet was denied access from anywhere.

    As I mentioned in the first part , JSOC worked with us in tandem , which integrated the infrastructure with its SIEM and monitored suspicious activity. It is worth giving the guys their due, they not only took under their hood any sneezing (processes, inputs, network activity) infrastructure, they also monitored it with extrasensory responsiveness. At first, before we learned each other's IP addresses, less than a minute after the action, we heard questions from them: “who went to the xxx server from the uyy address?”, “Sshd changed on the server - is that what?” and so on ... Well, they answered them like: "OK, mine, it's me, updated the leaky server." If they also monitor commercial customers, then applause, I recommend.

    From time to time, when the organizers felt sad and the topics for the story ended on the stage, the hackers were sad and bored, Misha Levin snapped his fingers and a holey server appeared in the DMZ. He didn’t click over our ears, so we discovered this thing ourselves, and then began to think what kind of server to scan, patch, change easy passwords. Just like the host of the Hunger Games in the eponymous saga of Zoyka the Interchangeer, where something unnatural was produced if the tributes did not die for a long time.



    In general, the first day passed more or less simply, we didn’t even fix special scans, calmly secreted everything that was needed, and wondered: why is it all so quiet? Re-checked: what if we don’t see something? They suggested that the virtual infrastructure could not stand a large number of scans and the hacker segment glued fins. It suited us.

    Only at the very beginning, as I wrote in the fourth part, recorded a scan from a network section not shown in the diagram. And my logic as a network security guard is simple: since there was no site on the network diagram, it means a mess. You never know, maybe, according to the conditions of the competition, we had a bookmark in the network that needs to be found and neutralized. Therefore, I quickly discovered a static route to the problem subnet in the direction of an unknown device, first hung the ACL on the interfaces. They blocked it - and immediately the organizers showed that they had all gone and needed to be returned. After a short discussion, they came to the conclusion that this is already too much. If you want to scan us and check the availability of services, join the pseudo-Internet and from there at least continuously create ugliness. As a result, the joint was rebuilt, I removed the problem network from routing.

    Toward evening of the first day, it turned out that the hackers were told the wrong network, for the most part they scanned something non-existent and were rather depressed. Then we realized that now just the gust will begin. In addition, the farther, the more and more leaky servers began to rise, and by night our DMZ began to look like a colander. Servers took off both Unix and Windows. Scans of new servers showed vulnerabilities, mainly PHP, OpenSSL and SSH. Actually, we poured overnight work so that it would not be boring. Moreover, the constant risk that the dark side of the force would quickly detect these holes urged us on. In general, they did not really let us sleep. With an overnight stay at the competition, the SOC was all but his boss, and three from the defenders. The guys managed to get some sleep, but I, apparently, was disturbed by the fear of being stolen,

    PHP, as far as I remember, was 5.3. <Chehototam> and 5.4. <Chegototam> still shaggy 2013-2014. I had to update, because hackers will first break the web. Immediately until fresh, it was not updated, it was necessary to stomp through thorns in the steps of minor versions.

    SSH was almost everywhere vulnerable to CVE-2015-5600, which meant the lack of protection against password guessing and CPU overload. Just as the order picked up. The most interesting thing is that not all repositories had updates patched for this case, it was necessary to update from source. A similar situation is with OpenSSL. If you can’t get anywhere with the latter, you had to work hand-to-hand, and even fewer of them were an order of magnitude, then with SSH I was wildly bothered to do the same thing on all DMZ servers. And in the course of nightly thinking, I got the idea to wrap all SSH and RDP, which arrives on the DMZ server from the pseudo-Internet, through the Balabit SCB, which I installed, thinking that I could transparently wrap control protocols through it for control. I still wonder how this idea came to me, but it turned out to be very effective. Resulting architecture,

    Despite the fact that Balabit is used as a means of controlling control sessions, and can additionally act as an integration layer for managing poorly / partially / in any way integrated networks, the invulnerability of the zorp-ssh proxy module to the night turned out to be a key factor in its application CVE-2015-5600.

    Since the technical solution turned out to be rather tricky and the possibilities were much more than just protection from the mentioned vulnerability, I will write a separate post about the resulting inversion of control traffic. Gene called it the "architecture of the engineer Dugin."

    At night, as expected, scans were flooded, and on the second day, an uprising of cars began, when new leaky servers began to grow, like mushrooms after rain. Misha Levin waved his sleeve - and all the trump cards spilled out into the DMZ subnet.

    In general, on May 18, it was pretty intense. It’s good that there are more of us during the day, you can parallelize tasks and at least have lunch quickly. In fairness, I’ll say that one of the leaky web servers that grew up on the second day, the dark force still managed to swing through the web, getting the only “flag” from our tandem. The server was in no way interconnected with the telecom business and reflected on us only in the form of a checkmark about the unchecked flag. JSOC discovered it after 5 minutes, we immediately diagnosed the cause, after which we activated the blocking signature on IPS (it was defaulted in monitoring mode and sent an alert) and patched the server.

    Separately, I want to note those hackers who used social engineering methods, grinding themselves near the tables of defenders and SOC. How realistic it would be to get an outsider in the corporate environment to the table of a real advocate or SOC, I think, is understandable. Despite attempts to mingle with the crowd, distinguishing them was quite simple. The main weakness of the defender in the fight against such methods is the employment and dedication to the technical process. But we still noticed them.



    At 16:00 the competition ended, and a few minutes before that, the entire network infrastructure living on virtual machines could not stand such a long abuse and began to crash wildly. This was not a hack, because we previously protected the network equipment in the first place.

    Access became possible only through the hypervisor, and while we and the organizers networker restored it, it was time for all participants to relax, which we started to do with great pleasure.

    Custom SSH Security Method


    “Why can't we reconfigure network equipment if we need to protect it?” No way without it.
    - Ah, you are telecom defenders. Then you can, but services should be available so that management is not lost. Otherwise, we will come to you and ask: "What is the matter?" If you can’t quickly fix it, we’ll return to the settings before the start of the competition, and you will set penalty points. Therefore, be careful when changing network settings.
    - It's not a problem. And if for security we need to deploy traffic in a different way, tie a knot, but will everything work?
    - Then we will come to you with a piece of paper - to write down what and how you did, it is interesting after all.

    - Why did you decide to use Balabit at Positive Hack Days? How will he help there?
    - We will monitor admins. They promised to periodically cock up holey services on servers.
    - Well as you know.

    These are two real conversations in preparation for PHDays. Until a certain moment, I myself did not think that they would be so tightly interconnected. Naturally, after a conversation with the organizers, I thought that it would be necessary to surprise, since I threatened. In the course of the competition, the mood vector slowly shifted from the initial desire towards “concreting”. But miraculously it turned out that in the end they merged. And this will be the story.

    As I wrote in Part 5 , and said on the PHDays scene, a rather interesting and non-standard solution was to protect vulnerable Unix servers managed by SSH.

    Let me remind you that architecture itself was simplified as follows:



    But, as mentioned in part 4 and part 5, the organizers did not allow us to use ACLs on the firewall with the default deny rule. Therefore, initially from the pseudo-Internet, traffic went to DMZ servers via all protocols and ports. But now we will talk about SSH.

    The servers that were delivered to us in the DMZ were almost all vulnerable to CVE-2015-5600, which meant the lack of protection against password cracking and CPU overload. We combine these 2 conditions - and we get the ideal situation for hackers: “I can’t pick it up - I’ll at least fill it up.” At the same time, starting from the evening of the first day, the servers appeared like cockroaches in a hostel. SSH was leaky everywhere. Updates containing fixes to the indicated vulnerability were far from all repositories. That is, to go to all Linux of different versions one-time and trample a couple of commands so that it is not an option further. Only raw, only hardcore. Laziness is the engine of progress. Perhaps a person who works more closely with Open Source solutions and Unix in everyday life would go towards implementing his repository or something else, but the architect woke up at 3 in the morning.

    I remembered the lonely-standing Balabit SCB, which, despite the ignore of the first day, spun around and ate some of the resources of the organizers' computing infrastructure. Since I set up the very first Balabit installation in the CIS, not counting the others for several years, the idea of ​​trying to use SSH through it looks quite natural.

    Having set up a couple of proxy rules on Balabit, I scanned it. The SSH zorp-ssh forwarding module turned out to be invulnerable to CVE-2015-5600, as well as to other vulnerabilities known to the MaxPatrol scanner that day. Now it remains to figure out how to split the traffic in such a way that SSH to the servers in the DMZ is balanced and the rest of the protocols go their usual way.

    Let's look at SSH as an example, but the same thing can be done with RDP, Telnet, VNC, and ICA (albeit a little sweat).

    According to the idea implemented in the network diagram, hackers got to the servers from DMZ via all protocols and ports, passing through the border firewall.



    Since CheckPoint was not used as an edge firewall, but CheckPoint, the standard features for the firewall like NAT could be configured to their full potential. Like a firewall protecting a data center server — Cisco ASAv.
    In order not to confuse two different manufacturers in network terminologies and NAT implementations, I will describe a vendor-independent concept applicable on equipment of any manufacturers supporting such technology.

    To invert SSH traffic to Balabit, you can go in the following ways:

    1. Dedicated ports for servers.

    • Highlight a range of ports on Balabit.
    • Implement dynamic many-to-one NAT on the border CheckPoint so that packets arriving from the pseudo-Internet in the direction of the DMZ servers on the SSH port are broadcast in the packet, where src ip is replaced by IP CheckPoint, dst ip - by IP Balabit, dst port - to the port dedicated to this server on Balabit.




    • Open access for these TCP sessions to enter the Firewall data center:



    • Configure proxy rules on Balabit and return to the correct server, port 22 (SSH):



    • Allow such packages to exit from Balabit to the DMZ side on both firewalls:



    Done. The SSH session is not functionally broken, other traffic is not affected, the game conditions are met.

    2. Dedicated address pool.

    • Select in the data center a subnet of the same dimension as the DMZ.
    • Place the Balabit interface that receives traffic into it.
    • Set up routing.
    • Configure Balabit with the IP addresses of the same subnet.
    • Implement static one-to-one NAT on the border firewall:



    You can, of course, not hide the SRC IP for the CheckPoint address, but then it is necessary that the DMZ is built on gray addresses. On the “confrontation” DMZ was built on the “whites” ...

    • Allow incoming traffic to the Firewall data center:



    • Configure proxy rules on Balabit:



    • and permission on both firewalls:



    where Balabit_ip0 is the Balabit address configured on the interface as the main one, and not IP alias (Balabit_ip1 ... 4). You can, of course, configure src ip to be Balabit_ip1 ... 4, but in the situation on PHDays this was superfluous.

    Done. The SSH session is not functionally broken, other traffic is not affected, the game conditions are met.

    ***


    В обоих случаях, чтобы иметь возможность пускать определенные подсети на сервера DMZ мимо Balabit, необходимо на пограничном firewall перед описанными правилами поставить такое:



    Учитывая то, что остальные правила срабатывают по умолчанию после описанных, в итоге схема прохождения трафика получается следующей:



    Видно, что подобное решение немного напрягает при масштабировании его на большое количество серверов. Нельзя сделать «настроил и забыл» для всей подсети одной строкой. Дополнительные затраты времени вызывает необходимость:

    • В случае 1 — добавления правила для каждого нового сервера DMZ и на Balabit, и на firewall.
    • В случае 2 — разработка сетевого дизайна и настройка сети (единоразово), а также добавление нового правила на Balabit для каждого сервера DMZ.

    That is, in both cases - configure Balabit for each server. The task, in fact, is not cumbersome, it takes 5-10 minutes, and if the server’s life is static, it’s quite tolerable, but if everything often flows and changes ...

    Both of these restrictions are related to the fact that the network from the point of view of L3 was flat, with one routing table . There is a solution that would help to transparently balabitize the entire subnet in one fell swoop, and there at least every second add-remove servers in the subnet. But for this, you will need to configure MPLS L3VPN or vrf lite on the active network equipment (if supported) one time and carefully connect Balabit and the border firewall with it.
    But this is a topic for a completely separate article ...

    I see a dumb question in the eyes of readers: how did you tune in to PHDays?
    Answer: according to option No. 1 - dedicated ports.

    conclusions


    So, having described our participation in the confrontation in six parts, it would be nice to draw conclusions on the results of the competition. Despite the fact that we were not defeated, I realized that we still have room to grow as defenders. The confrontation made it possible to work under the conditions of an unrealistic administrator’s jamb that were considered unrealistic: default permit on the border firewall and the uncontrolled appearance of holey servers in the DMZ. As a result, to check the degree of infrastructure security, a certain checklist is needed, with which you need to check.

    In order not to spray on a large number of words and phrases, I will take SANS Top 20 Critical Security Controls as a basis, which describes information security control points. They apply to both competition and real life. On the tablet I’ll try to differentiate between the areas of responsibility of the defenders (DEF) and SOC, although they are often closely intertwined.

    In a good way, both the defenders and the SOC should be preoccupied with all issues, but I put the pluses in accordance with the realities that the conditions of the competition dictated to us. If somewhere there are no pluses, then it’s not applicable or I didn’t notice the applicability precisely in the conditions of “confrontation”

    No.Control nameDefSOC
    1Inventory
    of Authorized and Unauthorized Devices    		++
    2Inventory, of Authorized and Unauthorized Software++
    3Secure
    Configurations for Hardware and Software on Mobile Device Laptops,
    Workstations, and Servers    		++
    4Continuous
    Vulnerability Assessment and Remediation    		++
    5Controlled
    Use of Administrative Privileges    		++
    6Maintenance,
    Monitoring, and Analysis of Audit Logs    		+
    7Email
    and Web Browser Protections    		+
    8Malware
    Defenses    		++
    9Limitation
    and Control of Network Ports, Protocols, and Services    		+
    10Data
    Recovery Capability
    11Secure
    Configurations for Network Devices such as Firewall Routers, and Switches    		+
    12Boundary
    Defense    		+
    13Data
    Protection    		++
    14Controlled
    Access Based on the Need to Know    		++
    15Wireless
    Access Control
    16Account
    Monitoring and Control    		+
    17Security
    Skills Assessment and Appropriate Training to Fill Gaps    		++
    18Application
    Software Security    		+
    19Incident
    Response and Management    		+
    20Penetration
    Tests and Red Team Exercises    		++

    In addition to the conclusions, of course, it is worth identifying the mistakes we made and the further defense strategy in future “confrontations”. I think competitions of this format will be quite popular now.

    What mistakes did we make in the confrontation with hackers:

    1.
    2.
    3.

    In order to avoid such mistakes, I see the following list of actions for the future:

    1.
    2.
    3.

    If you have problems reading the text while reading the points above, but you think that there should be something there - check not so much the technical parameters of your PC’s browser as the coefficient of naivete perception of information.

    Agree, it would be foolish of me to disclose to the whole Internet, including opponents, all action plans in future battles :).

    PS Aggregating my entire subconscious flow based on the results of participation in PHDays "Confrontation" competitions, I translated it into a more or less official language and described the semantic content in one article for the "System Administrator" magazine. The text of the article can be found in the June issue here .

    Posted by : Andrey Dugin
    Tags:

    Also popular now: