We found a large company that has not been involved in information security for 5 years, and it is still alive.
About 5-6 years ago, there was a really cool admin who set up the network like a clock and equipped it with modern economy segment equipment. The administrator compensated for the lack of budget with good configs and proper architecture. In general, it can be seen that a lot of work has been done.
Then the company split into two, expanded, everything changed a couple of times in it - and during all this time the network was supported on crutches. Since IT is not our customer’s core business, the situation is generally clear. There is so much where, but for a large network (geographically distributed company, dozens of branches) to last 5 years in this form - I have not seen this yet.
Actually, it did not hold out. We were invited to audit the network infrastructure after a hacking event, when their databases with all the information representing commercial secrets were simply downloaded. More precisely, the wrong people surfaced.
Circumstances, parts of the topology and other details are slightly changed so that it is impossible to recognize the customer. Nevertheless, the post is based on real events and as close to reality as our security guards allowed.
The company represents the main production (in the same server node) and dozens of branches across the country. Thin clients are installed in the branches, which go through VPN + RDP to the server node, where users work. There is also equipment in the branches that uses the services and databases of the central site. If the connection disappears in the branch, then it simply dies before connecting the network again.
Network - dozens of prehistoric switches, dusty "dumb" routers and MPLS L3 VPN connection from the provider for transferring data between sites.
Root switch in another company
As I said, the company was divided into two independent and slightly competing several years ago. So, parts of the network remained common, because then they decided not to break them. A strange, but logical result: the root switch belongs to competitors. Users of competitors can send tasks to network printers, admin of competitors - with some effort to watch their network balls and so on. Servers with video surveillance archives are generally still common. Laboratory mini-cluster too. III ...
Networks are not demarcated. More precisely, how: at the junction there is an old piece of iron, which until the new generation of firewalls was used, including for protection. In essence, it works like a simple statefull firewall. Traces of a normal config were found on it, which were commented out, because, apparently, at some point they prevented the development of the network.
Here are excerpts from the report:
- At the current switch settings, the network infrastructure is vulnerable to attacks on the most active network equipment: DoS / DDoS, unauthorized access, and attacks on user traffic and server equipment: DoS / DDoS attacks, traffic interception attacks by substituting MAC and IP addresses, attacks on the DHCP service.
- To prevent unauthorized access to active network equipment, it is necessary to configure centralized authentication and authorization, as well as limit the IP addresses of devices from which administrators can establish management connections to devices.
- To prevent unauthorized access to the network and attacks on active network equipment and devices on the network, you must turn off unused interfaces and place them in an unused VLAN.
- To prevent attacks on the DHCP service and subsequent compromise of the traffic, you need to use the DHCP Snooping functionality.
- To prevent attacks on the substitution of MAC addresses and attacks on the switching tables of switches with subsequent compromise of traffic, it is necessary to use the port-security functionality.
- To prevent attacks aimed at redirecting traffic by sending malicious ARP responses, you must use the Dynamic ARP Inspection functionality.
- To prevent attacks on switches and user traffic, you must use the spanning-tree BPDUguard functionality.
- To prevent DoS and DDoS attacks on the network infrastructure, as well as in connection with the likelihood of switching loops, it is necessary to use the storm control (unicast, broadcast, multicast) functionality.
- The site uses outdated switch models with outdated OS versions. It is recommended to update the OS versions on the switches, but even after that, these switches will not be able to provide the security of the data network necessary at the second level of the OSI model.
- Communication with other sites and Internet access at this site depend on the network infrastructure of another company. It is recommended that you connect to service providers on the local root switch.
- There are a number of third-party devices in the guest VLAN on the network. With endpoint devices supporting dot1x supplicant functionality, you must configure dot1x authentication for all of these devices with VLANs and dynamic access control lists that limit the traffic from these devices. In the absence of support for this functionality by end devices, it is recommended to configure access control lists and port-security, as well as QoS policing on the interfaces of the switches to limit the amount of allowed traffic.
- As access points, devices are used that do not support centralized authentication of administrators, which allow connections using the HTTP protocol, as well as HTTPS based on the SSL v1 protocol.
- SSIDs are advertised in broadcast mode; it is recommended that they be advertised in private.
- WPA2 keys used for authentication coincide both for the internal SSID and for the guest, and are 10 letters of the Latin alphabet (the name of the founder).
- The current IPsec VPN settings on the router in phases 1 and 2 of the IKE protocol use 3DES as the encryption algorithm, MD5 as the authentication algorithm and DH 2. It is recommended to use AES-256, SHA-512 and higher and DH 19 and higher. It is also necessary to enable PFS functionality with a group of DH 19 and above.
Remote access OpenVPN (Debian Linux 8). Depending on the connection port, different routes are returned to different clients as split-tunneling, thus implicitly allowing access only to certain networks for remote connection. But the client can manually register additional routes to the logical tunnel created when connecting to the OpenVPN server, thus gaining the ability to route traffic to any company network.
Traffic from users and servers at all sites is transmitted to the Internet and from the Internet through routers, without being tested and inspected by modern firewalls and intrusion prevention systems. Also, this traffic in most cases is either unlimited or almost unlimited. In this regard, the devices, applications and company data are practically not protected from a number of attacks that can be used against the company. These are attacks that exploit vulnerabilities in applications and operating systems, DoS and DDoS attacks on servers, applications and active network equipment of a company, attacks on user data using viruses and network worms, etc. All company traffic transmitted from or to public networks, as well as traffic between different sites of the company and traffic between the company and its partners should be limited and checked by the latest generation of firewalls and intrusion prevention systems. Due to the high cost of such solutions, it is advisable to use a centralized pair of devices (for fault tolerance) at the central office and route traffic from all sites to the Internet and to other sites of the company through these devices.
How was the audit?
Stage One. I sent the customer questionnaires and asked them to fill out before I carry out the work. The questionnaires were in the format of doc files and Excel tablets, the amount of information depended on the objects. For some equipment there was a lot of information, for some - for zeros. In general, filling out such questionnaires is a standard procedure, they indicate IP addresses, the number of servers, where, what and how, well, and so on.
Not all IT specialists involved in the network configuration process were full-time employees of the customer, there was also outsourced support. OpenVPN was configured for me by just an external engineer. I requested that he give me remote access and configuration files. There were no special problems. I received the information within a week, after connecting to the equipment. If there were questions, I solved them with the local administrator. Several portions of configurations from the Linuxoid met several times, and in order to understand them, I had to connect my colleagues. These were traces of the very mythical admin who launched everything like a clock many years ago.
In the second step, I requested the configuration of the proxy server and OpenSSL VPN server, which I subsequently analyzed. It took about another week.
Stage Three. He climbed onto all the hardware and looked. This is an optional procedure in such an audit. In principle, everything should be in the configuration file. However, there are such things that you can’t just see it. Some of the hardware was turned off, some in the forwarded documents showed default settings, not actual ones. Nobody kept a list of these settings, they were not indicated anywhere. Therefore, it was better to check everything manually. So did. On each interface of the equipment of any vendor, statistics are received and sent traffic. If the statistics are zero, it is clear that the equipment is not used and it can be turned off. There were also interfaces where the counters were not zero, but nothing was connected to the equipment. I reset the traffic counters and checked if the number of bytes through the interface increased or not.
I also watched what protocols there are, how authentication was configured. Conducted a full audit on each piece of hardware: remote access protocols, protocol settings, authentication, authorization, whether interfaces are turned on / off, NAT, Wi-Fi - that's all. Access lists on routers are very important. Either they were not, or they were very basic. It took three weeks to complete the reports. And, yes, I still checked the OS, firmware versions, can I upgrade and get more modern functionality, is there a suitable version.
The fourth stage - aggregated information and made recommendations for each "piece of hardware".
How did they survive?
Most likely - on the principle of the Elusive Joe. 5 years of luck ended in a hack, about which it became known. Much more interesting is how they over the past five years did not pick up another epidemic of blockers or catch miners or random DDoS that would “put” them?
In general, I can say that small companies (especially different industries in distant regions) live like that. Even private small banks sometimes work like this, colleagues told such tales. But to make a company of this size and with such a turnover is very strange. Elementary, all competitors should be gentlemen, so as not to put their infrastructure on one left.
How did the story end
They are now buying again modern equipment at the corporate level. There will be a couple of firewalls, there will be new, more functional routers, there will be the correct settings and rules for allowed traffic. Internet access routing will go through the main server node for the whole country, because a couple of firewalls are only there. Already, they have closed all unused ports and generally updated the settings. Update firmware where possible.
The new switches will have centralized authentication, logging, and delimitation of the rights of users to manage. The support will finally have different accounts - now they work one at a time. If you delete the config now and reboot the device, no one will know who it was.
After that, they will slowly save up for DLP and other features.