Reliable and secure Linux (our answer to Chamberlain)

After reading recent articles on the topic of Linux system security on Habré, I had a desire to share my point of view on this issue.
The article as a whole is designed for novice administrators, so it outlines things obvious to a good specialist. Valuable additions and comments are welcome.
I will hardly give copy-paste excerpts from the config files for three reasons:

1. This will lead to excessive growth of the article.
2. Nobody has canceled mana and google
3. Point 2 is very useful for the development of a specialist.

So, how to increase the security and reliability of a server (and a workstation) based on Linux?



I will divide this task into 3 parts:

1. (Pro) active security - tightening system settings and daemons. This also includes setting up a firewall.
2. Passive safety and reliability - system monitoring.
3. Backups

Tighter system settings

1. Disable unused services / daemons . Carefully review the list of processes (for example, using the ps -ef | less command) and identify those that you do not need. Make sure that they are not needed by the system itself. Disconnect.
2. If possible, then change the standard ports and interfaces on which the remaining services are listened and configure additional security restrictions using the services themselves (that is, by editing their configuration files / adding keys to startup parameters).
   I will illustrate with the example of sshd. Here it is necessary to do the following - change the standard port from the 22nd to any free one, for example to 6622. Deny access with the root login. Hard-set the list of user names allowed for access. Allow sshd to listen only to a specific address. As an option, allow access only by key, but I do not really like it
   UPD Keys are unanimously recognized as an excellent way to access the server, so use them (but do not permanently disable password entry).
   
3. All services that are used only by the system administrator or a limited circle of persons should be accessible only through vpn or ssh connections.
4. Wherever possible, switch to using an encrypted connection (pop3-> pops3, http-> https, etc.)
5. Iptables
   Configuring the firewall is a very interesting topic, which you need to devote a separate article to.
   The basic principles are as follows: The
   firewall should work on the principle of a white list, that is, everything that is not explicitly allowed is prohibited.
   This can be achieved in two ways:
   Method # 1
   Setting the default firewall policy with the iptables -P INPUT DROP, iptables -P OUTPUT DROP and iptables -P FORWARD DROP commands. After the execution of these commands, all incoming, outgoing and transit traffic for which no permitting rules have been created will be blocked, therefore, they should be followed with caution.
   Before at least doing something with iptables, I strongly recommend that you study the excellent guide from Oskar Andreasson
   When reading it, pay attention to the purpose (action) of the LOG, which allows you to write to the log various data on packets entering the system. This greatly helps with the initial writing of the firewall. I can advise about this procedure for setting up a firewall:
   a) Write all the necessary rules
   b) Add a logging rule to the end of each chain (it will catch all packets that you did not take into account in the previous rules and write information about them in the syslog)
   c) Turn on the firewall
   d ) Find unaccounted connections in the syslog and add rules for them in the firewall.
   Paragraph (d) is repeated until the entries in the syllable disappear.
   It is clear that you should not thoughtlessly add everything that appears in the syslog because the results of attempts of unauthorized access will be visible there.
   I note that the default DROP policy should be enabled only after the successful completion of item (d) - this ensures that existing services will continue to work while debugging the firewall.
   I also note one point that is not quite obvious for beginners:
   Here is an excerpt from what a standard firewall template should contain. These rules allow inbound and outbound packets that belong to already established sessions. This works of course only for stateful protocols, in particular for the tcp protocol.
   iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

   
   
   If you do not use these rules, then for each application you will have to write 2 rules - one for the INPUT chain, and the second for OUTPUT.
   If you use them, then you can limit yourself to only one, which will allow you to establish the original connection.
   
   Method No. 2
   At the end of each chain, write a rule that sends all packets to REJECT.
   That is, in the standard case, you need to add three rules - iptables -A INPUT -j REJECT, iptables -A OUTPUT -j REJECT and iptables -A FORWARD -j REJECT.
   As a result of adding these rules, all packets that are not explicitly allowed by the previous rules will be rejected with the message port-unreachable.
   Everything else needs to be done exactly the same as in method No. 1, including adding rules with --state RELATED, ESTABLISHED.
   
   Method number 2 is the most correct ideologically and I advise you to use it, but the choice is yours.
   
6. SELinux
   A very interesting and promising thing. Restricts access rights of processes anywhere. Each action that a process is allowed to do should be spelled out in a policy mapped to it.
   There is another similar system - Apparmor (as far as I know, it is used in ubuntu linux).
   Honestly, I have not worked closely with both systems yet (therefore, I can’t give any special recommendations), I partially used SELinux in one project, but due to the time pressure in the time I was in, I had to temporarily postpone this task. Soon I will definitely return to setting up SELinux because I really liked it.
   

Monitoring

Hacks and accidents, as you know, are better to prevent than to eliminate their consequences.
In this regard, a competently configured monitoring system is very relevant, which, at the first sign of a problem, will immediately signal the administrator.

1. Nagios
   There are several large monitoring systems (nagios, zabbix, cacti, munin).
   I tried all of them and eventually settled on Nagios. Very convenient and flexible system to configure.
   A huge number of plug-ins for monitoring, and if there is no suitable plug-in, you can write your own in any language convenient for you (for example, I use bash and python for this).
   At a minimum, you need to configure monitoring of processor load, memory, swap, free disk space, load average. It is highly advisable to monitor the availability of critical services (e.g. apache, mysql, nginx, tomcat, etc.).
2. MRTG / RRD The
   dynamics of various indicators can be conveniently viewed on charts generated using MRTG.
   MRTG allows you to view in a convenient form the dependence of various system parameters on time.
   For example, you can see how the CPU and memory load changes depending on the time of day. You can draw graphs for almost any indicator - from the temperature of the processor, to the number of queries to the database. MRTG is an extremely useful tool for analyzing the state of a system.
   MRTG has several limitations and disadvantages that can be circumvented by using the RRD utility from the same author.
3. Smartd + smartmontools
   Allows you to monitor the status of hard drives and identify suspicious readings at an early stage. Can be integrated into Nagios.
   For example, a non-zero value of the variable Reallocated_Sector_Ct indicates that bad sectors appeared on the disk, and appeared for a long time because smart only finds out about the presence of bads after the factory remap table is full (here I may be a little mistaken, the theory about this has long been I didn’t update it, maybe with modern hard drives the bads are immediately visible through smart).
   It is possible and necessary to configure smartd so that all suspicious readings are immediately sent by e-mail to the administrator.
4. Log analyzers
   It is clear that manually monitoring the system logs for errors is a thankless task.
   For this purpose, many log analyzers have long existed.
   I advise you to put several systems at once and choose the one that you like more.
   You can start with logwatch.
5. Remote syslog
   What do you think is the first target of an attacker who has penetrated a system?
   The goal is to hide your presence .
   To do this, he will surely delete all references to his actions from the system logs (and also try to replace some system utilities, but more on that below).
   In order to protect yourself from the negative consequences of deleting logs, you need to organize their recording on a remote server. It is desirable that ssh access to this server be impossible (you can access it through IP KVM yourself) or be very difficult, otherwise nothing will prevent the cracker from deleting the unmodified copy of the logs stored there. On the log server itself, it is most convenient to store the logs in any DBMS.
   
6. HIDS and NIDS
   HIDS monitors the state of the system (logs, system utilities integrity, etc.) and informs the administrator about suspected security breaches. It will help if the attacker from the previous paragraph replaces any system utility in order to hide his presence or secure a login to the system. For example, you can replace the utilities ps, who, w, last so that the administrator cannot see who is currently logged in to the system. Replace the iptables and sshd utilities so that they allow the attacker to log into the system freely.
   HIDS, of course, will not be able to prevent such actions, but it will be able to notify the administrator about them.
   One example of HIDS is OSSEC.
   
   NidsIf HIDS monitors the internal state of the system, then NIDS analyzes suspicious network activity (port scans, password attempts, various attacks against system services, etc.). The most famous NIDS is Snort.

Backups

1. As you know, system administrators are divided into those who do not backup, those who already do them and those who think that they are doing.
   This saying is filled with deep meaning.
   Backups may be needed in many cases, for example, after hardware problems with the system, after hacking it, after incorrect actions by the administrator or developer.
   You can conditionally divide backups into file system backups and database backups.
   It is very desirable to have a separate server for storing backups, ideally also located separately from all other equipment. Separately, it means at a distance of 5 kilometers, no less.
   
   A lot of various software has been written for backup, but I prefer to use scripts written by me personally. So I get full control over the process.
   For a file system backup, it is easier to use rsync with the appropriate keys. In most cases, it is more convenient to do an incremental backup.
   For database backups, it is best to use the utilities provided by the DBMS manufacturers, but in the case of mysql, I have slightly moved away from this rule.
   The standard backup utility - mysqldump does not behave very well when removing the backup, namely, it "locks" the tables for writing, forbidding writing to them at the time of backup (although in the case of innodb this could be completely avoided, but mysqldump is not able to to do this). This is very important when using large databases, at least 10-20 GB in size, where the table lock can last 10-20 minutes. In addition, the restoration of large databases from such a backup takes many hours.
   In such a situation, it is more reasonable to use the master-> slave system.
   On the backup server, mysql slave is configured for the database that needs to be backed up. In the future, backups are no longer removed from the main base, but from its slave. In the event of an accident, you can even not immediately restore the database, but simply redirect all requests to the slave server, which will greatly save time. It is convenient to use the Innobackupex utility from Percona to remove backups from the slave server and for its initial preparation . For example, it allows you to make a backup suitable for the subsequent raising of the slave server without stopping the main database (without locking it for writing).
   There is another very important point - it is necessary to periodically check backups for their correctness, otherwise you run the risk of finding empty or broken archives at the most inopportune moment.
   

It is very important to keep documentation (for example in the wiki) for all servers served and do not forget to update it with any changes in server settings. Very often there are cases when the system administrator cannot remember the details of setting up a system with which he has not worked for a long time.

That's all. Once again, I ask you to express all valuable additions in the comments to the article. I would very much like to bring the article to the main page so that as many experts as possible notice it.

PS The article made some changes based on the results of discussions in the comments. Thanks to everyone who spoke.