Fail2ban [incremental]: Better, faster, more reliable

Much has been written about fail2ban, including on the hub This article is a little about the other - how to make protection even more reliable for them and about the yet new fail2ban features that are still unknown in wide circles. I’ll add right away - we will talk about the development branch, although it has already been tested in battle for a long time.

Short introduction

For the most part, fail2ban is installed from the distribution kit (usually it’s some kind of stable old version) and is configured according to mana from the Internet in a few minutes. Then it works for years, without administrator intervention. Often even logs that seem to be followed by fail2ban are not visible.
So, I was prompted to write this post by a case that happened with one server of my good friend. Classics of the genre - abusa came, followed by the second and went. Well, the attacker also got lazy - he didn’t rub the logs, and he was even lucky that logrotate was configured to store logs for months.

It turned out pretty commonplace, picked up a password for his admin mail, which was also a password for ssh (naturally without a key). Not root, but sudoer, with all that it implies. His first question was: how to pick it up - I have fail2ban there. And here is just an ambush: not everyone imagines that today it is not individual computers, but entire botnets that are engaged in password selection, which, by the way, have become smarter. So we found out from the logs that this is exactly the case: I was sorting through a botnet, and in practice, I found out its settings in fail2ban (maxRetry = 5, findTime = 600 and banTime = 600). Those. to avoid a ban, the network made 4 attempts within 10 minutes from each IP. For a minute on the network about 10 thousand unique IP = something more than 5 and a half million passwords per day.
In addition, his mailer did a lot of stupidity - namely, a pause of up to 10 seconds, with a login with the wrong name. Those. To find out that some names, including admin, really exist, this grid was not difficult. Next was a focused enumeration of only passwords for existing names.
I will not dwell on the “repair” in more detail - this is a long story, and in general a topic for a separate article. I can only say that they cleaned everything and everything was resolved with a little blood, and he escaped with almost a “slight” fright.

So, the idea to write an article arose after I (partially deservedly) was told: “So you knew about this and said nothing, did not warn. Yes, and there is a solution and did not share. Well, you bastard. " In short, therefore, the post is to be.

My fail2ban

I take the security of “my” servers extremely seriously. In addition, fail2ban is always very custom, I have monitoring there and a lot of things. It just really pisses me off that because of the stupid gray mass that allows you to take control of your botnet hardware, you have to kill a lot of time for protection (and its constant maintenance and monitoring in the future). By the way, in order to minimize this control, I am actively involved in the development of fail2ban and other security projects.

So, my last extended version of [sebres: ban-time-incr] allows you to bring out this annoying zoo once and for all (well, or until they adapt again). This feature was often discussed by the entire community, but somehow the hands did not reach. It lived for me in the form of separate scripts and some custom changes, until it took shape into a ready-made functional.

In short, the system, remembering the bad IP addresses, allows each time to dynamically (exponentially) increase the ban time (banTime) depending on the number of previous bans (banCount). At the same time, each time also reducing the number (maxRetry) of possible failed attempts (failure) to the next ban. This can clearly be seen in the following example:
It is clearly visible here how each next ban prolongs the blocking time from 15 minutes (0:15:00) the first time, up to 5-plus days (5 days, 8:04:55) after the tenth block. I have an IP in my database that already has a “term” - from several months to a permanent ban.

Below you can see how the new functionality affected the decision to ban IP XXX.XXX.XX.XXX. In the example, the maxRetry parameter is set to 5. So we see that until the IP was recognized as bad it was banned for the first time after 5 attempts, the second time, already as bad - after 3 (each attempt was counted as 2), the third etc. - after 2 (the attempt goes for 3) and is banned for the fourth time immediately after the first attempt (counts immediately for 5):

Without this logic of counting failure, smart botnets have learned to fine-tune their work so as not to fall into the ban. When I finished this logic and rolled it into production, in a matter of days I got rid of practically all the evil that I used to see for years in my logs. For example, now the average normal daily gain of my auth.log is somewhere around 20-50 lines, before on some servers it was hundreds and thousands of times more.

So far this is a development branch, lies a pull request, the release is planned so far in version 0.9.2.
If you are interested, read more about the implementation and the history of the solution here - Ban time incr by sebres · Pull Request # 716 · fail2ban / fail2ban .

However, while this version will be updated on your server, a lot of time will pass - until the release is released in mainline, while it will be taken to distributions ... The story is long, for example, the same debian still uses 0.8.x - this is why the article. So shake hands, set ... profit.

Take this version here fail2ban-ban-time-incr.zip (sebres master branch).
Port for debian-s: ban-time-incr-debian.zip (merged master debian branch, and although not the main line - I will try to keep the branch up-to-date if possible).

Installing it is quite simple. If you already had fail2ban installed from the distribution kit, save the old "fail2ban.local" and "jail.local" from "/ etc / fail2ban /" (Well, the old "fail2ban.conf" and "jail.conf" are better ) I would just in case (due to possible personal changes in filter and action) save somewhere the whole directory "/ etc / fail2ban /".

Next, we demolish the old fail2ban distribution, for example:

sudo service fail2ban stop
sudo apt-get remove fail2ban

Installation itself:

cd /tmp
unzip ~/downloads/fail2ban-ban-time-incr.zip
cd fail2ban-ban-time-incr/
sudo python setup.py install

[UPD] Sometimes, on some distributions, for manual installation, for some reason the service is not installed (for example, there is no file /etc/init.d/fail2ban) - and accordingly, through the service it does not (auto) start, only through fail2ban-client start.[/ UPD]

Now, to make the new functionality work, you need to add the option:. In your jail.local in [default] (or for each specific jail) bantime.increment = true. An example and description can still be found in " jail.conf ".
Some here are brief:

• bantime.rndtime- maximum time, used to add to banTimerandom time, to prevent smart botnets from calculating the exact time when IP is unblocked again. Examplebantime.rndtime = 10m
• bantime.factor- coefficient for calculating the growth exponent for the formula bantime.formulaor factors bantime.multipliers, the default value of the coefficient is 1, which corresponds to an increase in the ban time by 1, 2, 4, 8, 16 ... By increasing this parameter for some jail, you can increase the blocking time more aggressively.
• bantime.formula- it is used by default to calculate the next value of the ban time, the default value: bantime.formula = ban.Time * (1<<(ban.Count if ban.Count<20 else 20)) * banFactor
  The same growth of the ban time will be achieved using factors bantime.multipliersequal to 1, 2, 4, 8, 16, 32 ...
  An example of a more aggressive formula for the factor “1” and has the same growth values ​​only for a factor equal to "2.0 / 2.885385":
  bantime.formula = ban.Time * math.exp(float(ban.Count+1)*banFactor)/math.exp(1*banFactor)
  
• bantime.multipliers- the parameter can be used instead of the formula discretely to calculate the next value of the prohibition time. The value of the multiplier is equal to "-1" (it can only be at the end of the list) and puts the address in a permanent ban (before manual unlocking).
  Example 1: bantime.multipliers = 1 2 4 8 16 32 64- increases the ban time by 1, 2, 4, ... and if the last ban count was greater than the last multiplier index, then the last factor will always be used (64 in the example), which with a factor of “1” and the original ban time ( 10 minutes) - corresponds to 10.6 hours.
  Example 2: bantime.multipliers = 1 5 30 60 300 720 1440 2880- can be used for a small initial ban time (bantime = 60) - because the increase becomes more aggressive, we have a bantime equal to: 1 min, 5 min, 30 min, 1 hour, 5 hours, 12 hours, 1 day, 2 days, respectively.

If during the samples or in the production some (good) IP accidentally flew into the ban repeatedly (and became accordingly bad), it will be forgotten (will become “white” again) after three times of the last ban and dbpurgeage (located in fail2ban.local) , or if you remove the ban from his hands using:

fail2ban-client set $JAIL unbanip $IP

I use fail2ban-regex for testing the regulars, and for the performance test something like:

logger -t 'test:auth' -i -p auth.info "pam_unix(test:auth): authentication failure; logname= uid=0 euid=0 tty=test ruser=admin rhost=1.2.3.4"

Do not forget about the start:

sudo service fail2ban start

That's all, now I hope your server has become a little more secure. Well, do not be lazy and still look at the logs (trust, but check).

PS Standard postscript: Fail2Ban is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY. In short - use it on your health, but at your own peril and risk ...
And may your servers be safe.

PPS Yes, I almost forgot, I’ve planned to finish something here, something is already ready, you just need to draw it up normally and put it out, and so on - the poll "What do you think should be (before) done first."