Express analysis of suspicious activity in the web server log

On most modern hosting sites, in addition to FTP access to the file system, SSH access is also provided (by default or by request for technical support). The webmaster’s ability to work with site files in the terminal (in command line mode) via SSH saves him a lot of time. An operation that can take tens of minutes via FTP is done through the command line in a couple of seconds. In addition, there are many operations that can only be done over SSH in command line mode.

The webmaster does not have to learn all the tools of the Unix operating system, for a start it is enough to get acquainted with the basic commands, and add some useful tricks when working with the command line via SSH to quickly search for files, change their attributes, copy, delete and perform operations with text data.

I will skip the description of the protocol and the process of connecting to the hosting account via SSH, on the network you can find many video tutorials and articles on this topic, I can only say that to connect you will need the Putty program (Windows OS) / Terminal (Mac OS X) or similar, and access to hosting via SSH: host, port, login and password (often they are the same name and password as access to cPanel, ISPManager or hosting control panel account).

So what can be useful on the command line? You can quickly search for a substring in a text file, sort, filter text data. For example, to analyze web server logs (logs), to identify suspicious requests to a site or to understand how a site was hacked.

Suppose you noticed suspicious activity on the site (it began to open slowly, access to the admin panel was lost, spam was sent from the site, etc.). The first thing to do in this case is to check the site files for malicious code with specialized scanners. But while the site is being scanned, you can conduct an express analysis of the web server logs using the find / grep commands to determine if there were any calls to some suspicious scripts, brute force attempts (password guessing), or hacker script calls. How to do it? About it below.

To analyze web server logs (logs) it is necessary that these logs be enabled and accessible in the user directory. If they are disabled by default, you must enable them in the hosting control panel and set, if there is such a setting, the maximum possible storage period (rotation). If there are no logs, but you need to perform an analysis over the past few days, you can try to request them from those hosting support. On most shared hosting sites, logs can be found in the logs directory, which is located one or two levels above the public_html (www) directory. So, we assume that there are logs on the hosting and the path to them is known.

We connect via SSH and go to the directory with the web server logs, which are usually stored on shared hosting for the last 5-7 days. If you list the files in the directory, most likely there will be access_log for today, as well as access_log.1.gz, access_log.2.gz, ... - these are archived logs for previous days.

You can start the analysis of the log with requests that were executed by the POST method:

grep ‘POST /’ access_log

or

cat access_log | grep ‘POST /’

The output result can be saved to a new text file for further analysis:

grep ‘POST /’ access_log > post_today.txt

How to do the same for gzip archived logs? There is a zcat command for this (similar to cat, but it prints the contents of the archived file).

zcat access_log.1.gz | grep ‘POST /’ > post_today.txt

For analysis of suspicious activity, it is advisable to use a sample of all available logs. Therefore, in the examples that follow, we will use the find command, which will search for all files, and then execute the corresponding command for each (for example, zcat).

How to identify hacking or searching for vulnerable scripts?
For example, you can find all calls to non-existent .php scripts in all available logs.

grep 'php HTTP.* 404' access_log
find . -name '*.gz' -exec zcat {} \; | grep 'php HTTP.* 404'

( instead of -exec, you can use xarg to call zcat. ) You

can also search for all unsuccessful calls to php scripts (which were denied access).

find . -name '*.gz' -exec zcat {} \; | grep 'php HTTP.* 403'

Here we are looking for queries in which the php extension and status

403 are encountered. Next, we will look at all available logs for the number of successful calls to scripts, sort them by the number of calls and display the TOP-50 of the most popular ones. We will make a selection in three steps: first we will search for access_log, then for all access_log. *. Gz, output the results to a file, and then use it to sort.

find . -name '*.gz' -exec zcat {} \; | grep 'php HTTP.* 200' > php.txt
grep 'php HTTP.* 200' access_log >> php.txt
cut -d '"' -f2 php.txt | cut -d ' ' -f2 | cut -d '?' -f1 | sort | uniq -c | sort -n | tail -50

For a site on Wordpress, the result may look like this:
( examples for Wordpress are provided solely for illustration, in fact, the described approach and commands are not limited to this CMS. The given commands can be used to analyze the logs of the web server of sites working on any php frameworks and control systems ( CMS), and also just in php scripts.
)

   1 /wp-admin/edit.php
   1 /wp-admin/index.php
   1 /wp-admin/update-core.php
   1 /wp-admin/upload.php
   2 /wp-admin/users.php
   3 /wp-admin/plugins.php
   4 /wp-includes/x3dhbbjdu.php
   4 /wp-admin/profile.php
   4 /wp-admin/widgets.php
  38 /wp-admin/async-upload.php
  58 /wp-admin/post-new.php
1635 /wp-admin/admin-ajax.php
6732 /xmlrpc.php
14652 /wp-login.php

The result shows that there were more than 14,000 hits on the wp-login.php file, which is not normal. Apparently there was (or is still) a brute force attack on the site with an attempt to gain access to the admin panel.

A large number of calls to xmlrpc.php may also indicate suspicious activity. For example, other Wordpress sites may attack (DDOS) through the site with XML RPC Pingback Vulnerability.

Successful calls to /wp-includes/x3dhbbjdu.php also seem suspicious on the list, since such a file cannot be in standard Wordpress. In the analysis, he turned out to be a hacker shell.

Thus, in just a few seconds, you can get statistics on calls to scripts, identify anomalies, and even find part of hacker scripts without scanning the site.

Now let's see if there were any attempts to hack the site. For example, searching for vulnerable scripts or accessing hacker shells. Find all requests to files with the extension .php with the status 404 Not Found:

find . -name '*.gz' -exec zcat {} \; | grep 'php HTTP.* 404' > php_404.txt
grep 'php HTTP.* 404' access_log >> php_404.txt
cut -d '"' -f2 php_404.txt | cut -d ' ' -f2 | cut -d '?' -f1 | sort | uniq -c | sort -n | tail -50

This time, the result may be like this:

   1 /info.php
   1 /license.php
   1 /media/market.php
   1 /setup.php
   1 /shell.php
   1 /wp-admin/license.php
   1 /wp-content/218.php
   1 /wp-content/lib.php
   1 /wp-content/plugins/dzs-videogallery/ajax.php
   1 /wp-content/plugins/formcraft/file-upload/server/php/upload.php
   1 /wp-content/plugins/inboundio-marketing/admin/partials/csv_uploader.php
   1 /wp-content/plugins/reflex-gallery/admin/scripts/FileUploader/php.php
   1 /wp-content/plugins/revslider/temp/update_extract/revslider/configs.php
   1 /wp-content/plugins/ultimate-product-catalogue/product-sheets/wp-links-ompt.php
   1 /wp-content/plugins/wp-symposium/server/php/fjlCFrorWUFEWB.php
   1 /wp-content/plugins/wpshop/includes/ajax.php
   1 /wp-content/setup.php
   1 /wp-content/src.php
   1 /wp-content/themes/NativeChurch/download/download.php
   1 /wp-content/topnews/license.php
   1 /wp-content/uploads/license.php
   1 /wp-content/uploads/shwso.php
   1 /wp-content/uploads/wp-admin-cache.php
   1 /wp-content/uploads/wp-cache.php
   1 /wp-content/uploads/wp-cmd.php
   1 /wp-content/uploads/wp_config.php
   1 /wp-content/wp-admin.php
   1 /wp-update.php
   1 /wso2.php
   2 /wp-content/plugins/dzs-zoomsounds/ajax.php
   2 /wp-content/plugins/hello.php
   2 /wp-content/plugins/simple-ads-manager/sam-ajax-admin.php
   3 /wp-content/plugins/dzs-zoomsounds/admin/upload.php
   4 /2010/wp-login.php
   4 /2011/wp-login.php
   4 /2012/wp-login.php
   4 /wp-content/plugins/wp-symposium/server/php/index.php

As we see from the result, there were similar calls. Someone "for luck" tried to access the hacker shell in the directory of the allegedly vulnerable component Revolution Slider /wp-content/plugins/revslider/temp/update_extract/revslider/configs.php and the WSO Shell in the root of the site and a number of other hacker and vulnerable scripts. Fortunately, to no avail.
Using the same find / cat / zcat / grep, you can get a list of IP addresses from which these requests were executed, date and time of access. But this is of little practical use.

It is more useful to select all successful POST requests, as this often helps to find hacker scripts.

find . -name '*.gz' -exec zcat {} \; | grep 'POST /.* 200' > post.txt
grep 'POST /.* 200' access_log >> post.txt
cut -d '"' -f2 post.txt | cut -d ' ' -f2 | cut -d '?' -f1 | sort | uniq -c | sort -n | tail -50

The result may look like this:

  2 /contacts/
  3 /wp-includes/x3dhbbjdu.php
  7 /
   8 /wp-admin/admin.php
  38 /wp-admin/async-upload.php
 394 /wp-cron.php
1626 /wp-admin/admin-ajax.php
1680 /wp-login.php/
6731 /xmlrpc.php
9042 /wp-login.php

Here you can see many calls to wp-login.php and xmlrpc.php, as well as 3 successful POST requests to the script /wp-includes/x3dhbbjdu.php, which should not be in Wordpress, that is, most likely this is a hacker shell.

Sometimes it’s useful to see a selection of all 403 Forbidden requests made by the POST method.

find . -name '*.gz' -exec zcat {} \; | grep 'POST /.* 403' > post_403.txt
grep 'POST /.* 403' access_log >> post_403.txt
cut -d '"' -f2 post_403.txt | cut -d ' ' -f2 | cut -d '?' -f1 | sort | uniq -c | sort -n | tail -50

In my case, it looked like this. Not very much, although it could be an attempt to exploit XML RPC Pingback.

   8 /xmlrpc.php

Finally, you can select the TOP-50 popular site requests for today:

cut -d '"' -f2 access_log | cut -d ' ' -f2 | cut -d '?' -f1 | sort | uniq -c | sort -n | tail -50

We get:

   6 /wp-admin/images/wordpress-logo.svg
   6 /wp-admin/plugins.php
   7 /wp-admin/post-new.php
   8 /wp-admin/async-upload.php
   9 /sitemap.xml
  10 /wp-admin/users.php
  13 /feed/
  13 /wp-admin/
  20 /wp-admin/post.php
  22 /wp-admin/load-styles.php
  38 /favicon.ico
  52 /wp-admin/load-scripts.php
  58 /wp-cron.php
  71 /wp-admin/admin.php
 330 /wp-admin/admin-ajax.php
1198 /
2447 /wp-login.php

The statistics of calls to /wp-login.php in access_log confirms that a brute force attack is still ongoing on the site (someone is trying to find a password), so you should limit access to wp-admin by IP or server authentication, and if on the Wordpress website no user registration, then you can restrict access to wp-login.php.

Thus, without any specialized applications and additional tools, you can quickly analyze the web server logs, find suspicious requests and their parameters (IP address, User Agent, Referer, date / time). All you need for this is SSH connectivity and basic command line skills.