June 24, 2016 at 12:25Web services for checking sites for viruses
Sooner or later, a web developer, webmaster, or any other specialist who maintains the site may encounter security problems: the resource is subject to search engine sanctions or starts to be blocked by antivirus software, they can send a notification about hosting malware detection from the hosting, and visitors begin to complain about pop-up ads or redirects to “left” sites.
At this point, the task of finding the source of the problem, that is, diagnosing the site for security problems, arises. With a competent approach, diagnosis consists of two stages:
Suppose you have already checked the files on the hosting with specialized scanners and cleaned the hosting account from “malware” (or nothing suspicious was found on it), but the search engine still swears on the virus code or the mobile redirect is still active on the site. What to do in this case? Web scanners come to the rescue, which perform dynamic and static analysis of site pages for malicious code.
Static analysis of pages is the search for malicious inserts (mainly javascript), spam links and spam content, phishing pages and other static elements on the page being checked and in the connected files. Detection of such fragments is performed on the basis of a signature database or some set of regular expressions. If the malicious code is constantly present on the page or in the downloaded files, and is also known to the web scanner (that is, it has been added to the signature database), then the web scanner will detect it. But this does not always happen. For example, malicious code can be downloaded from another resource or perform some unauthorized actions under certain conditions:
A few such examples:
If it is not known in advance which code provokes these unauthorized actions, then it is extremely difficult to detect it by static analysis. Fortunately, there is a dynamic analysis or sometimes it is also called “behavioral”. If the web scanner is smart, it will not only analyze the source code of the page or files, but also try to perform some operations, emulating the actions of a real visitor. After each action or under certain conditions, the scanner robot analyzes the changes and accumulates data for the final report: it loads the page in several browsers (and not just from different User-Agents, but with different values of the navigator object in javascript, different document.referer and etc.), speeds up the internal timer, catches redirects to external resources, tracks what is passed to eval (), document.write (), etc. An advanced web scanner will always check the page code and objects on it both before the start of all scripts execution (immediately after the page loads), and after some time, since modern malicious programs dynamically add or hide objects in javascript, as well as perform background downloads inside dynamic frames. For example, the code of an infected widget can load a script in 3 seconds or by mouse movement, which will paste into a javascript page with a redirect to download a dangerous .apk file. Naturally, no static analysis (except knowing in advance that the widget is dangerous) or a file search will not reveal this. since modern “malware” dynamically add or hide objects in javascript, and also perform background downloads inside dynamic frames. For example, the code of an infected widget can load a script in 3 seconds or by mouse movement, which will paste into a javascript page with a redirect to download a dangerous .apk file. Naturally, no static analysis (except knowing in advance that the widget is dangerous) or a file search will not reveal this. since modern “malware” dynamically add or hide objects in javascript, as well as perform background loads inside dynamic frames. For example, the code of an infected widget can load a script in 3 seconds or by mouse movement, which will paste into a javascript page with a redirect to download a dangerous .apk file. Naturally, no static analysis (except knowing in advance that the widget is dangerous) or a file search will not reveal this.
And now, with an understanding of the requirements for site diagnostics and web crawlers, let's try to find those that are really effective. Unfortunately, what is presented on the first page of the search engine for the query “check the site for viruses online” is no good at once. This is either “crafts”, which in the best case can perform static page analysis (for example, find an IFRAME, which may not be dangerous), or third-party API aggregators that check site URLs using the Google Safe Browsing API, Yandex Safe Browing API or VirusTotal API
If you check the site with a desktop antivirus, then the analysis will most likely also be static: the antivirus skillfully blocks downloads from infected sites known to it, but you should not expect some deep dynamic analysis of the site’s pages (although some antiviruses do detect signatures in files and on the page )
As a result, after checking two dozen well-known services, I would like to dwell on the following.
Searches for malicious code on pages using signatureless analysis. That is, it has a certain heuristic and performs dynamic page analysis, which allows it to detect 0-day threats. Of the nice features, it is worth noting the ability to check several pages of a site at once, since checking one at a time is not always effective.
It detects threats related to downloading or size of trojans, virus-infected executable files. It focuses on Western sites with their characteristic infections, but often helps out when checking infected Runet sites. Since the service is free, there is a queue for processing tasks, so you have to wait a bit.
Performs dynamic and static site analysis. Behavioral analysis detects hidden redirects, static analysis looks for virus fragments on pages and in downloaded files, and the resources downloaded from infected domains are determined by the blacklist base. It follows internal links, therefore, in addition to the main URL, it checks several more adjacent pages of the site. A nice addition is checking the site on black lists of Yandex, Google and VirusTotal. It is mainly focused on the malware that live in Runet. Since the service is free, the verification limit is 3 requests from one IP per day.
Searches for viral code by signature and heuristic. Sends requests to multiple URLs on a site with different User Agent / Referer. Detects spam links, doorway pages, dangerous scripts. In addition, it can check the current versions of CMS and the web server. There are no restrictions on the number of checks. From a small minus, it was found that the list of checked sites with the results is indexed by search engines, that is, you can see which site was infected and what it was infected with (now there are about 90,000 pages in the search index), but this does not detract from the effectiveness of the scanner.
Another western web site scanner. It can be a little frightening with its ascetic interface from the 90s, but, nevertheless, it allows you to perform a full static analysis of the site and the files connected to the page. When scanning, the user can set the parameters User Agent, referer, and page verification parameters. In the settings there is a page check from the Google cache. No limits on site verification were found.
And finally, familiar to many VirusTotal. It is not a fully web scanner, but it is also recommended to use it for diagnostics, since it is an aggregator of several dozens of antiviruses and antivirus services.
***
The mentioned web scanners can be added to bookmarks so that if necessary diagnostics can be carried out immediately with effective tools and not waste time on paid or inefficient services.
At this point, the task of finding the source of the problem, that is, diagnosing the site for security problems, arises. With a competent approach, diagnosis consists of two stages:
- checking files and databases on the hosting for server-side malicious scripts and injections,
- checking site pages for virus code, hidden redirects and other problems that, at times, cannot be detected by a static file scanner.
Suppose you have already checked the files on the hosting with specialized scanners and cleaned the hosting account from “malware” (or nothing suspicious was found on it), but the search engine still swears on the virus code or the mobile redirect is still active on the site. What to do in this case? Web scanners come to the rescue, which perform dynamic and static analysis of site pages for malicious code.
Bit of theory
Static analysis of pages is the search for malicious inserts (mainly javascript), spam links and spam content, phishing pages and other static elements on the page being checked and in the connected files. Detection of such fragments is performed on the basis of a signature database or some set of regular expressions. If the malicious code is constantly present on the page or in the downloaded files, and is also known to the web scanner (that is, it has been added to the signature database), then the web scanner will detect it. But this does not always happen. For example, malicious code can be downloaded from another resource or perform some unauthorized actions under certain conditions:
- upon completion of loading the page, javascript is added to it, which performs a drive-by download attack
- the user leaves the page, at this moment the code is loaded and popunder opens with the content “for adults”
- the site visitor is on the page for a few seconds and only after that he is redirected to a paid subscription for SMS
- etc.
A few such examples:
If it is not known in advance which code provokes these unauthorized actions, then it is extremely difficult to detect it by static analysis. Fortunately, there is a dynamic analysis or sometimes it is also called “behavioral”. If the web scanner is smart, it will not only analyze the source code of the page or files, but also try to perform some operations, emulating the actions of a real visitor. After each action or under certain conditions, the scanner robot analyzes the changes and accumulates data for the final report: it loads the page in several browsers (and not just from different User-Agents, but with different values of the navigator object in javascript, different document.referer and etc.), speeds up the internal timer, catches redirects to external resources, tracks what is passed to eval (), document.write (), etc. An advanced web scanner will always check the page code and objects on it both before the start of all scripts execution (immediately after the page loads), and after some time, since modern malicious programs dynamically add or hide objects in javascript, as well as perform background downloads inside dynamic frames. For example, the code of an infected widget can load a script in 3 seconds or by mouse movement, which will paste into a javascript page with a redirect to download a dangerous .apk file. Naturally, no static analysis (except knowing in advance that the widget is dangerous) or a file search will not reveal this. since modern “malware” dynamically add or hide objects in javascript, and also perform background downloads inside dynamic frames. For example, the code of an infected widget can load a script in 3 seconds or by mouse movement, which will paste into a javascript page with a redirect to download a dangerous .apk file. Naturally, no static analysis (except knowing in advance that the widget is dangerous) or a file search will not reveal this. since modern “malware” dynamically add or hide objects in javascript, as well as perform background loads inside dynamic frames. For example, the code of an infected widget can load a script in 3 seconds or by mouse movement, which will paste into a javascript page with a redirect to download a dangerous .apk file. Naturally, no static analysis (except knowing in advance that the widget is dangerous) or a file search will not reveal this.
And now, with an understanding of the requirements for site diagnostics and web crawlers, let's try to find those that are really effective. Unfortunately, what is presented on the first page of the search engine for the query “check the site for viruses online” is no good at once. This is either “crafts”, which in the best case can perform static page analysis (for example, find an IFRAME, which may not be dangerous), or third-party API aggregators that check site URLs using the Google Safe Browsing API, Yandex Safe Browing API or VirusTotal API
If you check the site with a desktop antivirus, then the analysis will most likely also be static: the antivirus skillfully blocks downloads from infected sites known to it, but you should not expect some deep dynamic analysis of the site’s pages (although some antiviruses do detect signatures in files and on the page )
As a result, after checking two dozen well-known services, I would like to dwell on the following.
QUTTERA Web Scanner
Searches for malicious code on pages using signatureless analysis. That is, it has a certain heuristic and performs dynamic page analysis, which allows it to detect 0-day threats. Of the nice features, it is worth noting the ability to check several pages of a site at once, since checking one at a time is not always effective.
It detects threats related to downloading or size of trojans, virus-infected executable files. It focuses on Western sites with their characteristic infections, but often helps out when checking infected Runet sites. Since the service is free, there is a queue for processing tasks, so you have to wait a bit.
Web scanner ReScan.pro
Performs dynamic and static site analysis. Behavioral analysis detects hidden redirects, static analysis looks for virus fragments on pages and in downloaded files, and the resources downloaded from infected domains are determined by the blacklist base. It follows internal links, therefore, in addition to the main URL, it checks several more adjacent pages of the site. A nice addition is checking the site on black lists of Yandex, Google and VirusTotal. It is mainly focused on the malware that live in Runet. Since the service is free, the verification limit is 3 requests from one IP per day.
Sucuri Web Scanner
Searches for viral code by signature and heuristic. Sends requests to multiple URLs on a site with different User Agent / Referer. Detects spam links, doorway pages, dangerous scripts. In addition, it can check the current versions of CMS and the web server. There are no restrictions on the number of checks. From a small minus, it was found that the list of checked sites with the results is indexed by search engines, that is, you can see which site was infected and what it was infected with (now there are about 90,000 pages in the search index), but this does not detract from the effectiveness of the scanner.
Redleg's file viewer
Another western web site scanner. It can be a little frightening with its ascetic interface from the 90s, but, nevertheless, it allows you to perform a full static analysis of the site and the files connected to the page. When scanning, the user can set the parameters User Agent, referer, and page verification parameters. In the settings there is a page check from the Google cache. No limits on site verification were found.
VirusTotal
And finally, familiar to many VirusTotal. It is not a fully web scanner, but it is also recommended to use it for diagnostics, since it is an aggregator of several dozens of antiviruses and antivirus services.
***
The mentioned web scanners can be added to bookmarks so that if necessary diagnostics can be carried out immediately with effective tools and not waste time on paid or inefficient services.
