Comparative testing of 15 antivirus programs in the new C'T magazine

At Security @ Interop, an event that was completely undeservedly ignored by many security professionals, one could get the pilot (April) issue of C'T-Russia magazine , which I gladly did. Now he walks the hands of the whole company.

C'T found a very interesting and thoughtful comparative test of 15 antivirus programs. Hands finally reached the "processing" of this very "tasty" content. I will give here only an introductory part about the methodology and conclusions. This is what will be the very comprehensive analysis, on the basis of which it has recently become fashionable (and now it is possible) to scold or praise the products of different vendors.

I believe that for the team of the new computer technology magazine, conducting such a comprehensive test is a big application for leadership, because the tests are no worse than those conducted by Neil Rubenking from PC Magazine USA. Now there is something to read in Russia as well - do-it-yourself testing, described in human language and impartially filed, which together is the “trick” of C'T. It is strongly recommended that Sergey Ilyin be adopted from Anti-Malware.ru.

I also liked the words of the editor-in-chief about the audience and the mission of the magazine (quote):
We are not doing nonsense. We make the most professional computer magazine in Russia ... for those who need honest and truthful information.

Respect to Pavel Shoshin, author of the review, and chief editor Andrei Kokourov, who moved from CHIP. I didn’t have such confidence in their professionalism before, but as it turned out, everything was decided by the “easy reading” format in CHIP, which too many media and test labs have been pressing lately.

In general, I wrote the praises of the magazine, you can proceed to reading. :)

Big hunt

Trojans, spyware, bots - everything seems to be the same here, but, nevertheless, is different: every hour more than a thousand newcomers fill up the virus “zoo” of anti-virus laboratories. And judging by the results of testing 15 antivirus programs, an increasing majority of them are slipping through antivirus protection.

Along with the luminaries of the anti-virus industry, some exotic programs passed our testing: Avast Antivirus Pro, AVG Anti-Malware, Avira AntiVir Personal Edition Premium, BitDefender, CA Antivirus Plus, ClamWin, Dr.Web, F-Secure Antivirus, McAfee VirusScan, Microsoft OneCare , and Trend Micro Antivirus + Antispyware.

The main mechanism for detecting malware is still searching for files by known character strings. The quality of such signature recognition is determined by checking the set of ITW viruses (from the English In The Wild - viruses that pose a real danger to users at the time of testing). However, this approach did not become dominant, if only because it does not represent the most active class of malware at the moment - trojans. In addition, this list of viruses is not replenished so often, it is constantly becoming obsolete, and the list itself, consisting of approximately 1,400 copies, is too small. Therefore, ITW testing in this case was carried out only for the sake of completeness of the experiment, and its results were not taken into account when summing up the results.

The fact that Avast! and Kaspersky Anti-Virus did not perform very well, although when scanning on demand, all viruses from the ITW set were detected, and on the fly both packages missed one file, which indicates problems related to the quality of scanning. But ClamWin and Dr.Web made more significant mistakes, skipping 25 and 7 viruses, respectively.

In the case of our own testing, antivirus programs had to scan a virus database with more than a million worms, backdoors, bots, and trojans. In this case, only those malicious codes were used that were really active over the past six months; extinct dinosaurs from the time of DOS and Windows 95 were not taken into account.

When testing, it was desirable to get the highest recognition rate, but we understood that none of the scanning methods could detect absolutely all viruses. In particular, the Avira product approached the ideal with an impressive result - over 99% recognition. The result of 95%, necessary for a positive assessment, was received by three more products: Avast !, AVG and BitDefender. In addition to ClamWin and Dr.Web, which, as already mentioned, did not show themselves very well in this test, CA Antivirus looked weak with a result of 55% - obviously not enough for reliable protection.

Good results were obtained by identifying adware and spyware, the test base of which totaled 25,000 programs. If before, special anti-spyware utilities were needed to combat it, now developers include data in this signature category of malware. We were not mistaken, the term “malicious” is quite appropriate here: utilities that can extract secret information or make lists of users' interests and send them over the network deserve this title. By the way, the programs that coped well with the virus “zoo” proved to be excellent in the tests with the detection of spyware. CA Antivirus and Norton Antivirus failed in this test.

The positive point in the overall picture was that the developers increased the frequency of software updates, as well as the speed of response to new virus threats. Softwin and Kaspersky Anti-Virus are leading in this regard. Only users of CA Antivirus, McAfee and Microsoft are forced to wait for updated signatures on average more than 12 hours compared to the fastest competitor. But updates from Microsoft One Care were late even more than a day.

I see what you don't see
The reason that good results of signature recognition, as well as short intervals between updates is not enough to protect your computer reliably, can be shown with a simple calculation. Suppose a developer has time to create the necessary signatures, check them and send them to their clients within an hour after the appearance of a new Trojan program. However, at this time, malware is spreading through the bot network using about 10,000 active zombie computers. Each of them can send an e-mail within three seconds (that is, more than a thousand per hour). As a result, the owner of the bot network will manage to reach about 10 million of its victims before the first signature appears. Of course, some figures can be disputed, but the general principle is clear: in the event of an epidemic, signatures are always late.

To create programs that can detect previously unknown pests, developers began to use heuristic methods. They allow, in particular, to identify typical sequences of codes or to issue warnings if the program being tested tries to load a resident module into memory.

The antivirus software we are testing should have detected new viruses with the help of old signatures.

Behavioral Analyzer Testing

Behavioral analysis (proactive defense) differs from signature and heuristic methods, which serve only to recognize the malicious files themselves. The difference in testing begins with the selection of viruses that none of the tested packages can identify using signatures. At startup, the virus must fall into the same environment each time, and, for example, be able to download its components from the network, despite the fact that the corresponding servers could have ceased to exist at the time of testing.

During the test itself, the virus is given the opportunity to work and monitor its behavior, as well as how the anti-virus program behaves. If the protection offers to interrupt the work, then the testing stops. After that, they check whether the virus managed to infiltrate and cause damage, or if its attack was successfully repelled, and open files and registration keys were deleted.

In total, twelve viruses that are dangerous for Windows were tested (Spy.VB.QJ, Packer, DNS-Changer.OL, Rbot.BMR, Hmir.DK, Delf.FYR, IRCBot.CHR, Agent.CDM, RBot.XKW , PcClient.BAL, Pakes.AKT, Zlob.KF). When evaluating the program, it received one point if, as a result, the computer was not infected, that is, the virus was unable to install the components it executes. If the anti-virus software succeeded that the system was not infected at the next start, then the anti-virus received half a point.

The behavior recognition method is prone to malfunctions, especially if it is used only to issue messages about certain actions that occur during normal operation. Therefore, based on the installation and updating of ten standard programs such as ICQ, Winamp or Microsoft Office, we also checked the proportion of false positives.

The life-saving anchor
The fact that signature recognition methods do not provide reliable protection, and heuristic analysis does not have high reliability, has been known for several years. The solution to this problem is also known: when monitoring the system, the antivirus should also recognize previously unknown malware by its behavior. With increasing suspicious activity, he warns the user and offers to stop tasks that could harm the computer, or even cancel some previously performed operations. This feature, called the Proactive Guard, a behavioral analyzer, may also be called the Behavioural Blocker, is also found in the description of many software products.

Our testing showed rather disappointing results. Only F-Secure's F-Secure Deepguard showed itself well, which recognized all 12 “pests”, and in most cases it managed to prevent the computer from being infected. In addition to it, Kaspersky Anti-Virus and BitDefender were also able to detect malware by their behavior. However, they could not completely prevent the infection of the system.

Development companies McAfee, Norton, Microsoft, and Trend Micro are on the path of least resistance and offer to evaluate the behavior of suspicious programs to users themselves. When using anti-virus programs from these manufacturers, you will often come across messages that were often found at the dawn of software firewalls: “The XYZ program wants to perform some action - do you want to allow it?” The catch is this: such messages almost always refer to individual actions, such as adding an autostart item to the registry. Even if the user prohibits this action, the virus will not be neutralized yet; some of its components may still remain active. In addition, such warnings often appear when installing harmless programs. Well disguised as a video codec,

These results are also confirmed by our experiments with simple keyloggers created from existing code fragments that silently save data entered from the keyboard to the user. One of the keyloggers was added to the registry as an autoload item, the other was installed as a kernel driver, the third as a service, and the latter additionally established a connection with an IRC server. In this case, only the programs of F-Secure, BitDefender and Kaspersky Lab showed acceptable results. F-Secure recognized all keyloggers as applications with a high level of danger, which was reflected in the form of a rating. Kaspersky Anti-Virus reported the installation of suspicious drivers, noticed the creation of new files and their registration as an startup point and as a service.

McAfee and Trend Micro programs recorded only registry changes made by the first keylogger. With NOD32 and OneCare, heuristic analysis revealed only the first keylogger that entered data into the registry branch responsible for autoload. Other antiviruses failed to detect anything at all.

Peekaboo
Although more and more viruses are using rootkit methods to hide their presence in the system, only a few antivirus programs are sufficiently prepared for this. Only programs from F-Secure, Norton, and Panda, during testing, were able to detect and remove all active rootkits, although we used only widely known and accessible samples. The fact that AVG recognizes only inactive rootkits cannot be attributed to it in any way, even taking into account the fact that the program has a separate module for searching rootkits. Microsoft's ClamWin, Dr.Web, McAfee, and OneCare programs also have almost nothing to counter this type of threat.

If a rootkit attack is suspected, a boot emergency disk is of particular importance, thanks to which you can scan the system without loading it. However, many manufacturers still save on such drives. And if they don’t save money, they’ll obviously cheat: from the Symantec disk, as before, DOS is loaded with an antivirus scanner whose signatures are dated 2002, and Panda offers a Linux-based boot disk that provides access to NTFS disk partitions only in read mode.

Another method used to trick antivirus scanners is to use archive files that they cannot open. At the same time, the attackers change the access rights to the content, for example, in a ZIP archive, so that the anti-virus scanner cannot open it, and with the help of the utilities installed on the computer, this could be done. Thus, the virus may go unnoticed when checked at the mail gateway.

We tested the resistance of individual antiviruses to similar masking methods during testing using 28 specially prepared archive files in the .rar, .zip and .cab formats. They could be opened using standard programs, such as integrated in Windows ZIP, WinZIP, WinRar or 7zip. However, only F-Secure detected hidden viruses in all modified archives.

In a similar way, malicious code that uses security holes in browsers can be made invisible to antiviruses. It has long been known that Internet Explorer completely ignores some elements of the HTML page code. For example, an attacker can fill his exploit with zero bytes so that it differs from the samples in the antivirus database. The trick with the addition of zero bytes took place on all antiviruses, with the exception of BitDefender. Other programs issued warnings, but the exploits managed to confuse them.

Vista and antiviruses
All tests were conducted on computers running Windows Vista. This did not cause any difficulties, since most manufacturers have already adapted their programs for it.

Only a new rights model can create some problems due to the fact that an antivirus scanner must obtain the necessary rights for a full scan of the system using the User Account Control (UAC) function. So, the AVG and CA Anti-Virus programs must be run manually as an administrator in order to be able to scan absolutely all files. Otherwise, the anti-virus scanner will simply skip the folders of other users. With F-Secure antivirus, it’s even more difficult - the program ignores other people's folders, even with the necessary rights. Such a folder can only be scanned if you go to it and enable the action in the appeared User Account Control window. When using Avast and Dr.Web to perform special operations, the user must enter the administrator password when using a limited account.

In addition, we tested software products on computers running Windows XP. At the same time, we found almost no differences between the results of the antivirus in Vista and XP. However, it is necessary to take into account that when using specific functions, such as proactive protection and rootkit search, on computers with Windows XP in other cases, you can get other results.

Little by little.
To determine how much the anti-virus slows down the computer, we conducted two tests: first, the program had to check for viruses about 8000 files with a total volume of 741 MB. To evaluate the operation of the antivirus when scanning on the fly, we also copied files on the local hard drive, which in the absence of antivirus takes 47 seconds.

High speed showed NOD32; programs such as Avast !, AVG, Antivir, OneCare, Norton, Panda and TrendMicro also did not slow down the system very much. Significantly worse results were shown by F-Secure, which, with four anti-virus engines, slowed down significantly, as well as Kaspersky Anti-Virus, which significantly increased the latency when copying file folders.

Most antiviruses quietly monitor the work of mail programs and scan incoming and outgoing messages. The presence of this function should not be a decisive factor in anti-virus software, because even if it is absent, the program will still give a warning when you try to open or save the attached file. In addition, the antivirus will still be powerless when using an encrypted connection.

A serious problem may be the insufficiently high quality of the source code of antivirus programs. In 2007, virtually all well-known antivirus products were found to have critical security holes that cast doubt on the reliability of the entire antivirus software industry. For our test, we asked manufacturers a question regarding the measures they take to improve the security of software products. The few answers could not be fully appreciated, and it is also impossible to verify them for compliance with reality. Therefore, this issue is still open.

I do not plan to publish ratings for each product, also given in the article, but you can evaluate the summary table and the conclusions along the way.

Conclusion

NOD32 was the only one of the test participants who managed to detect more than two-thirds of new malware; in second place is BitDefender with a result of 41%, and the remaining antiviruses recognized only every third threat or even less. In this case, it is hardly possible to talk about reliable computer protection. Such low results can be explained by the fact that malware developers began to pay more attention to optimizing their creations. Of particular concern is the fact that during the testing, antiviruses failed to detect some viruses that they successfully found a year ago.
The solution to this problem can be the behavioral recognition of malicious programs, which, however, is currently well implemented only in the Anti-Virus 2008 program of F-Secure. In this situation, it is rather difficult to recommend this or that product, since no program has shown excellent results and, therefore, cannot guarantee the security of your computer. F-Secure antivirus has not proved to be the best when searching for spyware modules and is very demanding on system resources. A reasonable compromise is the BitDefender program, which is also not very fast, but at the same time it has shown good heuristic analysis results, recognized almost all rootkits and has good prospects in the field of behavioral analysis.
Users who focus on high performance will either prefer Avira, or turn a blind eye to the results of signature analysis, and will use NOD32, which, thanks to the high efficiency of heuristic analysis, can protect your computer from many unknown viruses. Avast and AVG also showed very good signature analysis results. They are followed by a number of products that have yet to get rid of various specific problems. The worst results were shown by CE Anti-Virus Plus, which failed almost all the tests, as well as ClamWin, which does not pretend to compete with other participants. Dr.Web and McAfee can also find a better alternative.
Novice users can advise Norton Antivirus, and if you come to terms with some of the shortcomings, then Windows OneCare. Taking into account the fact that commercial products did not show high results, some users will probably prefer to save money and use a free program. Please note that the free versions of Avira and AVG antiviruses do not have a spyware search function, which must be compensated by installing an additional antispyware program. Integrated with Vista, Windows Defender enhances spyware recognition only to Norton Antivirus. Other restrictions, such as the lack of an email verification feature, can be turned a blind eye if necessary. Avast, which is also free,







In particular, this review shows in a different way the separation of “pure” antiviruses than Anti-Malware.ru presents.
In my interpretation, judging by the summary table and the description of the qualitative / quantitative indicators in the tests, the following picture develops:
1 tier - BitDefender, Avira, F-Secure
2 tier - NOD32, AVG, Avast !, KAV 7, Trend Micro, Panda 2008
3 echelon - Norton AV 2008, McAfee AV 2008, Dr. Web 4.44, MS OneCare
4 echelon - CA, ClamWin

Well, what opinions will the public have?