How to test antivirus software
For the past five years, I have been helping to make testing system applications faster and cheaper. As a youth (being young and hot) I tried to just work faster with my hands and more attentively with my head, but gradually I came to understand that manual testing has a limit of effectiveness: for a department consisting of a manager and two testers, this is about 30 features per month. It would seem that there are no problems, because there is a well-developed way for merciless testing automation - just getting rid of regression will give you more than free time, just choose a system from what is on the market, we will introduce it and we will live happily ever after. And here life declares: "No." And the point is not that all autotest systems have a fatal flaw , or do not meet the simple list of requirements generated by my exuberant fantasy (a link to my requirements ), or they can’t display the unimaginable complexity of user cases of use - a simple analysis tells us that the set of user cases of any antivirus must include relatively simple steps:
- Deploy to a clean machine
- Checking the static characteristics of antivirus files
- Treatment and detection of various threats
- Uninstall
- Reinstall
- Update old version
- Installation compatibility with competitor antivirus solutions
- Self Defense Tests
I would especially like to note the compatibility point - for me it seemed normal that the antivirus is someone from the list of kaspersky, drweb, eset, symantec, etc, but the world did not stand still and the new generation of users just couldn't find for themselves - now we have to parse the page download.cnet.com/s/antivirus with popular downloads per week and test with what is relevant.
As you can see, all cases directly beg for automation.
But in testing system software, including antiviruses, there are a couple of problems:
- Work on hardware differs from work in vm; among test machines there can and should be both physical and virtual environments.
- The second problem of testing system software was noted back in 2013 by the talented researcher Y. M. Fridlyand and it sounds like “providing support for rebooting in the autotest system”.
Almost a month of generating various queries to search engines led me to the fact that there are no other relevant answers to the question “how to do self-tests with a reboot” besides the presentation to the diploma project of our partners . Oh gods, I thought, reading that pdf - dragging a whole TFS with all the accompanying Microsoft infrastructure for the sake of life after a reboot?
Why am I doing this?
And I began to study other solutions. Yes, I really watched docker, read about cen based on jenkins, tried and was disappointed - either the solution didn’t completely fit, or the number of related technologies needed to study gave in the future a uniquely modified unsupported solution (or supported by a staff of high-class specialists), I I changed projects and employers (for more and more famous ones of course), but everywhere I saw the same thing - testing system functionality (updating drivers, shutting down the OS, treating viruses, checking startup) was hands the fastest, most reliable and cheapest way.
And then I discovered python. Using this technology alone, it was possible to write a simple virtual machine manager (import pysphere), a web interface to it (import web2py), a testing agent-win32 service (import win32service) delivered to the virtual machine via an iso image, and the tests themselves, assembled in separate exe files through pyinstaller (so as not to suffer from pre-training systems, installing python etc). Conceptually, it all looks like this:
All this stuff is supported by 1 specialist with knowledge of 1 technology, the bus factor is minimal, which I am extremely happy about.
The test results are collected in the daily report page:
So in a simple way, we got rid of regression and smoke tests in about six months of developing our bike, only new functionality falls on manual tests, from which the wrinkles of the entire testing department are smoothed out, and the hair has become soft and silky, which is what I wish you.
Actually, after gaining a certain amount of new features, the product is given to the probe team - beta testers and through feedback in technical support the quality is brought to unprecedented heights.