Patch Management. Testing monthly software updates

The material in this article is based on the experience of installing more than 5,000 updates for Microsoft and Adobe products.

Patch Management is the process of managing software updates (software), without which at least one modern company that thinks about the security of its IT infrastructure is unlikely to do.
Updates or patches is an additional software tool that is used to fix detected defects in the software or change its functionality.
There are 2 types of updates :

1. for operating systems and server software that are used to maintain an appropriate level of security and eliminate security holes;
2. for application software (for example, Microsoft Office, Adobe Acrobat, or client parts of business applications) that are necessary to solve problems with frequently used or important libraries and other parts of the source code.

Among other things, the importance of updates and their timely installation to maintain an appropriate level of information security is obvious. However, there have been notorious cases in history when the “patches” of vulnerabilities in one application caused critical malfunctions in other applications. Here are a few examples:

1. Microsoft's monthly update caused VMWare to crash (February 8, 2011);
2. The update for Internet Explorer, released by Microsoft, contained two vulnerabilities at once : a browser malfunction, as well as a buffer overflow that allows a hacker to run code on a vulnerable machine with browser user privileges (August 8, 2006);
3. Microsoft's December updates were designed to fix a serious security flaw in the use of the OpenType font. The update affected PowerPoint, Quark Xpress and Coreldraw users. The released updates did not allow programs to recognize OpenType font characters larger than 15 pixels (December 12, 2012);
4. Installing Microsoft updates for kernel mode drivers led to the appearance of a blue screen of death (BSOD) on users' computers in August 2014;
5. An update for the PowerPoint 2013 office application made it impossible to launch the application (February 10, 2015).

Such unpleasant consequences of applying updates led to business downtime and loss of money, and this risk has not gone away. However, troubles could have been avoided by carefully testing the updates by the development company before deploying them.

Microsoft has repeatedly come across criticism of the lack of testing of monthly security updates. In response to this, representatives of the corporation made it clear that they plan to switch to a new testing scheme by transferring the service for testing updates to “external” users, that is, non-Microsoft employees.
And, indeed, if you look at this problem through the eyes of the development company, it becomes clear that testing the updates on the developer's side is a rather time-consuming process. Each update should be tested in conditions as close as possible to those that are operated on workstations and servers of the consumers of the product. For example, Adobe spends up to 6,000 man-hours per month testing updates .

But as life shows, testing by a developer company will never be enough for 100% of users in the world. Accordingly, the risk that the next software update will stop the company's business processes remains. On the other hand, it’s also impossible not to update, as you can be vulnerable to hackers who can harm the company.

As a way to significantly reduce these risks, large companies choose to regularly install updates in batches (releases) with mandatory testing of updates before they are deployed throughout the company.

Update management methods

Method patch management is a combination of approaches to testingupdates and the approach to deploying updates with updates. We will talk about them further.

The two most common approaches to testing updates are:

1. testing on local virtual machines;
2. testing in a full test environment.

Modern technologies allow testing of updates using virtualization without the use of extra pieces of equipment. The most commonly used and available products for creating virtual machines and networks: VMware or Oracle VirtualBox. The benefits of virtualization are that:

• creating the virtual machine itself is not time consuming;
• there may be several virtual machines on one physical platform;
• each virtual machine has its own virtual hardware components: memory, processor, hard drive, network adapters;
• and most importantly, it is possible to take a “snapshot” of the current state of the system and the contents of the disks with one click of the mouse, and then return to the initial state within a very short period of time, which can be very useful if an update is detected, the installation of which causes critical malfunctions in the system .

As a rule, virtualization technology is used for small networks with the number of workstations no more than 45-70.
Also, the change management process can be simplified by using just a couple of corporate computers or so-called test clients. After installing the necessary updates on such clients and testing the system after installation, you can see the consequences of the changes and see the result of applying the update, while having minimal impact on the IT infrastructure. Testing using a full test environment is applicable for large industrial networks and guarantees high test purity. When testing in a full test environment, the same approaches to installing updates and tools for managing updates are used as in the industrial environment. And now we will go over to them and at the same time consider approaches to deploying releases with updates.

Update Management Tools

As a rule, all widely used software solutions for remote management of IT infrastructure provide the ability to manage updates, for example:

1. System Center Configuration Manager (SCCM) - Microsoft software product;
2. Unicenter Software Delivery - a software product from Computer Associates;
3. OpenView is a Hewlett-Packard software product.

The leader in demand and use in large industrial networks is SCCM. Therefore, we give the details of the implementation of update management using its example.

The principle of managing updates using Configuration Manager is quite simple. All SCCM managed systems are consolidated into sites. Sites contain:

• site servers;
• site systems that perform specific roles in infrastructure management;
• Actually managed clients.

Each of the site servers must have access to the Microsoft SQL Server database.
The update server uses Windows Server Update Services (WSUS). It synchronizes with the Microsoft website, downloading updates that can be distributed within the corporate LAN. This saves the company's external traffic and allows faster installation of bug fixes and vulnerabilities in Windows operating systems at workstations, as well as centralized management of server and workstation updates.



It should be noted here that the WSUS server is only for updates released by Microsoft.
However, there is another update server - System Center Updates Publisher (SCUP)from Microsoft, which allows you to manage third-party software updates and then import it to the WSUS server, followed by deployment to clients using SCCM.

Stages of the Patch Management



process The update management process consists of several stages:

1. Preparing test clients
   An image of the operating system, including applications, as well as previously tested updates approved, is rolled onto the machine for testing updates. The next time the operating system boots up, the data of the approved updates is automatically installed.
2. Creating update sheets
   After the test environment is ready to install new updates, the creation of update sheets, or, as it is called in SCCM, patch lists, begins.
   The patch list includes updates released this month that fit the definition of "required." The need for an update for the client is determined by SCCM itself. The logic is simple: if a Visio application is installed on the client and an update for Visio is released this month, then a similar patch will be “required” for this client.
   As for updates for operating systems and servers, here the need is determined depending on their capacity and the Windows Service Pack.
   After the patch list is created, a new package is formed, which includes all the selected updates, and the package is assigned to the collection into which the test clients are added.
3. Deployment in a test environment (LAB Deployment)
   When a package is added to a collection, SCCM receives information that a new task has been assigned to clients of this collection, synchronizes with the SCCM client on the test machine, and deployment begins. The procedure takes only a few minutes, after which a corresponding icon appears in the notification area of ​​the taskbar. After double-clicking on the icon, Configuration Manager opens, where after selecting updates, installation begins. Typically, the installation ends with a request to reboot the system. After it, testing begins with the task: to find a patch that breaks the system / system components, or to make sure that it is absent.
   At its core, logging in after a reboot is already one of the verification points. It is important that there are no warnings / errors when entering the system. Further testing is carried out on a specific test matrix. The tests included in the test matrix depend mainly on the software of the IT infrastructure under test. More details about the tests will be described in the section - "Testing stages" below.
4. Deployment on pilot users (PRE Deployment stage)
   At the PRE Deployment stage, a list of tested updates is prepared and sent using SCCM to the clients of pilot users. The principle is the same: the update package is added to the collection, which includes only pilot clients, SCCM receives information that a new task has been assigned for the clients of this collection and deployment is starting.
   As a rule, as pilot clients, those users who are
   well versed in the software for which they are responsible are selected and check whether the application works normally after installing the updates. The number of users who participate in the pilot usually varies depending on the number of network workstations.
   The PRE Deployment stage lasts for 4 days, after which the pilot users give their feedback. In the event of software conflicts with installed updates, the latter are excluded from the list. Thus, the final list of updates is formed, which will be installed on all workstations of the industrial network.
5. Deployment of tested updates in a production information environment (PRO Deployment)
   Occurs at the PRO Deployment stage, using SCCM, on the same principle as for test and pilot clients. In this case, the start time of the deployment and the end date are set.

Testing steps

As mentioned earlier, testing is based on the test matrix. Below we will describe the basic checks that are usually included in such matrices.

Testing the installation . Installing updates to the test client is initiated by SCCM. Typically, after installing updates, the system requires a reboot. Accordingly, the first check: reboot the system, log in after rebooting under a test account, make sure there are no error messages / warnings of the system.
The next step is to check the PATH system variable . We make sure that PATH does not contain characters and signs that can lead to problems.
The third step is to check the system event log (Event Viewer).Information about installing updates is written to the log under ID 19. The check allows you to verify that all updates are installed successfully and that there are no errors.
Next in line is checking office applications . Office verification, as a rule, intersects all the test matrices of the Patch Management process.

Simple examples of Office checks:

1. Launch Word, Excel, Power Point, Publisher, Access.
2. Verify that applications open without errors.
3. Make changes to the document, make sure that upon closing the issue of saving appears and the document is saved by default to the Documents folder (unless another default repository is provided (by default)).

This sample Office check is basic and general. The matrix may include more complex detailed checks. For example, in addition to opening MS Word and saving the document, you need to check that the "Spelling and Grammar" checkbox is checked in the "Spelling and Grammar" tab, etc.



One of the most common checks is also checking Adobe applications that are being updated. The minimum check for Adobe Flash Player is to check the version of this software on the site. If the update is for Adobe Reader, then the version is checked and the health of the application is checked.
Checking browser performancealso goes to the base list. Patches should not make changes to proxy and ActiveX settings, and this is also checked during testing.
In addition, it is worth paying attention to the general effect of updates on the system. At the stage of preparing the test client, which already contains previously tested updates, the tests described above are carried out, and the results of such testing are taken as a standard.
Any deviation of the system from the state preceding the newly installed patches is analyzed and, in some cases, a decision can be made to exclude patches leading to undesirable changes in the system.

It is worth noting that to save useful time, some checks can be automated using VBScript, for example:

1. creating office application objects, opening program files, making changes, saving;
2. launch a browser and go to web pages;
3. uploading Event Viewer logs to the Excel table.

Summing up, we want to emphasize once again that the streamlined update management process will allow us to verify in advance the success of the application of updates and maintain the reliability and performance of the infrastructure with the constant installation of updates that provide the necessary level of information security.

Posted by: Alisa_Khaliullina