Testing installers. Theory

Software installation testing is aimed at verifying successful installation and configuration, as well as updating or removing software.
At the moment, the most common installation of software using installers (special programs that themselves also require proper testing.

In real conditions, there may not be any installers. In this case, you will have to independently install the software using the documentation in the form of instructions or readme files , step by step, describing all the necessary actions and checks.

In distributed systems, where the application is deployed on an already running environment, a simple set of instructions may not be enough. To do this, often, a Deployment Plan is written that includes not only the steps to install the application, but also the roll-back steps to the previous version, in case of failure. The installation plan itself must also undergo a testing procedure to avoid problems during commissioning. This is especially true if the installation is performed on systems where every minute of downtime is a loss of reputation and a large amount of funds, for example: banks, financial companies or even banner networks. Therefore, testing installers can be called one of the most important tasks to ensure the quality of software.

It is such an integrated approach with writing plans, step-by-step verification of the installation and rollback of the installation, that can rightfully be called installation testing or Installation Testing.

Under the cut, there is a lot of text about what you should remember when testing.

Features of testing installers:

The installer is a “regular” program whose main functions are Installing (Installing), Updating and Removing (Uninstalling) software.
Everyone knows the folk wisdom: " They are greeted by clothes, but escorted by the mind ." The installation application is the very clothes that create the first impression of your product. That is why testing the installation is one of the most important tasks.

Being an ordinary program, the installer has a number of features, among which are the following:
● Deep interaction with the operating system and dependence on it (file system, registry, services and libraries).
● Compatibility of both native and third-party libraries, components or drivers, with different platforms.
● Ease of use: intuitive interface, navigation, messages and tips.
● Design and style of the installation application.
● Compatibility of user settings and documents in different versions of the application.
● And much more.

If these features have not charged you with a serious attitude to testing installation programs, I want to give a small list of risks that will show the significance of the correct operation of the installers:
● risk of loss of user data.
● risk of operating system crashes.
● risk of application inoperability.
● risk of incorrect application operation.

At the same time, like any program, some functional requirements are imposed on the installer. Combining them with a list of features, we get a more complete picture showing the amount of upcoming testing work. Further, based on the list of requirements, you will need to answer the questions: “What to test?”, And only then - “How to test?”.

With the modern abundance of personal computers, servers, and operating systems, there is a need to install the same software on different platforms. For this, installers must understand what and where they install depending on the environment.

What to test in the installation programs?

We will write in more detail, " What ?" it is necessary to check, to assess the correct operation of the installer:
● Installation (Installation).
● The correctness of the list of files in the installation package:
when choosing different types of installation or installation parameters, the list of files and their paths may also differ.
absence of unnecessary files (project files not included in the installation package should not get to the user's disk).
● Registration of the application in the OS.
● Register extensions for working with files:
for new extensions.
for existing extensions.
● Access rights of the user who installs the application:
rights to work with the system registry.
permissions to access files and folders, for example % Windir% \ system32 .
● The correctness of the installation wizard (Installation Wizard).
● Install multiple applications in one go.
● Installing the same application in different working directories of the same workstation.
● Update.
● The correctness of the file list, as well as the absence of unnecessary files:
checking the list of files with different installation parameters.
lack of extra files.
● Backward compatibility of created data:
safety and correct operation of data created before updating.
the ability to correctly work older versions of the application with data created in new versions.
● Update when the application is running.
● Aborting updates.
● Uninstall (Uninstall).
● Correct application deletion:
removal from the system registry of libraries and service records installed during the installation process.
Delete physical application files.
delete / restore previous file associations.
safety of files created during the work with the application.
removal when the application is running.
uninstall with restricted access to the application folder.
removal by the user without the appropriate rights.

How to test installations?

Getting the list of files should be done Before, and checking the files themselves - After installation.
The most correct is to try to get a list of files from the installation package installer. The phrase: “Take and make a list after installing the software, everything is right there” is a provocation and it is better not to give in to it.
If the program contains files signed by a certificate, for example from MS, then it is possible that temporary files may come across that are no longer needed for the application to work (* .tmp, etc.). Please note that the same installation package can install different sets of files under different operating systems. Therefore, testing should be done for each OS separately.

For almost any OS, it is important to register working libraries and service records. For example, in Windows, you will need to check the system registry to correctly record new data and register new / new applications.
If, when you open a document that your program is designed to work with, you will get a dialog to select a program to open files of this type or it will be opened by another application, then there will be an error registering the extension in the OS.

In a good way, the installation program should at the start check the user account and immediately correctly report any problems with rights. But the option is not ruled out that additional restrictions will be installed on any service folders. You can try to simulate the situation yourself when any of the folders is closed for writing, for example, " \ Program Files ", " \ Windows ", " % Windir% \ system32 ", and also check how the application will behave if it is impossible to write any of files on the right path. It is possible that without some files the functionality of the entire application will not be affected. In this case, it is enough to indicate a problem with copying the file (s) to the log, and DO NOT display an error message.

It is necessary to conduct a complete Testing of the Installation Wizard of the application.
Quite often there are situations when an application, besides itself, puts another product. It can be either a separate third-party application or a demonstration of your own product. During the installation process, usually this will not be another set of steps of the installation wizard, but just a small message that such and such a product is being installed. During testing, it will be necessary to pay special attention to the fact that we have a consistent installation of several products. If the first of them requires a reboot after its installation, the second may not be installed correctly, especially if drivers are placed deep into the system both there and there. Of interest are errors that occur precisely at the junction of the installation of two applications.
There are applications that can be installed in different working directories of the same workstation, and at the same time they will work independently of each other, without creating any conflict situations. But it's not always the case. There may be conflicts with access to shared resources on disk, in memory, and / or in the system. All of this should be tested in the most thorough manner.

Update

The backward compatibility check includes the following steps:
● After installing the update, all previously created objects by the application, such as documents, forms, saves (if this is a game) should open and work without errors. This behavior is called backward compatibility. User settings should remain the same, of course, if the updates did not affect their change.
● The documents of the same type created in the new version should open correctly in older versions, of course, if the purpose of the update was not to change the file format and structure. If a new format was introduced, then the new version should have the ability to save the document in the old format.
● If the updated application is running, the user should receive a warning that the update is not possible when the application is running.

Note that the update process can be stopped at any time, while the original application must remain unchanged and in working condition. Suppose you have installed version 1 application, you are trying to upgrade it to version 2, but during the installation process you changed your mind and interrupted the installation. In this case, the installer should return all the changes that have already been made, clean the temporary files used for the update, and complete their work. However, version 1 application remains operational.

Removal (Uninstall)

It is considered good practice to delete the ones created during the installation process (registry entries in the system registry, libraries in the % Windir% \ system32 system directories , files, etc.). Conditionally, the uninstallation testing process can be divided into several parts:
● Checking that the service records and links to libraries that were installed during the installation were removed from the system registry.
● Verify the physical deletion of application files.
● Verification that after uninstalling the application, the file extensions registered during installation are deleted, and the existing ones (registered before installation) are restored.
● Checking the safety of data created during the work with the application. It is likely that they lie somewhere deep in the catalog of the program itself. This can be service scripts, save from games or other user-created data, the removal of which will damage the user. Just imagine that when you uninstall MS Office, all your documents will be deleted along with it. Therefore, such data cannot be deleted without user confirmation.
● If the uninstalled application is running, the user should be warned that uninstallation is not possible while the application is running.

It is worth checking the behavior of the installer if the program directory is closed for deletion by access rights. In this case, the deletion process should not be performed, and the user should receive an appropriate message.

Testing the Installation Wizard

In most cases, the installer is an application in the form of a wizard, which may have specific requirements, the testing recommendations of which are discussed below:

Smart people wrote: "Wizards are evil." You can agree with this or not, but you still have to test them. The following test plan for the installation wizard is proposed:
● Identify all paths from start to finish, and then prioritize each of them. This will help us to avoid unnecessary costs and efforts when passing low-priority routes.
● Forget about the GUI. Try to describe test cases without being tied to interface elements. For example, the checkbox / radiobutton GUI controls or a two-item menu is just a choice between true and false, what matters is what it ultimately affects.
● If, according to the results of passing the wizard, one obtains a file (a file describing the properties in the form of a list: property = value), which is then passed on to the export procedure, in this case it is possible to divide the checks into two stages - the first one, create (generate) such wipe files and verify that the export is working correctly. The second is to verify that the correct file is obtained through the GUI.
● Do not forget to do such a routine type of testing wizards as going back and forth through the pages:
without changing anything, all answers should be saved; changing something on the previous page, the next one should undergo an adequate change or reset of answers.
● Make sure that the wizard responds adequately to the wrong answers and prevents it from moving on.
● The Cancel (Close) button should always work and on all pages of the wizard.
● Create a template result for each of the possible paths of the installation wizard (ideally, make several of them for different input data). Then, if possible, automatically or manually compare the result with the template result.
● Highlight those options that do not affect any others, and which others do not affect. The operation of these options can be tested in isolation from others.

Cross-platform testing of installers

A separate point I want to highlight is cross-platform testing of installers, which must be carried out for all three functions - installation, updating and removal:
● Correctness of the installer to work with different versions of the OS, ServicePack and installed updates.
● Check files, drivers and libraries when installing under different OS.
● Check access rights to files, folders and system records for different operating systems.
● Checking permissions set on application files.

To simplify the testing procedure, it is recommended to create a table where the required configurations will go in columns, and test cases or functions to be tested in rows. In the process of testing at the intersection of columns and rows, fill out the result, which can visually show the progress of testing and compliance with the “cross-platform” requirements.

If the application should work in several OSs, then the installer should check the necessary set of functions in the process of its work. It is possible that some kind of update package should be installed somewhere or just need to work with another library.
Case study: “The application must be closed for installation on Windows Vista. We launch it, the program thinks and displays a message that this version will not work under this OS. That's right, click OK. After that, the computer thinks for a couple of seconds and reports the following: “The application was completed incorrectly. Let's try to run it in compatibility mode? “We agree and launch. Our application starts automatically again, but this time it is put “with a bang”. We don’t know what Vista does and how, but the security code had to be redone. ”

Checking the list of installed files is done by analogy, as for the installation of drivers and libraries, then the special importance of this check should be noted. Not all drivers and libraries work equally well on different platforms.
Case study: “For some drivers, there is a dependency on the file system on which the OS is running. More precisely for some types of errors. There was a case when everything worked fine on NTFS, but not on FAT32. The reason was an invalid entry when installing the driver in the registry. ”You
must verify that the installer has access rights to files, folders, and system records. This is especially important when installing under the UNIX family of OSs, with their severe restrictions on access to resources for different categories of users. When / after installation on unix systems, files must have appropriate Permissions. Those. if the file is intended to run, then it must be runable, if it is a configuration file, for example, then there must be permissions for modification, etc.

---

The idea of ​​my friend is dartos .