Alert "Lida": security testing automation
Good afternoon, dear readers. My name is Victor Burov, I am a developer in the ISPsystem. In the last post I talked about the tool for creating autotests , today I will share the experience of automating security testing.
At first, an individual employee was looking for vulnerabilities in our products. Manual testing took a lot of time and did not guarantee that all vulnerabilities would be found. Having found out the basic laws of testing, we came to the conclusion that it can be automated. Then we decided to write a utility that will facilitate the life of the tester, save his time and allow you to check the products after each change. Since the tester was called Lida, we called the new application in her honor. In general, in our company, it has become a tradition to call testing tools by the names of testers.
After analyzing the vulnerability search utilities, I came to the conclusion that they all need to specify the function to call and the parameters used. We again decided to take advantage of the unified interface and formulated the requirements for Lida.
Implement all this was not easy.
To find vulnerabilities in a function, it must be executed by passing the necessary parameters. Our interface is based on lists and forms, so it is possible to automate data collection by processing xml documents describing the structure of interface elements.
I decided to start a crawl from the main menu, recursively going into all nested lists. Lida opens the first level list. As a rule, it has several buttons that call some functions.
If the button opens a form, then the result of the call will be an xml-document with nodes containing information about the fields: name, validator, range of acceptable values. Based on this information, field values are generated. For example, an int will generate a number. If there is no validator, a random string is generated. After filling in all the form fields, a request is sent.
If the function is a list, it will be opened and functions associated with the buttons will be called for its elements.
When checking lists, there is a problem - all lists must have a set of entries that will ensure that all buttons in the list are clickable.
SQL injection is probably one of the most common problems for applications, including ours. Many function calls spawn various queries to the DBMS. An error occurs when parameters that come from outside are inserted into the request body "as is". The consequences of such errors can be sad: from unauthorized data acquisition to the removal of tables.
To start searching for SQL injections, I organized the output of all SQL queries to a file. After executing the function, the application searches for the values of the passed parameters in the resulting SQL queries.
You could use the log of the SQL server itself. But in our case there is only one method for executing queries and adding logging to it was easy. Because of this, we know exactly what call originated a particular request.
When the value of the passed parameter is found, the utility passes in this parameter a value containing a single quote. If the quote is found in the same sequence, it means that the parameter is not escaped - we have found a place for SQL injections.
A similar problem occurs when we make any system calls. I decided to look for them with strace and selected the optimal launch parameters for it. Example of running ISPmanager:
As with SQL injections, Lida performs the function and analyzes the output of strace to get into the functions open, unlink, rmdir, chmod chown, chflags, mknod, mkfifo, fcntl, symlink, link, execve, mkdir of the passed parameters.
If the parameter is found, the utility passes in it a value containing, for example, the path with the transition to the directory above. If it hits as-is, then a potential vulnerability is found.
The analysis of the execve function was very useful. It allows you to determine in which function the launch arguments of executable files are not screened. This error is very expensive, because through it you can get root access to the server by simply changing the password.
Another interesting test: checking the order of calling the stat functions and others. Often, access is first checked through stat, and then any unsafe actions, which leaves room for substitution. But we did not automate this.
We check access to foreign objects from under the user for entities of another user and administrator. In this mode, we check the ability to read, modify and view lists of elements of another user.
"Lida" bypasses all functions available on behalf of the owner or administrator and remembers their elements. Then it calls the same functions with elements from under another user. In an ideal situation, the answer should be an error like Access or Missed. If this error is not received, therefore, with high probability you can read the data of another user.
In some cases, the absence of an error does not mean that you can get direct access to the objects of another user. At the beginning of such functions, we add a check on access rights to the element. This is how not only security is checked, but also the correctness of server responses.
Validating our APIs is an added bonus of function checking for vulnerabilities. Analyzing reports on Lida's work, we learned how to return the correct types of errors, made our API more convenient and logical. As a result, as with the tape recorder, we received not only a security check, but once again checked our API for "consistency."
The utility can work with all our products, therefore, it tests many different functions. Basically, false positives appeared in the mode of checking access to other objects. This is due to the peculiarities of the buttons and functions.
False alarms were Lida’s biggest problem. It seemed that it was completely debugged, but when checking on different panels all new false positives appeared. As a result, there were several stages of their correction.
The main part of actions in the panel is performed on an entity (domain name, database, etc.). To achieve maximum automation, Lida had to create these entities automatically. But in practice it turned out to be hard to implement. Sometimes validation is performed in code, so it is not always possible to substitute the value of the parameter automatically. The second reason is dependent entities. For example, to create a mailbox you need to create a mail domain.
Therefore, we decided not to fight for full automation. Before starting, the tester creates the entities manually and takes a snapshot of the machine, since after the entity has been checked, they will be changed. This allows you not to skip checking the group of functions in case of an unsuccessful creation of an entity.
Virtually every list has functions to remove or deactivate an entity. If you perform them in the order, then the entity will be deleted before performing other functions. I define such functions and perform after others. Additionally added a key that prohibits the execution of such functions.
Some functions reboot the server. They have to track and add to the list of ignored.
Due to the peculiarities of the operation logic, some functions restart the panel. During security testing, this leads to the fact that the panel is launched without a trace of SQL queries or strace - further testing becomes meaningless. It is necessary to monitor and re-launch the panel in trace mode.
On forms there are text entry fields, checkboxes, drop-down lists, the values of which determine the availability of the values of other fields. For each value of the dependent field there may be a separate section of code, therefore, parts when they can remain untested.
To solve this problem, I added algorithms for analyzing dependent fields and checking all combinations of dependent controls.
Service functions are not available for migration, but may contain vulnerabilities. To identify and verify them, we have a special function that returns a list of all registered functions. We compare this list with the list of proven functions. There are no metadata for service functions, therefore it is impossible to check the parameters processed within them. In order to at least check such functions somehow, I pass on our standard parameters elid, plid and others.
We included Lida in every Jenkins nightly build. According to the results of her work, a verification report is generated, information about the suspicious function is displayed in it with all parameters.
The task closed by the developer is now checked not only by the tester, but also by Lida. The tester processes the report received: copies the parameters of the suspicious function to the browser and analyzes the behavior and log of the panel. If the vulnerability is confirmed, log the error to the developer.
At first, an individual employee was looking for vulnerabilities in our products. Manual testing took a lot of time and did not guarantee that all vulnerabilities would be found. Having found out the basic laws of testing, we came to the conclusion that it can be automated. Then we decided to write a utility that will facilitate the life of the tester, save his time and allow you to check the products after each change. Since the tester was called Lida, we called the new application in her honor. In general, in our company, it has become a tradition to call testing tools by the names of testers.
After analyzing the vulnerability search utilities, I came to the conclusion that they all need to specify the function to call and the parameters used. We again decided to take advantage of the unified interface and formulated the requirements for Lida.
Requirements at the start:
- Automatic building lists of functions.
- Autocomplete settings.
- Making requests to the API.
- Analysis of data output after performing functions.
- Search for data vulnerabilities.
- Formation of reports.
- Flexible settings.
Implement all this was not easy.
Implementation
Bypassing forms and lists
To find vulnerabilities in a function, it must be executed by passing the necessary parameters. Our interface is based on lists and forms, so it is possible to automate data collection by processing xml documents describing the structure of interface elements.
I decided to start a crawl from the main menu, recursively going into all nested lists. Lida opens the first level list. As a rule, it has several buttons that call some functions.
If the button opens a form, then the result of the call will be an xml-document with nodes containing information about the fields: name, validator, range of acceptable values. Based on this information, field values are generated. For example, an int will generate a number. If there is no validator, a random string is generated. After filling in all the form fields, a request is sent.
If the function is a list, it will be opened and functions associated with the buttons will be called for its elements.
When checking lists, there is a problem - all lists must have a set of entries that will ensure that all buttons in the list are clickable.
SQL injection search
SQL injection is probably one of the most common problems for applications, including ours. Many function calls spawn various queries to the DBMS. An error occurs when parameters that come from outside are inserted into the request body "as is". The consequences of such errors can be sad: from unauthorized data acquisition to the removal of tables.
To start searching for SQL injections, I organized the output of all SQL queries to a file. After executing the function, the application searches for the values of the passed parameters in the resulting SQL queries.
You could use the log of the SQL server itself. But in our case there is only one method for executing queries and adding logging to it was easy. Because of this, we know exactly what call originated a particular request.
When the value of the passed parameter is found, the utility passes in this parameter a value containing a single quote. If the quote is found in the same sequence, it means that the parameter is not escaped - we have found a place for SQL injections.
System call analysis
A similar problem occurs when we make any system calls. I decided to look for them with strace and selected the optimal launch parameters for it. Example of running ISPmanager:
strace -f -s 1024 -e trace=file,process bin/core ispmgrAs with SQL injections, Lida performs the function and analyzes the output of strace to get into the functions open, unlink, rmdir, chmod chown, chflags, mknod, mkfifo, fcntl, symlink, link, execve, mkdir of the passed parameters.
If the parameter is found, the utility passes in it a value containing, for example, the path with the transition to the directory above. If it hits as-is, then a potential vulnerability is found.
The analysis of the execve function was very useful. It allows you to determine in which function the launch arguments of executable files are not screened. This error is very expensive, because through it you can get root access to the server by simply changing the password.
When users find a vulnerability in our products, the company pays a cash reward, the amount of which depends on the category of vulnerability. This approach may be cheaper than finding the error on its own (the tester may not find an error, and receive a salary).
Another interesting test: checking the order of calling the stat functions and others. Often, access is first checked through stat, and then any unsafe actions, which leaves room for substitution. But we did not automate this.
Check access to other objects
We check access to foreign objects from under the user for entities of another user and administrator. In this mode, we check the ability to read, modify and view lists of elements of another user.
"Lida" bypasses all functions available on behalf of the owner or administrator and remembers their elements. Then it calls the same functions with elements from under another user. In an ideal situation, the answer should be an error like Access or Missed. If this error is not received, therefore, with high probability you can read the data of another user.
In some cases, the absence of an error does not mean that you can get direct access to the objects of another user. At the beginning of such functions, we add a check on access rights to the element. This is how not only security is checked, but also the correctness of server responses.
API Validation
Validating our APIs is an added bonus of function checking for vulnerabilities. Analyzing reports on Lida's work, we learned how to return the correct types of errors, made our API more convenient and logical. As a result, as with the tape recorder, we received not only a security check, but once again checked our API for "consistency."
Difficulties
False alarms
The utility can work with all our products, therefore, it tests many different functions. Basically, false positives appeared in the mode of checking access to other objects. This is due to the peculiarities of the buttons and functions.
False alarms were Lida’s biggest problem. It seemed that it was completely debugged, but when checking on different panels all new false positives appeared. As a result, there were several stages of their correction.
Entity creation
The main part of actions in the panel is performed on an entity (domain name, database, etc.). To achieve maximum automation, Lida had to create these entities automatically. But in practice it turned out to be hard to implement. Sometimes validation is performed in code, so it is not always possible to substitute the value of the parameter automatically. The second reason is dependent entities. For example, to create a mailbox you need to create a mail domain.
Therefore, we decided not to fight for full automation. Before starting, the tester creates the entities manually and takes a snapshot of the machine, since after the entity has been checked, they will be changed. This allows you not to skip checking the group of functions in case of an unsuccessful creation of an entity.
Call destructive functions
Virtually every list has functions to remove or deactivate an entity. If you perform them in the order, then the entity will be deleted before performing other functions. I define such functions and perform after others. Additionally added a key that prohibits the execution of such functions.
Some functions reboot the server. They have to track and add to the list of ignored.
Due to the peculiarities of the operation logic, some functions restart the panel. During security testing, this leads to the fact that the panel is launched without a trace of SQL queries or strace - further testing becomes meaningless. It is necessary to monitor and re-launch the panel in trace mode.
Check dependent parameters
On forms there are text entry fields, checkboxes, drop-down lists, the values of which determine the availability of the values of other fields. For each value of the dependent field there may be a separate section of code, therefore, parts when they can remain untested.
To solve this problem, I added algorithms for analyzing dependent fields and checking all combinations of dependent controls.
Check functions not available in the interface
Service functions are not available for migration, but may contain vulnerabilities. To identify and verify them, we have a special function that returns a list of all registered functions. We compare this list with the list of proven functions. There are no metadata for service functions, therefore it is impossible to check the parameters processed within them. In order to at least check such functions somehow, I pass on our standard parameters elid, plid and others.
Conclusion
We included Lida in every Jenkins nightly build. According to the results of her work, a verification report is generated, information about the suspicious function is displayed in it with all parameters.
The task closed by the developer is now checked not only by the tester, but also by Lida. The tester processes the report received: copies the parameters of the suspicious function to the browser and analyzes the behavior and log of the panel. If the vulnerability is confirmed, log the error to the developer.