April 3, 2015 at 12:53Traffic Inspector: full reboot
Hello, Habr! Almost 12 years ago, we, Smart Soft , released the first version of our flagship product Traffic Inspector . During this time, it has evolved from a conventional proxy server to a universal security gateway that combines the functionality of a proxy, mail, antispam, web servers with Phishing Blocker, RBL, RAS Dialer service modules, NetPolice content filtering plug-in, and advanced CCC-certified billing system and FSTEC, and the ability to account for different types of traffic (IMAP, SMTP, POP, NAT and proxies).
As with any serious product, there were flaws in the Traffic Inspector over the course of evolution. Today we summarize the work on errors that has been carried out by the specialists of our company over the past couple of years.
The central component of the Traffic Inspector is the network driver, which is responsible for directly collecting traffic data and transmitting it to the program service. The driver code is executed in kernel mode, and any errors here are fraught with "falling into the blue screen", forced rebooting and loss of all unsaved data. It is important for customers that with a program of this class (Internet gateway) it is possible to work on the principle of "set - set - forget". The code and error messages were analyzed, the problem of low stability of the driver was successfully solved in early 2014 - “blue screens” are now a rare occurrence.
Other screenshots of the application
Alexander Yamashkin, head of the development department, about the driver:
It is known that the lower the level of the reference model (OSI) at which protection is implemented, the more transparent it is for applications. However, when this level is reduced, the set of available mechanisms decreases and the “error price” is higher. For guaranteed analysis of all packets passing through the network interfaces, NDIS was selected, which refers to the data link layer, but the cost of the error was BSOD. We spent a lot of time analyzing errors in order to say with great certainty that at the moment problems with the driver are possible only in case of a conflict with third-party drivers.
64-bit processors appeared back in 2003. At the same time, in Smart-Soft, the Traffic Inspector service remained 32-bit for a long ten years, which was often the cause of dissatisfaction with the program among system administrators. 32-bit platforms and software have a number of significant limitations: support for no more than 4 TB of hard disk space, support for no more than 3 GB of RAM and no more than 2 GB for user space applications. The Traffic Inspector proxy server (implemented as part of the service) is designed for active use of available memory. The 32-bit version of the service often quickly “consumed” limited memory resources, rested on the memory ceiling, the program could not open more than 4096 threads in the system, the system hung under load, and the proxy began to slow down significantly. The most important improvement of the program is the release of the 64-bit version of the service, released at the end of 2013. In the new version, there are both x86 and x64 versions and the user himself decides which one to install. You can forget about the old memory problems, and the proxy now easily processes up to 1000 simultaneous HTTP requests.
Alexander Yamashkin on introducing work on 64-bit platforms:
Now there are both 32-bit and 64-bit versions of the program, and the user himself decides what to install (at the moment, 70-80% of the installations are 64-bit). Using the latter removes the restrictions imposed by the operating system on the memory capacity of 2 GB, which can be allocated to the process, and therefore there are no errors that occur when it is impossible to allocate the next block of memory to the process.
The web portal Traffic Inspector provides the generation of reports on the functioning of the network and other aspects of network statistics. Many customers purchased Traffic Inspector for this purpose - collecting statistics and reporting. One could understand their disappointment, when the process of generating even a simple report could drag on for a long time. We managed to overcome this problem by translating the web portal to PHP and related technologies.
Alexander Yamashkin on translating the portal into PHP:
Prior to version 3.0.1, the portal was written in C # using the .Net Framework. In addition to changing the interpreter, a report preparation service was implemented, in which all the logic for generating reports was rendered. The portal now has only one function - the GUI.
In addition to accelerating the work of the portal, database distribution and data storage for a long period of time were introduced:
If you need to store data for a long interval, we recommended the use of third-party DBMSs. But even with them periodically there were problems. Imagine the complexity of maintaining a 1.5 TB database in MS SQL, while 90% of this data is not used. Therefore, a mechanism for dividing the database was implemented in 3.0.2, when at the end of the period (month / quarter / six months / year it is set by the administrator), the database, including SQLite, is disconnected from operational work and is used only when requesting reports for a specified period.
Many customers who first encountered the Traffic Inspector program often talked about the great difficulty in its development and an incomprehensible interface. Recently, a number of efforts have been made to improve the usability of the program.
The interface of the Traffic Inspector has noticeably changed. Now using various program elements has become easier and more convenient. In the new structure of the console tree, the elements are arranged more logically and conveniently. The content of the pages for the main nodes of the console tree (Objects, Traffic Accounting, Users and Groups, Rules, Services, Settings) has been completely revised. As a result, the administrator of the Traffic Inspector program receives simple and convenient control over all the features of the program, exhaustive information about its settings and links for solving basic administrative tasks. The new user adding system and a number of wizards simplify the initial setup of the program as much as possible.
Alexander Yamashkin about the new user adding system:
The system, having recorded an Internet connection from a user who is not registered in the program, can either automatically create an account based on the specified parameters, or create an application for the administrator, who can only confirm the creation of a new account and assign the necessary access rules.
The latest changes in the program will be commented by the head of the development department Alexander Yamashkin:
The main improvements were related to the reports and the new registration system, the rest of the improvements are quite specific. For example:
About this will tell Alexander Yamashkin and Vaidas Damoševičius, CTO and partner of Smart-Soft.
Alexander Yamashkin:
I really want to make the process of creating rules for access to information resources extremely simple and understandable to both sophisticated and novice administrators.
Vaidas Damoševičius:
Now we are actively exploring technological solutions for classifications of packages at the seventh level (Layer 7) - we are looking for what we could integrate into the Traffic Inspector. After successful integration, we will introduce new functionality to our users. When it will be? Then, when we ourselves will be satisfied with the results of integration, as we set high quality indicators for ourselves. We also did not forget about the reports - we want to improve them and make them not just reports, but a set of analytical tools that will help system administrators monitor and monitor anomalies in the network.
Smart-Soft company took into account the shortcomings of its software and carried out work to eliminate it. The program moved to a 64-bit platform, significantly increasing the performance of individual components. Numerous corrections and additions have affected most aspects of the program. The new Traffic Inspector is a multifunctional Internet gateway of a new generation with modern functionality, but at the same time understandable to every user interface.
PS: If there is something that we did not say, wishes for correction, we will be glad to hear your comments.
As with any serious product, there were flaws in the Traffic Inspector over the course of evolution. Today we summarize the work on errors that has been carried out by the specialists of our company over the past couple of years.
Low Traffic Inspector Stability
The central component of the Traffic Inspector is the network driver, which is responsible for directly collecting traffic data and transmitting it to the program service. The driver code is executed in kernel mode, and any errors here are fraught with "falling into the blue screen", forced rebooting and loss of all unsaved data. It is important for customers that with a program of this class (Internet gateway) it is possible to work on the principle of "set - set - forget". The code and error messages were analyzed, the problem of low stability of the driver was successfully solved in early 2014 - “blue screens” are now a rare occurrence.
Other screenshots of the application
Alexander Yamashkin, head of the development department, about the driver:
It is known that the lower the level of the reference model (OSI) at which protection is implemented, the more transparent it is for applications. However, when this level is reduced, the set of available mechanisms decreases and the “error price” is higher. For guaranteed analysis of all packets passing through the network interfaces, NDIS was selected, which refers to the data link layer, but the cost of the error was BSOD. We spent a lot of time analyzing errors in order to say with great certainty that at the moment problems with the driver are possible only in case of a conflict with third-party drivers.Slow Performance Traffic Inspector
64-bit processors appeared back in 2003. At the same time, in Smart-Soft, the Traffic Inspector service remained 32-bit for a long ten years, which was often the cause of dissatisfaction with the program among system administrators. 32-bit platforms and software have a number of significant limitations: support for no more than 4 TB of hard disk space, support for no more than 3 GB of RAM and no more than 2 GB for user space applications. The Traffic Inspector proxy server (implemented as part of the service) is designed for active use of available memory. The 32-bit version of the service often quickly “consumed” limited memory resources, rested on the memory ceiling, the program could not open more than 4096 threads in the system, the system hung under load, and the proxy began to slow down significantly. The most important improvement of the program is the release of the 64-bit version of the service, released at the end of 2013. In the new version, there are both x86 and x64 versions and the user himself decides which one to install. You can forget about the old memory problems, and the proxy now easily processes up to 1000 simultaneous HTTP requests.
Alexander Yamashkin on introducing work on 64-bit platforms:
Now there are both 32-bit and 64-bit versions of the program, and the user himself decides what to install (at the moment, 70-80% of the installations are 64-bit). Using the latter removes the restrictions imposed by the operating system on the memory capacity of 2 GB, which can be allocated to the process, and therefore there are no errors that occur when it is impossible to allocate the next block of memory to the process.
Slow web portal and reporting in Traffic Inspector
The web portal Traffic Inspector provides the generation of reports on the functioning of the network and other aspects of network statistics. Many customers purchased Traffic Inspector for this purpose - collecting statistics and reporting. One could understand their disappointment, when the process of generating even a simple report could drag on for a long time. We managed to overcome this problem by translating the web portal to PHP and related technologies.
Alexander Yamashkin on translating the portal into PHP:
Prior to version 3.0.1, the portal was written in C # using the .Net Framework. In addition to changing the interpreter, a report preparation service was implemented, in which all the logic for generating reports was rendered. The portal now has only one function - the GUI.
In addition to accelerating the work of the portal, database distribution and data storage for a long period of time were introduced:
If you need to store data for a long interval, we recommended the use of third-party DBMSs. But even with them periodically there were problems. Imagine the complexity of maintaining a 1.5 TB database in MS SQL, while 90% of this data is not used. Therefore, a mechanism for dividing the database was implemented in 3.0.2, when at the end of the period (month / quarter / six months / year it is set by the administrator), the database, including SQLite, is disconnected from operational work and is used only when requesting reports for a specified period.
Ease of Use Traffic Inspector
Many customers who first encountered the Traffic Inspector program often talked about the great difficulty in its development and an incomprehensible interface. Recently, a number of efforts have been made to improve the usability of the program.
The interface of the Traffic Inspector has noticeably changed. Now using various program elements has become easier and more convenient. In the new structure of the console tree, the elements are arranged more logically and conveniently. The content of the pages for the main nodes of the console tree (Objects, Traffic Accounting, Users and Groups, Rules, Services, Settings) has been completely revised. As a result, the administrator of the Traffic Inspector program receives simple and convenient control over all the features of the program, exhaustive information about its settings and links for solving basic administrative tasks. The new user adding system and a number of wizards simplify the initial setup of the program as much as possible.
Alexander Yamashkin about the new user adding system:
The system, having recorded an Internet connection from a user who is not registered in the program, can either automatically create an account based on the specified parameters, or create an application for the administrator, who can only confirm the creation of a new account and assign the necessary access rules.
New in Traffic Inspector 3.0.2
The latest changes in the program will be commented by the head of the development department Alexander Yamashkin:
The main improvements were related to the reports and the new registration system, the rest of the improvements are quite specific. For example:
- The ability to view requests blocked by access rules has been implemented. This will allow you to figure out the reason for the block, and see which user tried to get to the prohibited resources.
- Displaying extended information when a resource is blocked that has fallen into the Unified Register of domain names containing information whose distribution in the Russian Federation is prohibited.
- Work was done to minimize the installer by eliminating the signatures of the anti-virus databases and the library of anti-virus modules. This data will be loaded the first time you use the modules.
- Work was done to optimize the speed of reporting. Not only were all requests processed, but a data hashing system was implemented in the report generation service.
Future plans to change Traffic Inspector
About this will tell Alexander Yamashkin and Vaidas Damoševičius, CTO and partner of Smart-Soft.
Alexander Yamashkin:
I really want to make the process of creating rules for access to information resources extremely simple and understandable to both sophisticated and novice administrators.
Vaidas Damoševičius:
Now we are actively exploring technological solutions for classifications of packages at the seventh level (Layer 7) - we are looking for what we could integrate into the Traffic Inspector. After successful integration, we will introduce new functionality to our users. When it will be? Then, when we ourselves will be satisfied with the results of integration, as we set high quality indicators for ourselves. We also did not forget about the reports - we want to improve them and make them not just reports, but a set of analytical tools that will help system administrators monitor and monitor anomalies in the network.Conclusion
Smart-Soft company took into account the shortcomings of its software and carried out work to eliminate it. The program moved to a 64-bit platform, significantly increasing the performance of individual components. Numerous corrections and additions have affected most aspects of the program. The new Traffic Inspector is a multifunctional Internet gateway of a new generation with modern functionality, but at the same time understandable to every user interface.
PS: If there is something that we did not say, wishes for correction, we will be glad to hear your comments.
