Highload: ERP strength test. How it was
Greetings, reader!
In this article, I will describe how we tested the performance of our platform .
We planned to test performance from the very beginning of the development of a new version of the system, since we implemented many optimizations. However, they postponed it due to their high busyness, until one day they informed me that they managed to agree on testing on Oracle Exadata Database Machine X2-2.
Here such a server
inspires.
The machine is so powerful that for our tests it was not required to use it all. Actually, we were interested in the question whether the application has become more productive than the previous version.
The previous version worked and works for one of the clientsWe took its parameters (the number of users, goods in the warehouse, warehouses, offices, cash desks, etc.) as the base ones. In the tested configuration, the processes repeated the processes for this client almost completely (with the exception of some specific functions), so we took statistics on which user roles generate the main load.
Under the load, we mean the creation of documents, records in directories, search in directories and the like activity. The list of roles was not a surprise, but it was considered necessary to conduct a formal procedure for checking intuition. We considered roles on the history of changes and using FGA (a very convenient functionality, we used it to share access in the new platform) - again, a pleasant addition to the main purpose of the functional.
So these roles are:
Of course, there are many analysts and heads of departments and departments who spend all day counting all kinds of reports, but these reports are calculated on a StandBy server, and they don’t create loads on Primary, so we decided to exclude this kind of load from the script.
In a real situation, most of the orders are created through the site, and not by people, and at the same time a little less operation occurs. They decided that the top rating is important to us, and did not separately model the script for the site.
We generated the corresponding number of warehouses, offices, goods, managers, cash desks and other objects, built the distribution of sales of goods (if you count in pieces, the distribution almost exactly coincided with the 80/20 formula - 80% of the sales were made by 20% of the assortment).
It was important to accurately reproduce the transaction load - the performance of ERP systems and their scalability is limited only by blocking. Business processes are formulated in such a way that it is impossible to refuse blocking, and, accordingly, the only way is to reduce blocking time. Determine how we coped with this task and was the goal of this testing.
Roles were emulated inside application servers by running one thread per virtual user.
To begin with, it was necessary to check the work of 4000 users - for comparison with a real profile. The Oracle Database server was launched in a virtual environment on 4 cores and 8 gigabytes of memory. Application servers were run on the same server as virtual machines - 2 pieces, each with 2 cores and 2 gigabytes of memory. To our joy, the server easily withstood, the execution time was within one second, the CPU load was about 60-70% and 50% IO, i.e. there were practically no locks, and iron was used quite efficiently.
The results shown exceeded the profile of the real client in all respects.
In addition, the whole machine was at our disposal and we decided to continue, increasing the load by 2000 users for each run (one run took 2-3 hours, 2 runs worked all night).
Empirically established that one application server (2 cores, 2 GB of memory) can withstand 3,000 users.
As a result, we reached the figure of 18,000 users. Attempts were made to launch on 20,000 users, however, the results were unstable. The instability is explained by the stochastic scenario, and, as a result, the emergence of instability points, when hit, the system fell into blocking, from which it could no longer exit. It should be noted that with an increase in the number of users, we kept the user profile and other frequency characteristics - the number of offices, goods, warehouses, cash desks, and so on. Those. virtual company traded the same product through the same offices, without opening new ones! I repeat, the real rating “from above” was important to us, so we did not begin to suck out how to increase the frequency characteristics from the finger.
This article gives a somewhat lyrical description of the testing process, a formal and complete description can be found on our website. The other day, a new version of Oracle Exadata Database Machine X5-2 has appeared, and we are going to test it. To improve performance, try using features such as in-memory database, bulk storage and some others.
We will talk about the results in our blog. Do not switch.
In this article, I will describe how we tested the performance of our platform .
We planned to test performance from the very beginning of the development of a new version of the system, since we implemented many optimizations. However, they postponed it due to their high busyness, until one day they informed me that they managed to agree on testing on Oracle Exadata Database Machine X2-2.
Here such a server
inspires. The machine is so powerful that for our tests it was not required to use it all. Actually, we were interested in the question whether the application has become more productive than the previous version.
The previous version worked and works for one of the clientsWe took its parameters (the number of users, goods in the warehouse, warehouses, offices, cash desks, etc.) as the base ones. In the tested configuration, the processes repeated the processes for this client almost completely (with the exception of some specific functions), so we took statistics on which user roles generate the main load.
Under the load, we mean the creation of documents, records in directories, search in directories and the like activity. The list of roles was not a surprise, but it was considered necessary to conduct a formal procedure for checking intuition. We considered roles on the history of changes and using FGA (a very convenient functionality, we used it to share access in the new platform) - again, a pleasant addition to the main purpose of the functional.
So these roles are:
- Purchasing manager
- Warehouse clerk
- Creating stock replenishment requests
- Storekeeper Movement Set
- Sales Manager
- Cashier
- Storekeeper stock order set
- Storekeeper sending and receiving inter-warehouse transport
Of course, there are many analysts and heads of departments and departments who spend all day counting all kinds of reports, but these reports are calculated on a StandBy server, and they don’t create loads on Primary, so we decided to exclude this kind of load from the script.
In a real situation, most of the orders are created through the site, and not by people, and at the same time a little less operation occurs. They decided that the top rating is important to us, and did not separately model the script for the site.
We generated the corresponding number of warehouses, offices, goods, managers, cash desks and other objects, built the distribution of sales of goods (if you count in pieces, the distribution almost exactly coincided with the 80/20 formula - 80% of the sales were made by 20% of the assortment).
It was important to accurately reproduce the transaction load - the performance of ERP systems and their scalability is limited only by blocking. Business processes are formulated in such a way that it is impossible to refuse blocking, and, accordingly, the only way is to reduce blocking time. Determine how we coped with this task and was the goal of this testing.
Roles were emulated inside application servers by running one thread per virtual user.
To begin with, it was necessary to check the work of 4000 users - for comparison with a real profile. The Oracle Database server was launched in a virtual environment on 4 cores and 8 gigabytes of memory. Application servers were run on the same server as virtual machines - 2 pieces, each with 2 cores and 2 gigabytes of memory. To our joy, the server easily withstood, the execution time was within one second, the CPU load was about 60-70% and 50% IO, i.e. there were practically no locks, and iron was used quite efficiently.
The results shown exceeded the profile of the real client in all respects.
In addition, the whole machine was at our disposal and we decided to continue, increasing the load by 2000 users for each run (one run took 2-3 hours, 2 runs worked all night).
Empirically established that one application server (2 cores, 2 GB of memory) can withstand 3,000 users.
As a result, we reached the figure of 18,000 users. Attempts were made to launch on 20,000 users, however, the results were unstable. The instability is explained by the stochastic scenario, and, as a result, the emergence of instability points, when hit, the system fell into blocking, from which it could no longer exit. It should be noted that with an increase in the number of users, we kept the user profile and other frequency characteristics - the number of offices, goods, warehouses, cash desks, and so on. Those. virtual company traded the same product through the same offices, without opening new ones! I repeat, the real rating “from above” was important to us, so we did not begin to suck out how to increase the frequency characteristics from the finger.
Finally
This article gives a somewhat lyrical description of the testing process, a formal and complete description can be found on our website. The other day, a new version of Oracle Exadata Database Machine X5-2 has appeared, and we are going to test it. To improve performance, try using features such as in-memory database, bulk storage and some others.
We will talk about the results in our blog. Do not switch.