On the issue of performance of old and new versions of the node
From time to time, developers who support old nodovskie applications, have doubts about the need to transfer the application from the old versions of the node to the new. The main argument in favor of staying on old is “I don’t use new features, but their implementation probably slows down the processing as a whole”. Plus, the situation can be greatly complicated by the presence of dependencies on libraries that are no longer supported under new nodes, and the change of the node version automatically results in a significant reworking of the application architecture. My article, I hope, will help them to decide on this issue.
Before you begin, let me remind you the basic principle of IT business: WORKS, DO NOT TROUBLE!
If your application works as it should, if it completely copes with the tasks and there is no urgent need to recycle it, it is better to leave everything as it is. Recycling can be a very painful and lengthy process. As a result, you will not get any real tangible benefits, except for a sense of aesthetic satisfaction, and at this time your business will suffer.
My article is for those who still have a need. I will describe my recent situation, talk about the difficulties I encountered in resolving it, and give the results I finally got.
I am developing a system for collecting and processing logs of other applications. Established workloads - hundreds of thousands of logs per minute per component. The system scaled well horizontally, had a modular architecture, worked successfully for years and coped with its functions as a whole. The version of the node used is 0.10
What did you encounter?
Naturally, not a lack of functionality. The new features of es6 were not even considered as an argument. Of course, they make life a little more pleasant, but no more. For any of our tasks, the functionality of the old node was quite enough. Problems arose in the most unexpected place as the complexity of the functional.
Problem one:
One of the components at peak incoming loads and memory consumption for 5GB suddenly began to hellishly slow down. The problem did not arise every time, spontaneously, usually towards the end of the peak. If the application did not fall on timeouts, the speed gradually recovered within half an hour. Reboot cured immediately.
After the “deep debugging” process, it turned out that the whole process begins to slow down, even synchronous operations, which made it concluded that the node itself is “getting ill”. What to do with it, it was absolutely not clear. The only solution was a search through the history of changes to our code in a few months and a rollback of important functionality. On the other hand, the problem did not arise often, not every day, and a manual restart of the system also seemed to be an entirely acceptable solution (manual means a script at the lag detector signal). We are more inclined to the second option. And if it were not for the second problem, then most likely it would have been implemented.
Problem two:
Another of our components ate a lot of memory. Since this component was in fact a cache, its “greed” was in principle explicable. Until the moment it was necessary to limit it from above to a strictly specified volume. And then it turned out that the component is gaining memory and not in a hurry to give it. And even working at almost idle. That is, at the time of the peak load, the memory manager of the node took the memory to the maximum, even with a margin, and then kept it until the end of the centuries (reloading the component, for example). At once I will mention that naturally the variant of leaks was considered and checked. Alas, there were no leaks, and we were again at an impasse.
I tried to ask in various places on the Internet how to manage memory management of the node and how to solve our situation. But in response I received only a lot of negativity about the use of node 0.10. In general, it was this negative that led me to the task of switching to the latest versions of the node.
What was holding back?
1. Fears of loss of productivity.
Who worked on the python, he remembers that the transition from the 2.x to 3.x line was accompanied by a solid loss of performance. I do not know how things are in the python now, perhaps the situation has improved. But it is quite logical to expect that after adding all these new features of es6, the node could also solidly sink. I tried to google some benchmarks to compare new nodes with old ones, but I didn’t find anything sensible.
2. Performance JSON.parse
Since we work with logs, JSON.parse takes the lion's share of the processor. At one time, we compared it on different versions of the node and got a performance drop of about 30%. Compared node 4.x with 0.10 and 0.12.
In fact, these reasons were not decisive, as the processor was not a bottleneck. In addition, for such tasks there is horizontal scaling.
3. Package dependencies
And this was the stumbling block. Our central, most complex component used the libmysqlclient package, which works only under the 0.10 node. Worse, his synchronous calls. That is, it was impossible to simply change the mysql driver, replacing one call with another, without thoroughly reworking the component's architecture from partially synchronous to fully asynchronous processing. Moreover, the most difficult thing in this task was not so much the processing itself, as to justify to the management its necessity and to show what exactly it would give the project :)
What did it give ?
As a result, we still moved from node 0.10 to the last lts 8.11.x
1. There was no performance drop . On the contrary, we received an increase of about 10-15%
2. Memory consumption has significantly improved, by about 30-50%, depending on the component
3. The “intractable” problems voiced above resolved by themselves
4. We finally got the opportunity to use new features from es6! Although out of habit, we still do not use them yet)))
Before you begin, let me remind you the basic principle of IT business: WORKS, DO NOT TROUBLE!
If your application works as it should, if it completely copes with the tasks and there is no urgent need to recycle it, it is better to leave everything as it is. Recycling can be a very painful and lengthy process. As a result, you will not get any real tangible benefits, except for a sense of aesthetic satisfaction, and at this time your business will suffer.
My article is for those who still have a need. I will describe my recent situation, talk about the difficulties I encountered in resolving it, and give the results I finally got.
I am developing a system for collecting and processing logs of other applications. Established workloads - hundreds of thousands of logs per minute per component. The system scaled well horizontally, had a modular architecture, worked successfully for years and coped with its functions as a whole. The version of the node used is 0.10
What did you encounter?
Naturally, not a lack of functionality. The new features of es6 were not even considered as an argument. Of course, they make life a little more pleasant, but no more. For any of our tasks, the functionality of the old node was quite enough. Problems arose in the most unexpected place as the complexity of the functional.
Problem one:
One of the components at peak incoming loads and memory consumption for 5GB suddenly began to hellishly slow down. The problem did not arise every time, spontaneously, usually towards the end of the peak. If the application did not fall on timeouts, the speed gradually recovered within half an hour. Reboot cured immediately.
After the “deep debugging” process, it turned out that the whole process begins to slow down, even synchronous operations, which made it concluded that the node itself is “getting ill”. What to do with it, it was absolutely not clear. The only solution was a search through the history of changes to our code in a few months and a rollback of important functionality. On the other hand, the problem did not arise often, not every day, and a manual restart of the system also seemed to be an entirely acceptable solution (manual means a script at the lag detector signal). We are more inclined to the second option. And if it were not for the second problem, then most likely it would have been implemented.
Problem two:
Another of our components ate a lot of memory. Since this component was in fact a cache, its “greed” was in principle explicable. Until the moment it was necessary to limit it from above to a strictly specified volume. And then it turned out that the component is gaining memory and not in a hurry to give it. And even working at almost idle. That is, at the time of the peak load, the memory manager of the node took the memory to the maximum, even with a margin, and then kept it until the end of the centuries (reloading the component, for example). At once I will mention that naturally the variant of leaks was considered and checked. Alas, there were no leaks, and we were again at an impasse.
I tried to ask in various places on the Internet how to manage memory management of the node and how to solve our situation. But in response I received only a lot of negativity about the use of node 0.10. In general, it was this negative that led me to the task of switching to the latest versions of the node.
What was holding back?
1. Fears of loss of productivity.
Who worked on the python, he remembers that the transition from the 2.x to 3.x line was accompanied by a solid loss of performance. I do not know how things are in the python now, perhaps the situation has improved. But it is quite logical to expect that after adding all these new features of es6, the node could also solidly sink. I tried to google some benchmarks to compare new nodes with old ones, but I didn’t find anything sensible.
2. Performance JSON.parse
Since we work with logs, JSON.parse takes the lion's share of the processor. At one time, we compared it on different versions of the node and got a performance drop of about 30%. Compared node 4.x with 0.10 and 0.12.
In fact, these reasons were not decisive, as the processor was not a bottleneck. In addition, for such tasks there is horizontal scaling.
3. Package dependencies
And this was the stumbling block. Our central, most complex component used the libmysqlclient package, which works only under the 0.10 node. Worse, his synchronous calls. That is, it was impossible to simply change the mysql driver, replacing one call with another, without thoroughly reworking the component's architecture from partially synchronous to fully asynchronous processing. Moreover, the most difficult thing in this task was not so much the processing itself, as to justify to the management its necessity and to show what exactly it would give the project :)
What did it give ?
As a result, we still moved from node 0.10 to the last lts 8.11.x
1. There was no performance drop . On the contrary, we received an increase of about 10-15%
2. Memory consumption has significantly improved, by about 30-50%, depending on the component
3. The “intractable” problems voiced above resolved by themselves
4. We finally got the opportunity to use new features from es6! Although out of habit, we still do not use them yet)))