April 21, 2017 at 11:53Difficult lessons: five years with Node.js
- Transfer
After five years working with Node.js, I understood a lot. I have already shared some stories , but this time I want to talk about what knowledge was given most difficult. Bugs, problems, surprises and lessons that you can use in your own projects!
Each new platform has its own tricks , but at the moment these concepts are secondary to me. Understanding your bug is a good way to guarantee learning. Even if it's a little painful!
When I just started working with Node.js, I wrote a scraper . Very quickly, I realized that if nothing was done, then he would carry out many requests in parallel. This alone was an important discovery. But since I have not yet fully mastered the power of the ecosystem , I sat down and wrote my own limiter of parallelism. He worked and verified that no more than N requests were active at any given time.
Later, I needed a second level of concurrency restrictions to make sure that we only serve N users at any given time. But when I implemented the second instance of the class, very strange problems started to appear. Logs have lost their meaning. In the end, I realized that the syntax of class properties in CoffeeScriptdoes not provide a new array for each instance, but one common to all!
For a long time using object-oriented programming languages, I got used to classes. But I did not fully understand the result of the hidden work of CoffeeScript language constructions. Learn your tools well. Check all assumptions .
Once working on a contract, I implemented sorting based on user parameters, which was to be applied in a multi-stage production flow. But we saw really strange behavior - the order did not remain the same. Each time we sent the same set of user parameters, but the order of the elements changed!
I was confused. This was supposed to be a deterministic procedure. The same input data - the same output data. A superficial investigation failed, so I eventually implemented a detailed logging. And here the values surfaced
Be careful with math operations in JavaScript. It is better to have strong guarantees of the quality of incoming data , but you can also check the results of your calculations .
While working on one of my Postgres-connected apps, I noticed strange behavior after tests fell. If the initial test crashed, then all other tests timed out! This did not happen very often, because my tests do not fall often, but this happened. And it was starting to bother. I decided to figure it out.
My code used the Node pg module , so I started rummaging through the directory
Make it a habit to call callbacks last. It is also a good idea to precede their expression
Bugs can be in separate lines of code, but much more painful if the bug is in the application architecture itself ...
Under the contract, I was asked to take a one-page application written in React.js and transfer its rendering to the server. After fixing several parts, due to which it was supposed to render it in the browser, everything worked. But I was very afraid that synchronous operation would put the Node.js server on its knees , so I added a new data collection position to our statistics collected from the server: how long does each page render?
When the data began to arrive, it became clear that the situation was not good. Everything was fine, but some of the pages rendered for more than 400 ms. Too, too long for a production server. My experience with Gartsby prepared me well for the next step: render static files.
Think carefully about what you want from your Node.js. server. Synchronized work is really bad news. HTML rendering requires a lot of synchronous work, and this can slow down the process - not only with React.js, but also with lightweight tools like Jade / Pug ! Even the type checking phase on a large GraphQL load can take a lot of synchronous time!
Specifically for React.js, a promising approach is demonstrated by the Dale Bustad Rapscallion renderer . It splits all the synchronous work to render the React component tree into a string. Redfin's react-server is another, more heavyweight, attempt to solve this problem.
I already actively worked on the contract and implemented the functions at a good speed. But I was told that for the next function, I can check with another Node.js application for reference and assistance with implementation.
I looked at the linking function Express at the exit point that was discussed. And I found a whole bunch of links to
Make dependencies explicit, unless there is an absolute need to do otherwise. For example, instead of adding lines of local locale in
The client wanted me to add features to the Node.js API, which worked as a backend for a large number of installed native applications on tablets and smartphones. I quickly found that I couldn’t just add a field, because application developers used defensive programming - the first action of the application when receiving data was to check according to a strict scheme.
Given this check and the applications themselves, it became clear that two new types of versioning would be needed. One for API clients so that they can upgrade and access new features. The second for the data itself , so that we are confident in the reliable implementation of all these new functions on top of MongoDB. As I was adding this to the application, I was reflecting on how to develop that first version.
There is something in mutable JavaScript objects that delights people in relation to document-oriented DBMSs . “I can create any object in my code, just let me save it somewhere!” Unfortunately, these people seem to leave after writing the first version. They do not think about the second, third or fourth version. I’m learned, so I use Postgres and from the first version I’m thinking about versioning.
As the main expert on Node.js in a large contract project, the DevOps expert approached me to talk about production servers. It took a long time to equip new machines in the data center, and he wanted to make sure that he had the right plan. I appreciated that.
I nodded when he said that each server will have one Node.js. process running. But he stopped nodding at the mention that each server has four physical cores. I explained that only one core could be used on the server, and he shrugged - only such servers could be obtained. They used to work as a store under .NET, and they have standard solutions. Soon after, we introduced cluster .
It’s hard to understand what’s going on if you’re moving from other platforms. Node.js always runs on a single thread, while almost all other web server platforms scale with the server: just add kernels. Use
I have already written a lot of JavaScript code and have learned that testing is absolutely and completely necessary . It really is.
The client explained that their Node.js experts had left and I was replacing them. The company had big plans for this project, which was the main topic of discussion before I started work. Now the client has become more decisive: he realized that things are not going well. But this time I wanted to do everything right.
I was disappointed with the level of test coverage, but at least some tests are available. And I was glad that JSHint is already in place. Just to be sure, I checked the current set of rules. And I was surprised to find that the unused option is not activated. I turned it on and was shocked by the sheer flow of new errors. For several hours I sat and just deleted the code.
JavaScript programming is difficult. But we have the tools to make it more meaningful.Learn how to use ESLint effectively . With a little annotation, Flow can help catch the wrong function calls. A lot of good with a minimum of effort.
I was once asked to help a developer figure out why a test error occurs during a test run. When we looked at the mocha output , we did not see any error during this failure. When we carefully examined the call stack, it became clear that the error was caused by code that was completely unrelated to this test.
After a deeper study, it turned out that the previous test announced successful completion, while initiating a series of asynchronous operations. The exceptions that came from that code were
If you use callbacks, then all tests must complete with the method
On this project under this contract, the standard way of testing was to conduct unit tests or integration tests separately. At least with local development. But Jenkins does full testing, running both test suites together. In one of the pull requests, I added a couple of new tests, and they caused a crash in Jenkins. It really surprised me. Tests normally completed locally!
After some fruitless thought, I turned on the detailed study mode. I ran exactly the command that I knew that Jenkins used to run the tests. It took some time, but the problem was reproduced. My head was spinning while I was trying to figure out what was the difference between the runs. There were no ideas. Detailed logging comes to the rescue! After two runs, I managed to detect some differences. After several false starts, the correct logging was established and it became clear: unit tests changed some of the key application data that were used in the integration tests!
Bugs of this type are extremely difficult to track. Although I am proud that I found this bug, but more and more I think about immutability. Immutable.js librarynot bad, but you have to abandon lodash . And seamless-immutable silently crashes when you try to change something (which then works fine in production).
You can understand now why I am interested in Elixir : all data in Elixir is always immutable.
To a large extent, the benefit of Node.js is the efficient use of a large ecosystem. Choosing good dependencies and managing them properly.
I use the tool
Since I sent a certain number of pull requests to Gatsby, I know the code base pretty well. First, I added some key logging expressions. And an error message appeared! True, it was difficult to interpret. Then I plunged in
Now I quickly followed the trail. After a few minutes, I had a specific piece of code that came up with the same error message. The problem turned out to be in the dependencies of the new ES6 language functions when running under Node.js version 4! It turned out that this dependency has an unlimited sub-dependency that pulls out a too new version
Commit your entire dependency tree to specific versions with Yarn. If you can’t, then fix direct dependencies on specific versions. But be aware that the remaining loose versions of dependencies can lead to this situation.
On one of the projects, I used Async.js , especially the filterLimit function . I had a list of paths, and I wanted to go to the file system to get the characteristics of a file, which would then determine if the paths should remain in the list. I wrote a method for the filter in a normal asynchronous manner, with a standard async signature
I turned to the documentation, which at that time was on the main page of the GitHub project . I looked at the description
Still at a loss, I returned to the code and added exceptionally detailed logging. Uselessly. In the end, I went to the Async.js sources on GitHub. What does this stupid function do? And then I realized what happened - in the branch
Look very carefully at the version number of the documentation you are referring to. It's easy to get lazy, because the first documentation that comes across is often suitable: the methods rarely change, or you are already on the latest version. But it is better to check twice.
I was busy with the Node.js server performance for the client, so I was given access to the monitoring tool: New Relic . Several years ago, New Relic had to be used to monitor a client application on Rails , and I also prepared for another client an analysis of the appropriateness of the New Relic / Node.js bundle, so I generally knew how the system worked and how it integrates with Express and asynchronous calls .
So what I started. The system had surprisingly comprehensive traces of how incoming requests are handled: Express middleware and calls to other servers. But I searched everywhere and could not find a key indicator for processes: the state of the event loop. So I had to resort to a workaround: I manually sent the indicators from toobusy-js to New Relic and created a new chart based on them.
With these additional data, there was more confidence in the correctness of the analysis. Of course, leaps in latency metrics are what New Rellic calls 'time spent in requests'. I looked and found with concern that the sum of the terms there did not match the result. The total time spent on the request and its components do not match. Sometimes there is a category of “Other” that tries to correct the situation, in other cases it is not.
Do not use New Relic to monitor Node.js applications. This tool has no idea about the event loop. Its piled-up graphs of 'average time spent' are completely misleading - given a slow cycle of events, the highlighted endpoints will be the ones that most delay the cycle of events. New Relic can determine if a problem exists, but will not help to determine its source.
If you still need reasons not to use New Relic, here you are:
Emotional events are remembered with the greatest clarity and accuracy . Therefore, each of these situations is safely preserved in my memory. You do not remember it as good as I do. Maybe if you imagine my struggle, it will help?
Surprisingly, initially in this article I wanted to describe twice as many situations worthy of mention, many of which are not directly related to Node.js. So wait for more posts like this!
Basic concepts
Each new platform has its own tricks , but at the moment these concepts are secondary to me. Understanding your bug is a good way to guarantee learning. Even if it's a little painful!
Classes
When I just started working with Node.js, I wrote a scraper . Very quickly, I realized that if nothing was done, then he would carry out many requests in parallel. This alone was an important discovery. But since I have not yet fully mastered the power of the ecosystem , I sat down and wrote my own limiter of parallelism. He worked and verified that no more than N requests were active at any given time.
Later, I needed a second level of concurrency restrictions to make sure that we only serve N users at any given time. But when I implemented the second instance of the class, very strange problems started to appear. Logs have lost their meaning. In the end, I realized that the syntax of class properties in CoffeeScriptdoes not provide a new array for each instance, but one common to all!
For a long time using object-oriented programming languages, I got used to classes. But I did not fully understand the result of the hidden work of CoffeeScript language constructions. Learn your tools well. Check all assumptions .
NaN
Once working on a contract, I implemented sorting based on user parameters, which was to be applied in a multi-stage production flow. But we saw really strange behavior - the order did not remain the same. Each time we sent the same set of user parameters, but the order of the elements changed!
I was confused. This was supposed to be a deterministic procedure. The same input data - the same output data. A superficial investigation failed, so I eventually implemented a detailed logging. And here the values surfaced
NaN. They appeared as a result of previous calculations and wreaked havoc on the sorting algorithm. These values are not equal to themselves or anything else , so they broketransitivity, which is necessary for sorting . Be careful with math operations in JavaScript. It is better to have strong guarantees of the quality of incoming data , but you can also check the results of your calculations .
Logic after a callback
While working on one of my Postgres-connected apps, I noticed strange behavior after tests fell. If the initial test crashed, then all other tests timed out! This did not happen very often, because my tests do not fall often, but this happened. And it was starting to bother. I decided to figure it out.
My code used the Node pg module , so I started rummaging through the directory
node_modulesand adding logging. I found that the module pgneeded to do some internal cleaning after completing the request, which it did after calling the user-provided callback. So if an exception was thrown, this code was skipped. For this reason, pgwas in poor condition and not ready for the next request. Isent a pull request , which was completely redone and added in version 1.0.2 . Make it a habit to call callbacks last. It is also a good idea to precede their expression
return. Sometimes they cannot be put on the last line, but they should always be the last expression.Architecture
Bugs can be in separate lines of code, but much more painful if the bug is in the application architecture itself ...
Event Loop Lock
Under the contract, I was asked to take a one-page application written in React.js and transfer its rendering to the server. After fixing several parts, due to which it was supposed to render it in the browser, everything worked. But I was very afraid that synchronous operation would put the Node.js server on its knees , so I added a new data collection position to our statistics collected from the server: how long does each page render?
When the data began to arrive, it became clear that the situation was not good. Everything was fine, but some of the pages rendered for more than 400 ms. Too, too long for a production server. My experience with Gartsby prepared me well for the next step: render static files.
Think carefully about what you want from your Node.js. server. Synchronized work is really bad news. HTML rendering requires a lot of synchronous work, and this can slow down the process - not only with React.js, but also with lightweight tools like Jade / Pug ! Even the type checking phase on a large GraphQL load can take a lot of synchronous time!
Specifically for React.js, a promising approach is demonstrated by the Dale Bustad Rapscallion renderer . It splits all the synchronous work to render the React component tree into a string. Redfin's react-server is another, more heavyweight, attempt to solve this problem.
Implicit Dependencies
I already actively worked on the contract and implemented the functions at a good speed. But I was told that for the next function, I can check with another Node.js application for reference and assistance with implementation.
I looked at the linking function Express at the exit point that was discussed. And I found a whole bunch of links to
req.randomThingand even some calls req.randomFunction(). Then I went to look at all the connecting functions that I had already walked over to understand what was happening. Make dependencies explicit, unless there is an absolute need to do otherwise. For example, instead of adding lines of local locale in
req.messages, pass req.localeinvar getMessagesForLocale = require('./get_messages')with direct access. Now you will clearly see what your code depends on. This works the other way random_thing.jsaround - if you are a developer , then you definitely want to know which parts of the project use your code!Data, APIs and Versions
The client wanted me to add features to the Node.js API, which worked as a backend for a large number of installed native applications on tablets and smartphones. I quickly found that I couldn’t just add a field, because application developers used defensive programming - the first action of the application when receiving data was to check according to a strict scheme.
Given this check and the applications themselves, it became clear that two new types of versioning would be needed. One for API clients so that they can upgrade and access new features. The second for the data itself , so that we are confident in the reliable implementation of all these new functions on top of MongoDB. As I was adding this to the application, I was reflecting on how to develop that first version.
There is something in mutable JavaScript objects that delights people in relation to document-oriented DBMSs . “I can create any object in my code, just let me save it somewhere!” Unfortunately, these people seem to leave after writing the first version. They do not think about the second, third or fourth version. I’m learned, so I use Postgres and from the first version I’m thinking about versioning.
Equipment
As the main expert on Node.js in a large contract project, the DevOps expert approached me to talk about production servers. It took a long time to equip new machines in the data center, and he wanted to make sure that he had the right plan. I appreciated that.
I nodded when he said that each server will have one Node.js. process running. But he stopped nodding at the mention that each server has four physical cores. I explained that only one core could be used on the server, and he shrugged - only such servers could be obtained. They used to work as a store under .NET, and they have standard solutions. Soon after, we introduced cluster .
It’s hard to understand what’s going on if you’re moving from other platforms. Node.js always runs on a single thread, while almost all other web server platforms scale with the server: just add kernels. Use
cluster, but watch carefully - this is not a magical solution.Testing
I have already written a lot of JavaScript code and have learned that testing is absolutely and completely necessary . It really is.
"Easy work"
The client explained that their Node.js experts had left and I was replacing them. The company had big plans for this project, which was the main topic of discussion before I started work. Now the client has become more decisive: he realized that things are not going well. But this time I wanted to do everything right.
I was disappointed with the level of test coverage, but at least some tests are available. And I was glad that JSHint is already in place. Just to be sure, I checked the current set of rules. And I was surprised to find that the unused option is not activated. I turned it on and was shocked by the sheer flow of new errors. For several hours I sat and just deleted the code.
JavaScript programming is difficult. But we have the tools to make it more meaningful.Learn how to use ESLint effectively . With a little annotation, Flow can help catch the wrong function calls. A lot of good with a minimum of effort.
Cleaning after the test
I was once asked to help a developer figure out why a test error occurs during a test run. When we looked at the mocha output , we did not see any error during this failure. When we carefully examined the call stack, it became clear that the error was caused by code that was completely unrelated to this test.
After a deeper study, it turned out that the previous test announced successful completion, while initiating a series of asynchronous operations. The exceptions that came from that code were
mochaperceived by the process level handler as coming from the current test. Further study showed that mock objects, which also did not clean up, were leaked to other tests. If you use callbacks, then all tests must complete with the method
done(). This is easy to verify while viewing the code: if there is some kind of nested function in the test, then it probably should be done(). Although, there is a little complexity, because you cannot call done, which was not initially passed to the function. One of those classic code viewing errors . Also use the sandbox function in Sinon - it will help make sure that everything is back in place at the end of your test.Mutability
On this project under this contract, the standard way of testing was to conduct unit tests or integration tests separately. At least with local development. But Jenkins does full testing, running both test suites together. In one of the pull requests, I added a couple of new tests, and they caused a crash in Jenkins. It really surprised me. Tests normally completed locally!
After some fruitless thought, I turned on the detailed study mode. I ran exactly the command that I knew that Jenkins used to run the tests. It took some time, but the problem was reproduced. My head was spinning while I was trying to figure out what was the difference between the runs. There were no ideas. Detailed logging comes to the rescue! After two runs, I managed to detect some differences. After several false starts, the correct logging was established and it became clear: unit tests changed some of the key application data that were used in the integration tests!
Bugs of this type are extremely difficult to track. Although I am proud that I found this bug, but more and more I think about immutability. Immutable.js librarynot bad, but you have to abandon lodash . And seamless-immutable silently crashes when you try to change something (which then works fine in production).
You can understand now why I am interested in Elixir : all data in Elixir is always immutable.
Ecosystem
To a large extent, the benefit of Node.js is the efficient use of a large ecosystem. Choosing good dependencies and managing them properly.
Dependencies and Versions
I use the tool
webpack-static-site-generatorto generate my blog . As part of preparing my Git repository for public release, I deleted the directory node_modulesand installed everything from scratch. This usually works as I use the exact version numbers in package.json. But not at this time. And everything stopped working in the strangest way: without any meaningful error message. Since I sent a certain number of pull requests to Gatsby, I know the code base pretty well. First, I added some key logging expressions. And an error message appeared! True, it was difficult to interpret. Then I plunged in
webpack-static-site-generatorand found that he was using Webpackto create a large bundle.jsapplication-wide code that is then passed in to run under Node.js. Madhouse! And it was from there that the error got out - from the depths of this file, during startup under Node.js. Now I quickly followed the trail. After a few minutes, I had a specific piece of code that came up with the same error message. The problem turned out to be in the dependencies of the new ES6 language functions when running under Node.js version 4! It turned out that this dependency has an unlimited sub-dependency that pulls out a too new version
punycode. Commit your entire dependency tree to specific versions with Yarn. If you can’t, then fix direct dependencies on specific versions. But be aware that the remaining loose versions of dependencies can lead to this situation.
Documentation and Versions
On one of the projects, I used Async.js , especially the filterLimit function . I had a list of paths, and I wanted to go to the file system to get the characteristics of a file, which would then determine if the paths should remain in the list. I wrote a method for the filter in a normal asynchronous manner, with a standard async signature
callback(err, result). But nothing worked. I turned to the documentation, which at that time was on the main page of the GitHub project . I looked at the description
filterLimit, and there was the expected signature: callback(err, result). I went back to the project and started npm outdated. I stood v1.5.2, and the last one v2.0.0-rc.1. I was not going to upgrade to this version, so I launchednpm info asyncto check if I have the latest version 1.x. So it was. Still at a loss, I returned to the code and added exceptionally detailed logging. Uselessly. In the end, I went to the Async.js sources on GitHub. What does this stupid function do? And then I realized what happened - in the branch
masteron GitHub was code 2.x. To see the documentation for my installed version 1.5.2, you had to look in the history. When I did this, I found the correct signature callback(result)without the possibility of spreading errors. Look very carefully at the version number of the documentation you are referring to. It's easy to get lazy, because the first documentation that comes across is often suitable: the methods rarely change, or you are already on the latest version. But it is better to check twice.
What New Relic Doesn't Understand
I was busy with the Node.js server performance for the client, so I was given access to the monitoring tool: New Relic . Several years ago, New Relic had to be used to monitor a client application on Rails , and I also prepared for another client an analysis of the appropriateness of the New Relic / Node.js bundle, so I generally knew how the system worked and how it integrates with Express and asynchronous calls .
So what I started. The system had surprisingly comprehensive traces of how incoming requests are handled: Express middleware and calls to other servers. But I searched everywhere and could not find a key indicator for processes: the state of the event loop. So I had to resort to a workaround: I manually sent the indicators from toobusy-js to New Relic and created a new chart based on them.
With these additional data, there was more confidence in the correctness of the analysis. Of course, leaps in latency metrics are what New Rellic calls 'time spent in requests'. I looked and found with concern that the sum of the terms there did not match the result. The total time spent on the request and its components do not match. Sometimes there is a category of “Other” that tries to correct the situation, in other cases it is not.
Do not use New Relic to monitor Node.js applications. This tool has no idea about the event loop. Its piled-up graphs of 'average time spent' are completely misleading - given a slow cycle of events, the highlighted endpoints will be the ones that most delay the cycle of events. New Relic can determine if a problem exists, but will not help to determine its source.
If you still need reasons not to use New Relic, here you are:
- Windows defaults to averages, a poor way to present data from the real world . It is necessary to go further beyond the default windows to get intelligible charts, such as 95%. But do not feel too comfortable, because you cannot add these intelligible charts to dashboards created for you!
- He does not understand use
clusteron one machine. If you manually send data like latency metrics for an event loop outtoobusy-js, only one metric per second will win for the entire server. Even if there are four workers.
All clear!
Emotional events are remembered with the greatest clarity and accuracy . Therefore, each of these situations is safely preserved in my memory. You do not remember it as good as I do. Maybe if you imagine my struggle, it will help?
Surprisingly, initially in this article I wanted to describe twice as many situations worthy of mention, many of which are not directly related to Node.js. So wait for more posts like this!
