Proper use of require in node.js

Foreword

Not so long ago, the project I'm working on at the moment, began to use the modular system ES2015. I will not focus on this JavaScript technology, because the article is not about that at all, but about how the technology has led me to one thought.

As many people know, ES2015 Modules are importing / exporting scripts that are very similar in syntax to pythonmany other programming languages. Example:

// Helper.js
export function includes(array, variable) {
    return array.indexOf(variable) !== -1;
}
// main.js
import {includes} from 'Helper';
assets(includes([1,2,3], 2), true);

Everyone who is interested in JavaScript modules knows that importing and exporting is possible only at the top level of the module (file with code).

The following crude code example will cause errors:

// sendEmail.js
export default function sendEmails(emails_list) {
    import sender from 'sender';
    export sender;
    // сделать что-то
}

Exception: SyntaxError: import/export declarations may only appear at top level of a module

Unlike ES2015 Modules - in a modular system, node.jsimporting and exporting is possible at any level of nesting.

Similar code node.jsdoes not cause an error:

// sendEmail.js
module.exports = function sendEmails(emails_list) {
    const sender = require('sender');
    exports.sender = sender;
    // сделать что-то
}

The advantage of this method is that the modules necessary in the handler are explicitly imported inside and do not clog the module namespace (this is especially important if the imported module is needed in only one handler). There is also the possibility of delayed export of module data.

The main disadvantages:

1. You will only learn about the absence of a module when you call the corresponding handler
2. The path to the imported module may change, which will lead to a change at each import location (for example, in your module, it is used in various handlers lodash/object/defaultsand you decide to upgrade to the 4.x version where you need to connect lodash/defaults).

Debriefing

In most tasks for which it is used node.js- front-end or the main web server, and a high load on a node.jsfrequent occurrence. The throughput of your server should be as high as possible.

Bandwidth measurement

Apache's awesome utility, ab, is used to measure web server throughput . If you are not familiar with it, then I highly recommend doing it.

The web server code is the same except for the handlers.
The test was run on node.js 6.0using a module ifnodemade on the basis ofexpress

Importing modules directly into the handler

The code:

const app = require('ifnode')();
const RequireTestingController = app.Controller({
    root: '/',
    map: {
        'GET /not_imported': 'notImportedAction'
    }
});
RequireTestingController.notImportedAction = function(request, response, next) {
    const data = {
        message: 'test internal and external require'
    };
    const _defaults = require('lodash/object/defaults');
    const _assign = require('lodash/object/assign');
    const _clone = require('lodash/lang/clone');
    response.ok({
        _defaults: _defaults(data, {
            lodash: 'defaults'
        }),
        _assign: _assign(data, {
            lodash: 'assign'
        }),
        _clone: _clone(data)
    });
};

Result:

$ ab -n 15000 -c 30 -q "http://localhost:8080/not_imported"
Server Hostname:        localhost
Server Port:            8080
Document Path:          /not_imported
Document Length:        233 bytes
Concurrency Level:      30
Time taken for tests:   4.006 seconds
Complete requests:      15000
Failed requests:        0
Total transferred:      6195000 bytes
HTML transferred:       3495000 bytes
Requests per second:    3744.32 [#/sec] (mean)
Time per request:       8.012 [ms] (mean)
Time per request:       0.267 [ms] (mean, across all concurrent requests)
Transfer rate:          1510.16 [Kbytes/sec] received
Percentage of the requests served within a certain time (ms)
  50%      6
  66%      7
  75%      8
  80%      8
  90%     10
  95%     15
  98%     17
  99%     20
 100%    289 (longest request)

Importing modules at the beginning of a file

The code:

const app = require('ifnode')();
const _defaults = require('lodash/object/defaults');
const _assign = require('lodash/object/assign');
const _clone = require('lodash/lang/clone');
const RequireTestingController = app.Controller({
    root: '/',
    map: {
        'GET /already_imported': 'alreadyImportedAction'
    }
});
RequireTestingController.alreadyImportedAction = function(request, response, next) {
    const data = {
        message: 'test internal and external require'
    };
    response.ok({
        _defaults: _defaults(data, {
            lodash: 'defaults'
        }),
        _assign: _assign(data, {
            lodash: 'assign'
        }),
        _clone: _clone(data)
    });
};

Result:

$ ab -n 15000 -c 30 -q "http://localhost:8080/already_imported"
Server Hostname:        localhost
Server Port:            8080
Document Path:          /already_imported
Document Length:        233 bytes
Concurrency Level:      30
Time taken for tests:   3.241 seconds
Complete requests:      15000
Failed requests:        0
Total transferred:      6195000 bytes
HTML transferred:       3495000 bytes
Requests per second:    4628.64 [#/sec] (mean)
Time per request:       6.481 [ms] (mean)
Time per request:       0.216 [ms] (mean, across all concurrent requests)
Transfer rate:          1866.83 [Kbytes/sec] received
Percentage of the requests served within a certain time (ms)
  50%      5
  66%      6
  75%      6
  80%      7
  90%      8
  95%     14
  98%     17
  99%     20
 100%     38 (longest request)

Results Analysis

Importing the modules at the beginning of the file reduced the time of a single request by ~ 23% (!) (In comparison with importing directly into the handler), which is very significant.

Such a big difference in the results lies in the function require. Before importing, it requirerefers to the algorithm for finding the absolute path to the requested component (the algorithm is described in the node.js documentation). When the path was found, it requirechecks whether the module has been cached, and if not, does nothing supernatural, except for calling the usualfs.readFileSync for .jsboth .jsonformats and undocumentedprocess.dlopen for loading C ++ modules.

Note: I tried to "warm up" the cache for the case of directly importing the modules into the handler (before running the ab utility , the modules were already cached) - the performance improved by 1-2%.

conclusions

If you use node.js as a server (there is no difference which one is TCP / UDP or HTTP (S)), then:

1. Importing all modules must be done at the beginning of the file to avoid unnecessary synchronous operations related to loading modules (one of the main anti-patterns for using node.js as an asynchronous server).
2. You can not waste resources on calculating the absolute path of the requested module (this is the main place for performance loss).