Hunting for Memory Leaks in Node.js (1 of 12 Mozilla Identity Team Node.js Articles)

From a translator: This is the first article in the Node.js series from the Mozilla Identity team that is involved in the Persona project . Both the client and server side of Persona are written in JavaScript. In the course of work, the project team created several tools for all occasions - from localization to debugging, dependency management and much more. In this series of articles, Mozilla developers share their experience with the community and these tools that are useful to anyone who writes a highly loaded service on Node.js.

The first article of the series is devoted to a common Node.js problem - memory leaks, leakage features in high-load projects, and the node-memwatch library, which helps to find and fix such leaks in Node.

All articles of the cycle:

1. " Hunting for memory leaks in Node.js "
2. "We load Node to the eyeballs "
3. " We store sessions on the client to simplify application scaling "
4. " Front End Performance. Part 1 - Concatenation, Compression, Caching "
5. " We are writing a server that does not crash under load "
6. " Front End Performance. Part 2 - Caching Dynamic Content Using etagify "
7. " Taming web application configurations using node-convict "
8. " Front End Performance. Part 3 - Font Optimization "
9. " Localizing Node.js. Applications Part 1 "
10. " Localizing Node.js. Applications Part 2: Toolkit and Process "
11. " Localizing Node.js. Applications Part 3: Localization in Action "
12. " Awsbox - PaaS Infrastructure for Deploying Node.js Applications on Amazon Cloud "

---

Why bother?

You may ask, why monitor memory leaks at all? Are there really more important matters? Why not just restart the process from time to time, or just add memory to the server? There are three reasons why eliminating leaks is still important:

1. You may not be very worried about memory leaks, but this cannot be said for V8 (the JavaScript engine that Node runs on). The more memory is used, the more active the garbage collector works, slowing down your application. So in Node, leaks directly harm performance.
2. Leaks can lead to other problems. Leaking code can block limited resources. You may run out of file descriptors or you may not be able to open another connection to the database. Such problems can occur long before the memory runs out, but your application will crash no worse.
3. Sooner or later your application will crash. And this is likely to happen during an influx of visitors. Everyone will laugh at you and they will write nasty things about you on Hacker News.

Where does the sound of falling drops come from?

There are many places in a complex application where leaks can occur. Probably the most famous and sore spot - closures. Since closures can store variable references from their scope for a long time, they become the most frequent leak point.

Some leaks can be detected sooner or later, simply by looking for them in the code, but in the asynchronous world of Node, we constantly create many closures in the form of callbacks. And if they work out more slowly than they are created, memory fragments of each callback are piled up so that code that does not look current at all behaves like the current one. Tracking such leaks is much more difficult.

An application may also leak due to an error in someone else's code on which it depends. Sometimes you can find a place in your program that causes a leak, but it happens that you just have to look at your perfectly debugged code, bewildered, wondering how it can leak at all?

It was such leaks, difficult to track, that created the need for node-memwatch. Legend has it that a long time ago, a few months ago, our programmer Lloyd Hilayel locked himself in a closet for two days, trying to track down a leak that appeared only under very high load (by the way, he is the author of the next article in this series devoted to just load testing )

After two days of searching, he discovered that the desired error was in the Node core: event handlers inhttp.ClientRequestthey didn’t free the memory properly (the patch to fix this error added only two small, but very important characters to the Node.js code). The torment suffered forced Lloyd to write a tool that would help to search for such leaks.

Leak detection tools

There is already a good and constantly growing selection of tools for finding leaks in Node.js applications. Here is some of them:

• node-mtrace Jimb Esser, which uses the GCC mtrace utility to profile heap usage;
• Dave Pacheco's node-heap-dump , which takes a snapshot of the V8 heap and writes it to a huge JSON file. Includes tools for researching and parsing this snapshot using JavaScript;
• Danny Coates's v8-profiler and node-inspector , based on the V8 profiler and WebKit Web Inspector debugger;
• Felix Geisendorfer’s Node memory leak guide , which briefly and clearly talks about the basics of working with v8-profiler and node-inspector;
• platform Joyent SmartOS, which provides an arsenal of tools to find and fix leaks in Node.js .

We like these and many other tools, but not one of them was perfect for our environment. Web Inspector is great for applications in development, but difficult to use on a combat, running application, especially when many servers and processes are used. Therefore, with its help it is difficult to catch errors that do not appear immediately and only under heavy load. Tools like dtrace or libumem are great too, but they don’t work on all operating systems.

Meet: node-memwatch

We needed a cross-platform debugging library that did not require any additional tools, with which we could find places and causes of memory leaks. And so we wrote node-memwatch .

It provides three things:

• Event 'leak'
  

  memwatch.on('leak', function(info) {
  // Взглянуть на info, и выяснить, есть ли утечка
  });
  

  
  
• Event 'stats'
  

  var memwatch = require('memwatch');
  memwatch.on('stats', function(stats) {
  // изучить статистику использования памяти после сборки мусора
  });
  

  
  
• Class tracking heap diff:
  

  var hd = new memwatch.HeapDiff();
  // код приложения ...
  var diff = hd.end();
  

  
  

In addition, there is a function for forcing the garbage collector, which may be useful in testing. That is, after all, four things, not three.

var stats = memwatch.gc();

memwatch.on ('stats', ...): heap statistics after garbage collection

node-memwatch can display statistics on heap usage immediately after the garbage collector finishes work, before memory is allocated to any new objects. A hook is used for this V8::AddGCEpilogueCallback.

The statistics include the fields:

• usage_trend
• current_base
• estimated_base
• num_full_gc
• num_inc_gc
• heap_compactions
• min
• max

Here is an example that shows how the memory usage statistics of a leaking application look like. The zigzag green line shows the memory usage by data process.memoryUsage(), and the red line shows the value current_basefrom the node-memwatch statistics. Bottom left are additional data.



Notice how often incremental garbage collection occurs. This is an alarming sign indicating that the V8 has to sweat while clearing memory.

memwatch.on ('leak', ...): memory usage dynamics

We use simple heuristics to warn of a possible leak. If after five consecutive garbage collections the memory usage increases, the 'leak' event is triggered. Information about a possible leak is displayed in a convenient readable form:

{ start: Fri, 29 Jun 2012 14:12:13 GMT,
  end: Fri, 29 Jun 2012 14:12:33 GMT,
  growth: 67984,
  reason: 'heap growth over 5 consecutive GCs (20s) - 11.67 mb/hr' }

memwatch.HeapDiff (): find leaks

Finally, node-memwatch can compare heap snapshots, including the names of the number of objects taking up memory. Diff helps you find intruders.

var hd = new memwatch.HeapDiff();
// Код приложения ...
var diff = hd.end();

The contents of the object difflook like this:

{
  "before": {
    "nodes": 11625,
    "size_bytes": 1869904,
    "size": "1.78 mb"
  },
  "after": {
    "nodes": 21435,
    "size_bytes": 2119136,
    "size": "2.02 mb"
  },
  "change": {
    "size_bytes": 249232,
    "size": "243.39 kb",
    "freed_nodes": 197,
    "allocated_nodes": 10007,
    "details": [
      {
        "what": "Array",
        "size_bytes": 66688,
        "size": "65.13 kb",
        "+": 4,
        "-": 78
      },
      {
        "what": "Code",
        "size_bytes": -55296,
        "size": "-54 kb",
        "+": 1,
        "-": 57
      },
      {
        "what": "LeakingClass",
        "size_bytes": 239952,
        "size": "234.33 kb",
        "+": 9998,
        "-": 0
      },
      {
        "what": "String",
        "size_bytes": -2120,
        "size": "-2.07 kb",
        "+": 3,
        "-": 62
      }
    ]
  }
}

HeapDiffcalls the garbage collector before collecting statistics so that there is not too much unnecessary nonsense in the data. However, the event 'stats'does not occur, so you can safely call HeapDiffinside the handler 'stats'.

Here is an example of statistics in which we added a list of objects that occupy the most space on the heap

What's next?

Features of node-memwatch:

• accurate tracking of memory usage;
• notifications of possible leaks;
• tools for getting diff heap;
• cross-platform;
• lack of dependencies on other tools.

We want more. In particular, we want node-memwatch to be able to provide specific data for flowing objects - variable names, array indices, closure code chunks.

We hope that node-memwatch will be useful to you when debugging leaking Node.js applications, you fork it and take part in improving it.

---