Diarrhea for your backend on Node.JS - reducing build weight

    Surely you often noticed how much garbage lies inside node modules. These are tests, benchmarks, readme files, licenses, time script, and another crazy amount of garbage that can be more or less safely removed. What we actually will do in this post.
    I already mentioned the last few publications about the weight of the node module, so here is another one that generally reflects the current situation. As a soundtrack to the post, Little Big, “Life in da trash”, is recommended.



    Start


    So, once again, by accidentally going inside node modules, I was sad, and I started by writing several scripts for the experiment with my hands that I ran through my project. They looked something like this:


    find ./ -iname "tests" -exec rm -rf {} \;
    find ./node_modules -iname "test" -exec rm -rf {} \;
    find ./node_modules -iname "*.gif" -exec rm -rf {} \;
    find ./node_modules -iname "*.jpg" -exec rm -rf {} \;
    find ./node_modules -iname "*.jpeg" -exec rm -rf {} \;
    find ./node_modules -iname "*.png" -exec rm -rf {} \;
    find ./ -iname "*.md" -exec rm -rf {} \;
    find ./ -iname "*.log" -exec rm -rf {} \;
    find ./ -iname "LICENSE*" -exec rm -rf {} \;
    find ./ -iname "README*" -exec rm -rf {} \;
    find ./ -iname "LICENCE*" -exec rm -rf {} \;
    find ./ -iname "AUTHORS*" -exec rm -rf {} \;
    find ./ -iname "NOTICE*" -exec rm -rf {} \;
    find ./ -iname "changelog*" -exec rm -rf {} \;
    find ./ -iname ".travis.yml" -exec rm -rf {} \;
    find ./ -iname ".coveralls.yml" -exec rm -rf {} \;
    find ./ -iname ".npmignore" -exec rm -rf {} \;
    find ./ -iname ".gitignore" -exec rm -rf {} \;
    find ./ -iname ".jshintrc" -exec rm -rf {} \;
    find ./ -iname ".eslintrc" -exec rm -rf {} \;
    find ./ -iname ".jscs.json" -exec rm -rf {} \;
    find ./ -iname ".editorconfig" -exec rm -rf {} \;
    find ./ -iname ".vs" -exec rm -rf {} \;
    find ./ -iname ".babelrc" -exec rm -rf {} \;

    In general, this was already not bad, and gave some savings, but I wanted some more elegant and general solution. And quickly enough, I found a module that does exactly what I wanted - ModClean .


    Modclean


    The module has a bunch of settings, and also lists of “extra files” in the form of plugins that connect as npm packages. By default, there is such a list.


    It has three levels of files for deleting security - I highly recommend looking at the list and see what might cause you problems. For example, I didn’t understand why the “semver” was included in the list ...


    The only negative - for some reason, the author of the module did not think that you might also want to remove all the garbage before assembling in your own project - for example, you absolutely do not need test files on the production ...
    True, it turned out that it was easy to bypass this by specifying as a directory modules project directory:


    modclean --modules-dir .

    results


    It is clear that the results can be very different on different projects, but I got from 10% to 20% savings. In advanced cases, there may be more. This may not seem like the biggest figure, but this is a 20% reduction in storage space required and a 20% faster roll-out of projects from literally nothing - so why not?
    For those who are curious, the module’s site also has removal benchmarks, but it’s better to just try it on yourself.


    Diarrhea


    On this positive note, I would like to say that I have solved my problem, and I recommend everyone to use this beautiful module. But you saw the word “diarrhea” in the title, and not “modclean”, right? This means that there is one more nuance. The nuance is quite thick in the literal sense of the word - it turned out that the module has 123 transitive dependencies and it weighs 4MB. That, in general, is already considered more or less the standard for node modules, but cannot but deliver mental pain.


    At the same time, the wildly popular update-notifier module occupies 1.2 megabytes , all he does is draw such a frame in case updates are available:

    1 megabyte for this, Karl! As a Node.JS developer, I want to fail in shame.


    And most of the rest of the place is occupied by the clui module , which contains all sorts of decorations for working with the terminal - of which only a couple of functions are used.


    In general, I could not bear this, and made my assembly - without blackjack and everything else. I completely cut out the notification about updates from there, and I collected the necessary functions from thick modules with a large number of dependencies into a minified bundle using webpack. In general, it was possible to play a little more and even reduce the weight of the module - but I got to the weight of 700 kilobytes and so far calmed down. I also added the --root option, which allows you to more clearly clean not the dependencies, but the root of your project.


    I threw Issues for all these reasons into modclean, as well as made a pull request there, and if everything is approved, then I will fix this fork in favor of the main module - but so far there is a feeling that modclean is abandoned - so if there are good ideas, throw them to me.


    You can install my version, as always, from npm .


    Warning


    Double-check that you have saved everything you need in the repository - diarrhea can delete unnecessary files, necessary files, teeth, and kill kittens. I take off all responsibility for this.


    Integration in CI


    It is best to use diarrhea like this:


    1. First, we remove all the garbage from node modules;
    2. Then we run unit tests to make sure that they did not take anything superfluous;
    3. Then we remove the garbage from the main project, including the tests themselves;
    4. We do npm prune --production;
    5. We collect the artifact.

    Todo


    The module was made in haste for personal use, so in the future I would like to add the following:


    • Tests. Oddly enough, in modClean they are not at all;
    • Compatibility with a node to the 8th version. Since there is already webpack, it is not difficult;
    • Cut out a few more dependencies that don't make much sense;
    • Minimal version without terminal decorations for CI

    conclusions


    • Once again I’m afraid of what terrible things a low threshold for entering my favorite ecosystem does. Something definitely needs to be done with this, and I hope that it will be somehow solved at the npm level. But right now, I see that this beautiful update-notifier is in the dependencies of npm itself, which is very upsetting. This miserable notifier has 2 million downloads per week! 2 million megabytes (almost 2 terabytes) spent on a fucking frame with an update message!
    • Always check the weight of your addictions, even on the backend;
    • Clean your addictions;
    • Oddly enough, little by little I come to the idea that backend dependencies also need to be assembled into a common bundle. What is now being done quite easily and naturally by any tool you choose, and does not add pain when debugging - after all, you can generate surrogates. The only minus is that far from every package can be included in the bundle, I regularly encounter some problems in this, for the most part related to the fact that everyone likes to connect dynamic dependencies in any way. For example, the same update-notifier cannot be thrust into a bundle.

    Well, that’s all.


    If you were interested, you may also like my other articles on this topic:



    I can also tell you about various cases of persistent use of Node.JS on the backend, participate in the survey if you are interested.

    Only registered users can participate in the survey. Please come in.

    What else to tell?

    • 11.6% Why and how to use mocha in multiple threads 5
    • 27.9% How to implement linting in a project where 4000 files contain a huge number of errors 12
    • 6.9% Why and how to load logs from kibana into MySQL or Slack bot 3
    • 41.8% Why build backend modules in bundle 18
    • 11.6% About nothing, stop this heresy already and go to Go 5

    Also popular now: