June 3, 2019 at 12:30Node.js projects in which it is better not to use lock files
- Transfer
The author of the material, the translation of which we publish today, says that one of the problems that programmers have to deal with is that their code works for them, and it gives errors to someone else. This problem, perhaps one of the most common, arises due to the fact that different dependencies that the program uses are installed in the systems of the creator and user of the program. To combat this phenomenon, so-called lock files exist in the yarn and npm package managers . They contain information about the exact versions of the dependencies. The mechanism is useful, but if someone is developing a package that is planned to be published in npm, he better not use lock files. This material is dedicated to the story of why this is so.
Lock files are extremely useful when developing Node.js applications like web servers. However, if you are talking about creating a library or command line tool with the goal of publishing in npm, you need to know that lock files in npm are not published. This means that if these files are used during development, then the creator of the npm package, and those who use this package, will use different versions of the dependencies.
The lock file describes the complete dependency tree in the form that it acquired during the work on the project. This description also includes nested dependencies. The file contains information about specific versions of the packages used. In the npm package manager, such files are called
Here's what the file might look like
Here is an example file
Both of these files contain some critical dependency information:
If all the dependencies are listed in the lock-file, why then do they add information about them to
The purpose of a
Suppose a command was executed to install the dependencies of a certain project.
If you look at the npm semantic versioning documentation , you can see that the icon
If npm or yarn find the corresponding lock file, they will install packages based on this file, and not on
This is extremely useful if you are developing a project like a web application or server, since in a CI environment you need to simulate user behavior. As a result, if we include a lock-file in the project repository (for example, created using git tools), we can be sure that every developer, every server, every code building system and every CI environment uses the same versions dependencies.
Why not do the same when publishing libraries or other software tools in the npm registry? Before answering this question, we need to talk about how the process of publishing packages works.
Some developers believe that what is published in npm is exactly what is stored in the git repository, or what the project turns into after completion of work on it. This is actually not the case. When publishing a package, npm finds out which files to publish by accessing the key
After that, npm takes all the appropriate files and packs them into a file
Results of the npm pack --dry-run command
Then the resulting file is
As a result, it turns out that if someone installs someone else's package, the file
This can, by unlucky coincidence, lead to the problem that we spoke about at the very beginning. In the developer's system, the code works fine, and in other systems it produces errors. But the fact is that the project developer and those who use the project use different versions of the packages. How to fix it?
First you need to prevent the inclusion of lock files in the project repository. When using git, you need to include the
The yarn documentation says that
You
When working with yarn, you can use the command
However, the fact that we got rid of the file
In general, this is the same file as
In order to automate this operation, the command
It is worth noting that this technique is recommended to be used responsibly. By creating shrinkwrap files, you commit specific versions of the dependencies. On the one hand, this is useful for ensuring the stable operation of the project, on the other, it can prevent users from installing critical patches, which, otherwise, would be done automatically. In fact, npm strongly recommends not using shrinkwrap files when developing libraries, limiting their use to something like CI systems.
Unfortunately, with all the wealth of information about dependency management in the npm documentation , it is sometimes difficult to navigate this information. If you want to know what exactly is installed during the installation of dependencies or is packed before sending the package to npm, you can use the flag with different commands
Here are some similar commands:
A lot of what we talked about here relies on the specifics of performing various operations using npm. We are talking about packaging, publishing, installing packages, working with dependencies. And given the fact that npm is constantly evolving, we can say that all this can change in the future. In addition, the possibility of practical application of the recommendations given here depends on how the package developer perceives the problem of using different versions of dependencies in different environments.
Hopefully this material has helped you better understand how the npm dependency ecosystem works. If you want to delve even deeper into this question - here you can read about the differences between the
Dear readers! Do you use the npm-shrinkwrap.json file in your projects?
The most important thing in a nutshell
Lock files are extremely useful when developing Node.js applications like web servers. However, if you are talking about creating a library or command line tool with the goal of publishing in npm, you need to know that lock files in npm are not published. This means that if these files are used during development, then the creator of the npm package, and those who use this package, will use different versions of the dependencies.
What is a lock file?
The lock file describes the complete dependency tree in the form that it acquired during the work on the project. This description also includes nested dependencies. The file contains information about specific versions of the packages used. In the npm package manager, such files are called
package-lock.json, in yarn - yarn.lock. In both managers, these files are located in the same folder as package.json. Here's what the file might look like
package-lock.json.{
"name": "lockfile-demo",
"version": "1.0.0",
"lockfileVersion": 1,
"requires": true,
"dependencies": {
"ansi-styles": {
"version": "3.2.1",
"resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-3.2.1.tgz",
"integrity": "sha512-VT0ZI6kZRdTh8YyJw3SMbYm/u+NqfsAxEpWO0Pf9sq8/e94WxxOpPKx9FR1FlyCtOVDNOQ+8ntlqFxiRc+r5qA==",
"requires": {
"color-convert": "^1.9.0"
}
},
"chalk": {
"version": "2.4.2",
"resolved": "https://registry.npmjs.org/chalk/-/chalk-2.4.2.tgz",
"integrity": "sha512-Mti+f9lpJNcwF4tWV8/OrTTtF1gZi+f8FqlyAdouralcFWFQWF2+NgCHShjkCb+IFBLq9buZwE1xckQU4peSuQ==",
"requires": {
"ansi-styles": "^3.2.1",
"escape-string-regexp": "^1.0.5",
"supports-color": "^5.3.0"
}
}
}
}Here is an example file
yarn.lock. It is not designed in the same way package-lock.json, but contains similar data.# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1
ansi-styles@^3.2.1:
version "3.2.1"
resolved "https://registry.yarnpkg.com/ansi-styles/-/ansi-styles-3.2.1.tgz#41fbb20243e50b12be0f04b8dedbf07520ce841d"
integrity sha512-VT0ZI6kZRdTh8YyJw3SMbYm/u+NqfsAxEpWO0Pf9sq8/e94WxxOpPKx9FR1FlyCtOVDNOQ+8ntlqFxiRc+r5qA==
dependencies:
color-convert "^1.9.0"
chalk@^2.4.2:
version "2.4.2"
resolved "https://registry.yarnpkg.com/chalk/-/chalk-2.4.2.tgz#cd42541677a54333cf541a49108c1432b44c9424"
integrity sha512-Mti+f9lpJNcwF4tWV8/OrTTtF1gZi+f8FqlyAdouralcFWFQWF2+NgCHShjkCb+IFBLq9buZwE1xckQU4peSuQ==
dependencies:
ansi-styles "^3.2.1"
escape-string-regexp "^1.0.5"
supports-color "^5.3.0"Both of these files contain some critical dependency information:
- The exact version of each installed dependency.
- Dependency information for each dependency.
- Information about the downloaded package, including the checksum used to verify the integrity of the package.
If all the dependencies are listed in the lock-file, why then do they add information about them to
package.json? Why are two files needed?Comparison of package.json and lock files
The purpose of a
dependenciesfile field package.jsonis to show the project dependencies that must be installed for it to work properly. But this does not include information about the dependencies of these dependencies. Dependency information may include exact package versions or a certain range of versions specified in accordance with the rules of semantic versioning . When using the npm or yarn range, the most suitable version of the package is selected. Suppose a command was executed to install the dependencies of a certain project.
npm install. During the installation process, npm picked up suitable packages. If you run this command again, after a while, and if new versions of dependencies are released during this time, it may well happen that the second time other versions of the packages used in the project are loaded. For example, if a dependency is established, like twiliousing the command npm install twilio, then the following entry may appear in the dependenciesfile section package.json:{
"dependencies": {
"twilio": "^3.30.3"
}
}If you look at the npm semantic versioning documentation , you can see that the icon
^indicates that any version of the package is suitable, the number of which is greater than or equal to 3.30.3 and less than 4.0.0. As a result, if there is no lock file in the project and a new version of the package is released, the command will npm installeither yarn installautomatically install this new version of the package. Information in package.jsonthis case will not be updated. When using lock files, everything looks different. If npm or yarn find the corresponding lock file, they will install packages based on this file, and not on
package.json. This is especially useful, for example, when using Continuous Integration (CI) systems on platforms on which you need to ensure uniform operation of code and tests in an environment whose characteristics are known in advance. In such cases, you can use special commands or flags when invoking the appropriate package managers:npm ci # установит именно то, что перечислено в package-lock.json
yarn install --frozen-lock-file # установит то, что перечислено в yarn.lock, не обновляя этот файлThis is extremely useful if you are developing a project like a web application or server, since in a CI environment you need to simulate user behavior. As a result, if we include a lock-file in the project repository (for example, created using git tools), we can be sure that every developer, every server, every code building system and every CI environment uses the same versions dependencies.
Why not do the same when publishing libraries or other software tools in the npm registry? Before answering this question, we need to talk about how the process of publishing packages works.
Package Publishing Process
Some developers believe that what is published in npm is exactly what is stored in the git repository, or what the project turns into after completion of work on it. This is actually not the case. When publishing a package, npm finds out which files to publish by accessing the key
filesin the file package.jsonand the file .npmignore. If none of this can be detected, the file is used .gitignore. In addition, some files are always published, and some are never published. You can find out what these files are here . For example, npm always ignores a folder .git. After that, npm takes all the appropriate files and packs them into a file
tarballusing the commandnpm pack. If you want to look at what exactly is packed into such a file, you can execute a command in the project folder npm pack --dry-runand look at the list of materials in the console.Results of the npm pack --dry-run command
Then the resulting file is
tarballuploaded to the npm registry. When you run the command, npm pack --dry-runyou can pay attention to the fact that if there is a file in the project package-lock.json, it is not included in the tarball file. This is due to the fact that this file, in accordance with npm rules , is always ignored. As a result, it turns out that if someone installs someone else's package, the file
package-lock.jsonwill not participate in this. What is in this file that the package developer has will not be taken into account when installing the package by someone else.This can, by unlucky coincidence, lead to the problem that we spoke about at the very beginning. In the developer's system, the code works fine, and in other systems it produces errors. But the fact is that the project developer and those who use the project use different versions of the packages. How to fix it?
Refusal of lock-files and use of technology shrinkwrap
First you need to prevent the inclusion of lock files in the project repository. When using git, you need to include the
.gitignorefollowing in the project file :yarn.lock
package-lock.jsonThe yarn documentation says that
yarn.lockyou need to add to the repository even if it comes to developing the library that you plan to publish. But if you want you and your library users to work with the same code, I would recommend including it yarn.lockin the file .gitignore. You
package-lock.jsoncan disable automatic file creation by adding a file .npmrcwith the following contents to the project folder :package-lock=falseWhen working with yarn, you can use the command
yarn install --no-lockfilethat allows you to disable the reading of the file yarn.lock. However, the fact that we got rid of the file
package-lock.jsondoes not mean that we cannot capture information about dependencies and nested dependencies. There is another file called npm-shrinkwrap.json. In general, this is the same file as
package-lock.jsonit is created by the team npm shrinkwrap. This file gets into the npm registry when the package is published. In order to automate this operation, the command
npm shrinkwrapcan be added to the file script description sectionpackage.jsonin the form of a prepack script. You can achieve the same effect using the git commit hook. As a result, you can be sure that in your development environment, in your CI-system, and the users of your project use the same dependencies. It is worth noting that this technique is recommended to be used responsibly. By creating shrinkwrap files, you commit specific versions of the dependencies. On the one hand, this is useful for ensuring the stable operation of the project, on the other, it can prevent users from installing critical patches, which, otherwise, would be done automatically. In fact, npm strongly recommends not using shrinkwrap files when developing libraries, limiting their use to something like CI systems.
Finding package and dependency information
Unfortunately, with all the wealth of information about dependency management in the npm documentation , it is sometimes difficult to navigate this information. If you want to know what exactly is installed during the installation of dependencies or is packed before sending the package to npm, you can use the flag with different commands
--dry-run. The use of this flag leads to the fact that the team does not affect the system. For example, the command npm install --dry-rundoes not actually install dependencies, and the command npm publish --dry-rundoes not start the package publishing process. Here are some similar commands:
npm ci --dry-run # имитирует установку, основываясь на package-lock.json или на npm-shrinkwrap.json
npm pack --dry-run # выводит сведения обо всех файлах, которые были бы включены в пакет
npm install --verbose --dry-run # имитирует процесс установки пакета с выводом подробных сведений об этом процессе Summary
A lot of what we talked about here relies on the specifics of performing various operations using npm. We are talking about packaging, publishing, installing packages, working with dependencies. And given the fact that npm is constantly evolving, we can say that all this can change in the future. In addition, the possibility of practical application of the recommendations given here depends on how the package developer perceives the problem of using different versions of dependencies in different environments.
Hopefully this material has helped you better understand how the npm dependency ecosystem works. If you want to delve even deeper into this question - here you can read about the differences between the
npm ciand commands npm install. HereYou can find out exactly what gets into the files package-lock.jsonand npm-shrinkwrap.json. Here is the npm documentation page where you can find out which project files are included and not included in packages. Dear readers! Do you use the npm-shrinkwrap.json file in your projects?