How to organize your own repository of Node.js modules with blackjack and versioning

The ISPsystem currently has three front-end teams developing three major projects: ISPmanager for managing web servers, VMmanager for working with virtualization, and BILLmanager for automating hosters' business. The teams work simultaneously, in the mode of short deadlines, so it’s impossible to do without optimization. To save time, we apply uniform decisions and we carry out the general components in separate projects. Such projects have their own repositories that are supported by members of all teams. On the device of these repositories, as well as work with them, and this article will be.

How repositories of common projects are arranged

We use our own server with GitLab to store remote repositories. For us it was important to keep the usual working environment and to be able to work with common modules in the process of their development. Therefore, we refused to publish in the private repositories of npmjs.com . Benefit Node.js modules can be installed not only from NPM, but also from other sources , including git-repositories.

We write in TypeScript, which is subsequently compiled into JavaScript for later use. But nowadays, unless a lazy fronder does not compile your JavaScript. Therefore, we need different repositories for the source code and the compiled project.

After going through the thorns of long discussions, we have developed the following concept. There should be two separate repositories for the source and for the compiled version of the module. And the second repository should be a mirror of the first.

This means that when developing, a feature should be published until the moment of release in a branch with the exact same name as that of the branch in which the development is being carried out. Thus, we have the opportunity to use the experimental version of the module, installing it from a specific branch. The one in which we are developing is very convenient to test it in action.

Plus, for each publication we create a label that preserves the status of the project. The label name corresponds to the version specified in package.json. When installing from a git repository, the label is indicated after the grid, for example:

npm install git+ssh://[url репозитория]#1.0.0

Thus, we can fix the used version of the module and not worry that someone will change something.

Labels are also created for unstable versions, but an abbreviated hash of the commit is added to them in the source repository from which the publication was made. Here is an example of such a label:

1.0.0_e5541dc1

This approach allows you to achieve unique tags, as well as link them to the source repository.

Since we started talking about stable and unstable versions of the module, here’s how we distinguish them: if the publication is run from the master or develop branch, the version is stable, otherwise it’s not.

How is work with common projects organized?

All our arrangements would not make sense if we could not automate them. In particular, automate the process of publishing. Below, I will show how work is organized with one of the common modules - a utility for testing user scripts.

Using the puppeteer library, this utility prepares the Chromium browser for use in docker containers and runs tests using Mocha . Members of all teams can modify the utility without fear of breaking something from each other.

The following command is written in the package.json utility file for testing:

"publish:git": "ts-node ./scripts/publish.ts"

It runs the next lying script:


In turn, this code through the Node.js module child_process executes all the necessary commands.

Here are the main stages of his work:

1. Check for uncommitted changes

const isDiff = !!spawnSync('git', ['diff'], getSpawnOptions(rootDir, 'pipe')).stdout.toString().trim();

Here we check the output of the git diff command . It is not good if the publication includes changes that are not in the source code. In addition, it will break the link between unstable versions and commits.

2. Build utility

const build = spawnSync('npm', ['run', 'build'], getSpawnOptions(rootDir));

The build constant is the result of the build. If everything went well, the status parameter will be 0. Otherwise, nothing will be published.

3. Deploying the compiled versions repository

The entire publishing process is nothing more than sending changes to a specific repository. Therefore, the script creates in our project a temporary directory in which it initializes the git repository and links it to the remote assembly repository.

/* временная директория для разворачивания репозитория сборок */
const tempDir = join(rootDir, 'temp');
if (existsSync(tempDir)) {
  spawnSync('rm', ['-rf', 'temp'], getSpawnOptions(rootDir));
}
mkdirSync(tempDir);
/* получение параметров из package.json */
const { name, version, repository } = require(join(rootDir, 'package.json'));
const originUrl = repository.url.replace(`${name}-source`, name);
spawnSync('git', ['init'], getSpawnOptions(tempDir));
spawnSync('git', ['remote', 'add', 'origin', originUrl], getSpawnOptions(tempDir));

This is a standard process using git init and git remote .

4. Generating the name of the label

To begin, we find out the name of the branch from which we are publishing, using the git symbolic-ref command . And we set the name of the branch to which the changes will be uploaded (there is no develop branch in the assembly repository).

/* имя текущей ветки из репозитория исходников модуля */const branch = spawnSync(
  'git',
  ['symbolic-ref', '--short', 'HEAD'],
  getSpawnOptions(rootDir, 'pipe')
).stdout.toString().trim();
/* имя ветки в репозитории сборок */const buildBranch = branch === 'develop' ? 'master' : branch;

Using the git rev-parse command , we get the shortened hash of the last commit in the branch we are in. You may need it to generate the label name of the unstable version.

<source lang="typescript">/* сокращенный хеш последнего коммита в репозитории исходников,
используемый при формировании метки нестабильной версии */
const shortSHA = spawnSync(
  'git',
  ['rev-parse', '--short', 'HEAD'],
  getSpawnOptions(rootDir, 'pipe')
).stdout.toString().trim();

Well, actually make up the name of the label.

/* метка */const tag = buildBranch === 'master' ? version : `${version}_${shortSHA}`;

5. Check for the absence of the exact same label in the remote repository.

/* проверка существования сформированной метки в репозитории сборок */const isTagExists = !!spawnSync(
  'git',
  ['ls-remote', 'origin', `refs/tags/${tag}`],
  getSpawnOptions(tempDir, 'pipe')
).stdout.toString().trim();

If a similar label was created earlier, the result of the git ls-remote command will not be empty. The same version should be published only once.

6. Creating the appropriate branch in the build repository

As I said earlier, the repository of compiled versions of the utility is a mirror of the repository with the sources. Therefore, if the publication is not from a master or develop branch, we must create a corresponding branch in the assembly repository. Well, or at least make sure it exists

/* проверка существования ветки в репозитории сборок */const isBranchExits = !!spawnSync(
  'git',
  ['ls-remote', '--exit-code', 'origin', buildBranch],
  getSpawnOptions(tempDir, 'pipe')
).stdout.toString().trim();
if (isBranchExits) {
  /* переход в целевую ветку */
  spawnSync('git', ['fetch', 'origin', buildBranch], getSpawnOptions(tempDir));
  spawnSync('git', ['checkout', buildBranch], getSpawnOptions(tempDir));
} else {
  /* переход в ветку master */
  spawnSync('git', ['fetch', 'origin', 'master'], getSpawnOptions(tempDir));
  spawnSync('git', ['checkout', 'master'], getSpawnOptions(tempDir));
  /* создание целевой ветки */
  spawnSync('git', ['checkout', '-b', buildBranch], getSpawnOptions(tempDir));
  /* создание начального коммита */
  spawnSync('git', ['commit', '--allow-empty', '-m', '"Initial commit"'], getSpawnOptions(tempDir));
}

If the branch was absent earlier, we initialize with an empty commit using the --allow-empty flag .

7. Preparing files

First you need to delete everything that could be in the expanded repository. After all, if we use a previously existing branch, it contains the previous version of the utility.

/* удаление старых файлов сборки */
spawnSync(
  'rm',
  ['-rf', 'lib', 'package.json', 'package-lock.json', 'README.md'],
  getSpawnOptions(tempDir)
);

Next, transfer the updated files required for publication, and add them to the repository index.

/* копирование файлов сборки */
spawnSync('cp', ['-r', 'lib', 'temp/lib'], getSpawnOptions(rootDir));
spawnSync('cp', ['package.json', 'temp/package.json'], getSpawnOptions(rootDir));
spawnSync('cp', ['package-lock.json', 'temp/package-lock.json'], getSpawnOptions(rootDir));
spawnSync('cp', ['README.md', 'temp/README.md'], getSpawnOptions(rootDir));
/* индексация файлов сборки */
spawnSync('git', ['add', '--all'], getSpawnOptions(tempDir));

After this manipulation, git recognizes well the changes made by file lines. This way we get a consistent change history even in the compiled versions repository.

8. Commit and submit changes

As a commit message in the build repository, we use the label name for stable versions. And for unstable, a commit message from the source repository. Thus supporting our idea of ​​storage mirror.

/* сообщение последнего коммита в репозитории исходников */const lastCommitMessage = spawnSync(
  'git',
  ['log', '--oneline', '-1'],
  getSpawnOptions(rootDir, 'pipe')
).stdout.toString().trim();
/* сообщение коммита в репозитории сборок */const message = buildBranch === 'master' ? version : lastCommitMessage;
/* создание коммита в репозитории сборок */
spawnSync('git', ['commit', '-m', `"${message}"`], getSpawnOptions(tempDir));
/* создание метки в репозитории сборок */
spawnSync('git', ['tag', tag], getSpawnOptions(tempDir));
/* отправка изменений в удаленный репозиторий */
spawnSync('git', ['push', 'origin', buildBranch], getSpawnOptions(tempDir));
spawnSync('git', ['push', '--tags'], getSpawnOptions(tempDir));

9. Delete temporary directory

spawnSync('rm', ['-rf', 'temp'], getSpawnOptions(rootDir));

Review of updates in shared projects

One of the most important processes after making changes to common projects becomes review. Despite the fact that the developed technology allows you to create completely isolated versions of modules, no one wants to have dozens of different versions of the same utility. Therefore, each of the common projects should follow a single path of development. It is worth negotiating between the teams.

A review of updates in shared projects is carried out by members of all teams as far as possible. This is a difficult process, as each team lives on its own sprint and has a different workload. Sometimes the transition to the new version may be delayed.

Here you can only recommend not to neglect and not to delay with this process.