Machine learning with Node.js using the Tensorflow.js library

Transfer

Hello everyone, colleagues!

Perhaps the fans of the Tensorflow library, who have already noticed this book in pre-order , also looked closely at the possibilities of machine and in-depth training in the browser, especially since the topic was not overlooked by Francois Chollet himself . We invite those interested in the cat section, where it is described how images are recognized using the Tensorflow.js library.

TensorFlow.js is a new version of the popular open source library that enriches the JavaScript language with deep learning capabilities. Developers can now define, train, and run models using the high-level library API .

Thanks to pre-trained models, developers can now easily solve such complex tasks as pattern recognition , music generation, or defining human poses in just a few lines of JavaScript.

Tensorflow.js started as a front-end browser library, but this year added experimental support to it.Node.js. Thus, TensorFlow.js can be used in JavaScript backend applications, which completely eliminates the need to resort to Python.

Reading about this library, I decided to try it on a simple task ...

Use TensorFlow.js to visually recognize patterns on images when using JavaScript from Node.js

Unfortunately, the documentation and code samples basically describe the use of this library in the browser. Project utilities designed to simplify the loading and use of pre-trained models at the time of this writing did not yet support Node.js. I had to spend a lot of time to thoroughly read the Typescript sources for this library.

However, after a few days of chiselling, I did it anyway ! Hooray!

Before turning to a detailed analysis of the code, let's talk about other implementations of the TensorFlow library.

TensorFlow

TensorFlow- It is a free software library for machine learning applications. TensorFlow can be used to create neural networks and implement other deep learning algorithms.

This library, released by Google in November 2015, was originally written in Python . In order to train and evaluate the models created in it, calculations on a CPU or a GPU are applied. Initially, this library was created to work on high-performance servers using resource-intensive GPUs.

Recent updates have made it possible to optimize this library and use it in environments with more limited resources — for example, on mobile devices and in web browsers.

TensorFlow Lite

Tensorflow LiteThe light version of this library for mobile and embedded systems was released in May 2017. Along with it, a new set of pre-trained deep models for problems related to pattern recognition is provided; This collection is called MobileNet . MobileNet models have been designed specifically to work effectively in environments with limited resources, such as mobile devices.

TensorFlow.js

Following Tensorflow Lite in March 2018, TensorFlow.js was announced . This version of the library is designed to work in the browser and is based on an earlier project called deeplearn.js. WebGL provides GPU library access. Developers use the JavaScript API to train, load, and run models.

Later, TensorFlow.js was extended to work with Node.js, for which the library supplement is used tfjs-node.

Importing existing models into TensorFlow.js

Ready-made models TensorFlow and Keras can be performed using the TensorFlow.js library. Before running the model, you need to translate into a new format with this tool . Pre-trained and transformed models for image classification, posturing and k-nearest-neighbor detection are available on Github .

Using TensorFlow.js with Node.js

Installing TensorFlow Libraries

TensorFlow.js can be installed from the NPM registry .

@tensorflow/tfjs- Core TensorFlow.js library
@tensorflow/tfjs-node- extension TensorFlow.js Node.js
@tensorflow/tfjs-node-gpu- Extension TensorFlow.js Node.js with support for GPU computing

npm install @tensorflow/tfjs @tensorflow/tfjs-node
// или...
npm install @tensorflow/tfjs @tensorflow/tfjs-node-gpu

Both extensions for Node.js use native dependencies that will be compiled on demand.

Loading TensorFlow

API libraries into JavaScript for Tensorflow is provided from the core library. Extension modules that support Node.js do not provide additional APIs.

const tf = require('@tensorflow/tfjs')
// Загружаем привязку (вычисления CPU)require('@tensorflow/tfjs-node')
// Или загружаем привязку (вычисления GPU)require('@tensorflow/tfjs-node-gpu')

Loading TensorFlow Models TensorFlow.js

provides the NPM ( tfjs-models) library, which makes it easy to load pre-trained and transformed models for classifying images , determining poses, and finding the k-nearest neighbors .

The MobileNet model for image classification is a deep neural network, trained to distinguish between 1000 different image classes .

The following code is used as an example in the project's README file for the model loading.

import * as mobilenet from'@tensorflow-models/mobilenet';
// загрузить модельconst model = await mobilenet.load();

One of the first problems I encountered was that this code does not work with Node.js.

Error: browserHTTPRequest is not supported outside the web browser.

After examining the source code , we see that the mobilenet library is a wrapper for the class tf.Model. When called, the method load()automatically loads the necessary model files located at the external HTTP address and instantiates the TensorFlow model.

The Node.js extension at the time of this writing did not yet support HTTP requests for dynamic model retrieval. It remained only to manually upload the model to the file system.

However, after reading the library source code, I found a workaround ...

Loading models from the file system

In the event that the MobileNet class is created manually, you can not call the loadmodule method , but overwrite the automatically generated variablepaththat contains the HTTP address of the model, replacing this address with the local path in the file system. After that, when the method is called loadin the class instance, the file system loader class will work ; in this case, we refuse to use the HTTP browser loader.

const path = "mobilenet/model.json"const mn = new mobilenet.MobileNet(1, 1);
mn.path = `file://${path}`await mn.load()

Cool, it works!

But where do the model files come from?

MobileNet Models TensorFlow.js

models consist of two types of files: a model configuration file stored in JSON format, and model weights stored in binary format. Model weights are often fragmented into multiple parts to optimize browser caching.

After reviewing the automatic load code for MobileNet models, we see that the models, their configurations and weight fragments are retrieved from the public container at the following address.

https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v${version}_${alpha}_${size}/

The template parameters in the URL describe the model versions listed here . The resulting classification accuracy is also displayed on the same page.

The source code states that only models of the MobileNet v1 version can be downloaded using the library tensorflow-models/mobilenet.

The HTTP retrieval code loads the file model.jsonfrom the storage location, and then recursively selects all model fragments with weights referenced. These are files in a format groupX-shard1of1.

Downloading models manually

If you need to save all model files in the file system, you can do this: extract the model configuration file, parse the syntax of all the weight files referenced in the configuration file, and then download each weight file manually.
I was going to use the MobileNet V1 module with an alpha value of 1.0 and an image size of 224 pixels . So I get the following URL for the model configuration file.

https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v1_1.0_224/model.json

Once this file is downloaded locally, you can use the tooljq to parse the names of all weight files.

$ cat model.json | jq -r ".weightsManifest[].paths[0]"
group1-shard1of1
group2-shard1of1
group3-shard1of1
...

Using the tool, sedyou can put in front of the name of each HTTP URL element to generate a URL for each weight file.

$ cat model.json | jq -r ".weightsManifest[].paths[0]" | sed 's/^/https:\/\/storage.googleapis.com\/tfjs-models\/tfjs\/mobilenet_v1_1.0_224\//'
https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v1_1.0_224/group1-shard1of1
https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v1_1.0_224/group2-shard1of1
https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v1_1.0_224/group3-shard1of1
...

The commands paralleland curlthen allow you to download all these files to my local directory.

cat model.json | jq -r ".weightsManifest[].paths[0]" | sed 's/^/https:\/\/storage.googleapis.com\/tfjs-models\/tfjs\/mobilenet_v1_1.0_224\//' |  parallel curl -O

Image Classification

This sample code , provided with TensorFlow.js, demonstrates how to return the result of image classification.

const img = document.getElementById('img');
// Классифицируем изображениеconst predictions = await model.classify(img);

This does not work in Node.js due to the lack of DOM support.

The methodclassify takes many DOM elements ( canvas, video, image), and automatically extracts and converts "kartinochnye" bytes of these elements in a class tf.Tensor3Dthat is used as input to the model. Alternatively, input information tf.Tensor3Dcan be transmitted directly.

I decided not to try to use an external package to simulate a DOM element manually, but found that it is tf.Tensor3Deasier to assemble manually .

Generating Tensor3D from the image.

Reading the source code of the method used to convert the DOM elements to the Tensor3D classes, we find that the following input parameters are used to generate the Tensor3D class.

const values = newInt32Array(image.height * image.width * numChannels);
// заполняем пикселы информацией пиксельных каналов, взятой с картинки const outShape = [image.height, image.width, numChannels];
const input = tf.tensor3d(values, outShape, 'int32');

pixelsIs a two-dimensional array of type (Int32Array)containing a sequential list of channel values for each pixel. numChannelsIs the number of channel values per pixel.

Creating input values for JPEG A

libraryjpeg-js is a JPEG encoder / decoder for Node.js written in pure JavaScript. Using this library, you can extract RGB values for each pixel.

const pixels = jpeg.decode(buffer, true);

As a result, we obtain a Uint8Array with four channel values ( RGBA) for each pixel ( width * height). The MobileNet model uses only three color channels for classification ( RGB), the alpha channel is ignored. This code converts a four-channel array into the correct three-channel version.

const numChannels = 3;
const numPixels = image.width * image.height;
const values = newInt32Array(numPixels * numChannels);
for (let i = 0; i < numPixels; i++) {
  for (let channel = 0; channel < numChannels; ++channel) {
    values[i * numChannels + channel] = pixels[i * 4 + channel];
  }
}

Input Value Requirements for MobileNet Models The MobileNet model

used here classifies images that are 224 pixels high and wide. Input tensors must contain floating-point values in the range from -1 to 1 for each of the three channel values of each pixel. Input values for images with a different dimension must be recalculated to the correct size before classification. In addition, pixel values received from a JPEG decoder are in the range of 0 - 255, and not -1 - 1. These values must also be converted before classification. TensorFlow.js has library methods that simplify this process, but, better yet, there is a special library that automatically solves this problem.

tfjs-models/mobilenet!

The developer can transfer input Tensor3D types int32, as well as various dimensions to the method classify, which prior to classification translates the input values into the correct format. That is, we do not have to do anything here. Super!

Obtaining forecasts

MobileNet models in Tensorflow are taught to recognize objects from 1000 of the most important classes from the ImageNet data set . At the output of the model, probabilistic values are given, which characterize the chances of finding these objects in the classified image.

A complete list of the trained classes for the model used is in this file .

Library tfjs-models/mobilenetoffers classifyclass methodMobileNetthat returns the top X most likely classes based on what is shown in the picture.

const predictions = await mn_model.classify(input, 10);

predictions - This is an array of X classes and probabilities in the following format.

	{
  className: 'panda',
  probability: 0.9993536472320557
}

Example

So, we figured out how to use the TensorFlow.js library and MobileNet models in Node.js, and now we will look at how this script classifies the image specified as a command line argument.

Source Code

Save this script file and package descriptor in local files.

{
	  "name": "tf-js",
	  "version": "1.0.0",
	  "main": "script.js",
	  "license": "MIT",
	  "dependencies": {
	    "@tensorflow-models/mobilenet": "^0.2.2",
	    "@tensorflow/tfjs": "^0.12.3",
	    "@tensorflow/tfjs-node": "^0.1.9",
	    "jpeg-js": "^0.3.4"
	  }
	}

const tf = require('@tensorflow/tfjs')
	const mobilenet = require('@tensorflow-models/mobilenet');
	require('@tensorflow/tfjs-node')
	const fs = require('fs');
	const jpeg = require('jpeg-js');
	const NUMBER_OF_CHANNELS = 3const readImage = path => {
	  const buf = fs.readFileSync(path)
	  const pixels = jpeg.decode(buf, true)
	  return pixels
	}
	const imageByteArray = (image, numChannels) => {
	  const pixels = image.data
	  const numPixels = image.width * image.height;
	  const values = newInt32Array(numPixels * numChannels);
	  for (let i = 0; i < numPixels; i++) {
	    for (let channel = 0; channel < numChannels; ++channel) {
	      values[i * numChannels + channel] = pixels[i * 4 + channel];
	    }
	  }
	  return values
	}
	const imageToInput = (image, numChannels) => {
	  const values = imageByteArray(image, numChannels)
	  const outShape = [image.height, image.width, numChannels];
	  const input = tf.tensor3d(values, outShape, 'int32');
	  return input
	}
	const loadModel = async path => {
	  const mn = new mobilenet.MobileNet(1, 1);
	  mn.path = `file://${path}`await mn.load()
	  return mn
	}
	const classify = async (model, path) => {
	  const image = readImage(path)
	  const input = imageToInput(image, NUMBER_OF_CHANNELS)
	  const  mn_model = await loadModel(model)
	  const predictions = await mn_model.classify(input)
	  console.log('classification results:', predictions)
	}
	if (process.argv.length !== 4) thrownewError('incorrect arguments: node script.js <MODEL> <IMAGE_FILE>')
	classify(process.argv[2], process.argv[3])

Testing

Download the model files to the mobilenet directory, following the instructions above.
Set project dependencies using NPM.

npm install

Download a sample JPEG file for classification.

wget http://bit.ly/2JYSal9 -O panda.jpg

Run a script, the arguments of which are the model file and the input image.

node script.js mobilenet/model.json panda.jpg

If everything worked correctly, then the following output should appear in the console.

	classification results: [ {
    className: 'giant panda, panda, panda bear, coon bear',
    probability: 0.9993536472320557
} ]

The image is correctly classified as containing a panda with a probability of 99.93%!

Conclusion

The TensorFlow.js library opens up deep learning opportunities for JavaScript developers. The use of pre-trained models with the TensorFlow.js library makes it easy to build new applications in JavaScript applications to solve complex machine learning tasks, with minimal effort and concise code.

The TensorFlow.js library was created especially for working in the browser, but now it interacts with Node.js, although not all tools and utilities support this new execution environment. Having fiddled with the library for several days, I learned to use it with MobileNet models for visual recognition of images from a local file.

Tags:

Machine learning with Node.js using the Tensorflow.js library

Also popular now: