Node.js Part 10 tutorial: standard modules, streams, databases, NODE_ENV

This material completes the Node.js tutorial translation series. Today we will talk about the modules os, events and http, discuss work with threads and databases, touch upon the use of Node.js in application development and production.

Node.js os module

The module osgives access to many functions that can be used to obtain information about the operating system and the hardware of the computer running Node.js. This is a standard module, it is not necessary to install it, to work with it from the code it is enough to connect it:

const os = require('os')

There are several useful features that, in particular, can be useful when working with files.

Thus, the property os.EOLallows you to find out which line separator is used in the system (end of line sign). On Linux and macOS \n, on Windows \r\n.

It should be noted that by mentioning “Linux and macOS” here, we are talking about POSIX-compatible platforms. For the sake of brevity, less popular platforms are not mentioned here.

The property os.constants.signalsprovides information about the constants that are used for the treatment of processes such as signaling SIGHUP, SIGKILLand so on. Here you can find details about them.

The property os.constants.errnocontains the constants that are used for error messages - like EADDRINUSE, EOVERFLOW.

Now consider the basic methods of the module os.

▍os.arch ()

This method returns a string identifying the architecture of the system, for example - arm, x64, arm64.

▍os.cpus ()

Returns information about the processors available in the system. For example, this information might look like this:

[ { model: 'Intel(R) Core(TM)2 Duo CPU     P8600  @ 2.40GHz',
    speed: 2400,
    times:
     { user: 281685380,
       nice: 0,
       sys: 187986530,
       idle: 685833750,
       irq: 0 } },
  { model: 'Intel(R) Core(TM)2 Duo CPU     P8600  @ 2.40GHz',
    speed: 2400,
    times:
     { user: 282348700,
       nice: 0,
       sys: 161800480,
       idle: 703509470,
       irq: 0 } } ]

▍os.endianness ()

Returns BEor LEdepending on which byte order (Big Engian or Little Endian) was used to compile the Node.js binary file.

▍os.freemem ()

Returns the amount of free system memory in bytes.

▍os.homedir ()

Returns the path to the current user's home directory. For example - '/Users/flavio'.

▍os.hostname ()

Returns the host name.

▍os.loadavg ()

Returns, as an array, data on average load values ​​calculated by the operating system. This information makes sense only in Linux and macOS. It can look like this:

[ 3.68798828125, 4.00244140625, 11.1181640625 ]

▍os.networkInterfaces ()

Returns information about the network interfaces available on the system. For example:

{ lo0:
   [ { address: '127.0.0.1',
       netmask: '255.0.0.0',
       family: 'IPv4',
       mac: 'fe:82:00:00:00:00',
       internal: true },
     { address: '::1',
       netmask: 'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff',
       family: 'IPv6',
       mac: 'fe:82:00:00:00:00',
       scopeid: 0,
       internal: true },
     { address: 'fe80::1',
       netmask: 'ffff:ffff:ffff:ffff::',
       family: 'IPv6',
       mac: 'fe:82:00:00:00:00',
       scopeid: 1,
       internal: true } ],
  en1:
   [ { address: 'fe82::9b:8282:d7e6:496e',
       netmask: 'ffff:ffff:ffff:ffff::',
       family: 'IPv6',
       mac: '06:00:00:02:0e:00',
       scopeid: 5,
       internal: false },
     { address: '192.168.1.38',
       netmask: '255.255.255.0',
       family: 'IPv4',
       mac: '06:00:00:02:0e:00',
       internal: false } ],
  utun0:
   [ { address: 'fe80::2513:72bc:f405:61d0',
       netmask: 'ffff:ffff:ffff:ffff::',
       family: 'IPv6',
       mac: 'fe:80:00:20:00:00',
       scopeid: 8,
       internal: false } ] }

▍os.platform ()

Returns information about the platform for which Node.js was compiled. Here are some of the possible return values:

• darwin
• freebsd
• linux
• openbsd
• win32

▍os.release ()

Returns a string identifying the operating system release number.

▍os.tmpdir ()

Returns the path to the directory specified in the system for storing temporary files.

▍os.totalmem ()

Returns the total amount of system memory in bytes.

▍os.type ()

Returns information that identifies the operating system. For example:

• Linux - Linux.
• Darwin - macOS.
• Windows_NT - Windows.

▍os.uptime ()

Returns the system time in seconds since the last restart.

Node.js events module

The module eventsprovides us with a class EventEmitterthat is designed to work with events on the Node.js platform. We already talked a little about this module in the seventh part of this series of materials. Here is the documentation for it. Here we consider the API of this module. Recall that to use it in the code you need, as is usually the case with standard modules, to connect it. After that you need to create a new object EventEmitter. It looks like this:

const EventEmitter = require('events')
const door = new EventEmitter()

The class object EventEmitteruses standard mechanisms, in particular - the following events:

• newListener - this event is raised when an event handler is added.
• removeListener - called when removing the handler.

Consider the most useful methods of class objects EventEmitter(a similar object in the names of the methods is indicated by emitter).

▍emitter.addListener ()

An alias for the method emitter.on().

▍emitter.emit ()

Generates an event. It synchronously calls all event handlers in the order in which they were registered.

▍emitter.eventNames ()

Returns an array that contains registered events.

▍emitter.getMaxListeners ()

Returns the maximum number of handlers that can be added to a class object EventEmitter. The default is 10. If necessary, this parameter can be increased or decreased using the method setMaxListeners().

▍emitter.listenerCount ()

Returns the number of event handlers whose name is passed to this method as a parameter:

door.listenerCount('open')

▍emitter.listeners ()

Returns an array of event handlers for the corresponding event, whose name is passed to this method:

door.listeners('open')

▍emitter.off ()

The alias for the method emitter.removeListener(), which appeared in Node 10.

▍emitter.on ()

Register a callback that is called when an event is generated. Here's how to use it:

door.on('open', () => {
  console.log('Door was opened')
})

▍emitter.once ()

It registers a callback that is called only once - at the first occurrence of the event for which this callback is registered. For example:

const EventEmitter = require('events')
const ee = new EventEmitter()
ee.once('my-event', () => {
  //вызвать этот коллбэк один раз при первом возникновении события
})

▍emitter.prependListener ()

When registering a handler using methods, on()or addListener()this handler is added to the end of the handler queue and called to process the corresponding event last. When using the method, the prependListener()handler is added to the top of the queue, which causes it to be called first to handle the event.

▍emitter.prependOnceListener ()

This method is similar to the previous one. Namely, when a handler intended for a single call is registered using the method once(), it is the last in the handler queue and the last is called. The method prependOnceListener()allows you to add such a handler to the beginning of the queue.

▍emitter.removeAllListeners ()

This method removes all handlers for the specified event registered in the corresponding object. Use it like this:

door.removeAllListeners('open')

▍emitter.removeListener ()

Removes the specified handler to pass to this method. In order to save a handler for later deletion, the corresponding callback can be assigned to a variable. It looks like this:

const doSomething = () => {}
door.on('open', doSomething)
door.removeListener('open', doSomething)

▍emitter.setMaxListeners ()

This method allows you to specify the maximum number of handlers that can be added to a single event in a class instance EventEmitter. By default, as already mentioned, you can add up to 10 handlers for a specific event. This value can be changed. Use this method as follows:

door.setMaxListeners(50)

Node.js http module

In the eighth part of this series of materials, we already talked about the standard Node.js module http. It provides the developer with mechanisms for creating HTTP servers. It is the main module used for solving network communication problems in Node.js. You can connect it in code like this:

const http = require('http')

It includes properties, methods and classes. Let's talk about them.

▍Properties

http.METHODS

This property lists all supported HTTP methods:

> require('http').METHODS
[ 'ACL',
  'BIND',
  'CHECKOUT',
  'CONNECT',
  'COPY',
  'DELETE',
  'GET',
  'HEAD',
  'LINK',
  'LOCK',
  'M-SEARCH',
  'MERGE',
  'MKACTIVITY',
  'MKCALENDAR',
  'MKCOL',
  'MOVE',
  'NOTIFY',
  'OPTIONS',
  'PATCH',
  'POST',
  'PROPFIND',
  'PROPPATCH',
  'PURGE',
  'PUT',
  'REBIND',
  'REPORT',
  'SEARCH',
  'SUBSCRIBE',
  'TRACE',
  'UNBIND',
  'UNLINK',
  'UNLOCK',
  'UNSUBSCRIBE' ]

http.STATUS_CODES

This contains the HTTP status codes and their descriptions:

> require('http').STATUS_CODES
{ '100': 'Continue',
  '101': 'Switching Protocols',
  '102': 'Processing',
  '200': 'OK',
  '201': 'Created',
  '202': 'Accepted',
  '203': 'Non-Authoritative Information',
  '204': 'No Content',
  '205': 'Reset Content',
  '206': 'Partial Content',
  '207': 'Multi-Status',
  '208': 'Already Reported',
  '226': 'IM Used',
  '300': 'Multiple Choices',
  '301': 'Moved Permanently',
  '302': 'Found',
  '303': 'See Other',
  '304': 'Not Modified',
  '305': 'Use Proxy',
  '307': 'Temporary Redirect',
  '308': 'Permanent Redirect',
  '400': 'Bad Request',
  '401': 'Unauthorized',
  '402': 'Payment Required',
  '403': 'Forbidden',
  '404': 'Not Found',
  '405': 'Method Not Allowed',
  '406': 'Not Acceptable',
  '407': 'Proxy Authentication Required',
  '408': 'Request Timeout',
  '409': 'Conflict',
  '410': 'Gone',
  '411': 'Length Required',
  '412': 'Precondition Failed',
  '413': 'Payload Too Large',
  '414': 'URI Too Long',
  '415': 'Unsupported Media Type',
  '416': 'Range Not Satisfiable',
  '417': 'Expectation Failed',
  '418': 'I\'m a teapot',
  '421': 'Misdirected Request',
  '422': 'Unprocessable Entity',
  '423': 'Locked',
  '424': 'Failed Dependency',
  '425': 'Unordered Collection',
  '426': 'Upgrade Required',
  '428': 'Precondition Required',
  '429': 'Too Many Requests',
  '431': 'Request Header Fields Too Large',
  '451': 'Unavailable For Legal Reasons',
  '500': 'InternalServer Error',
  '501': 'Not Implemented',
  '502': 'Bad Gateway',
  '503': 'Service Unavailable',
  '504': 'Gateway Timeout',
  '505': 'HTTP VersionNot Supported',
  '506': 'Variant Also Negotiates',
  '507': 'Insufficient Storage',
  '508': 'Loop Detected',
  '509': 'Bandwidth Limit Exceeded',
  '510': 'Not Extended',
  '511': 'Network Authentication Required' }

http.globalAgent

This property points to a global instance of the class http.Agent. It is used to manage connections. It can be considered a key component of the Node.js HTTP subsystem. http.AgentWe'll talk more about the class below.

▍ Methods

http.createServer ()

Returns a new instance of the class http.Server. Here is how to use this method to create an HTTP server:

const server = http.createServer((req, res) => {
  //в этом коллбэке будут обрабатываться запросы
})

http.request ()

Allows you to perform an HTTP request to the server, creating an instance of the class http.ClientRequest.

http.get ()

This method is similar to http.request(), but it automatically sets the HTTP method to a value GETand automatically invokes the form command req.end().

▍Classes

HTTP module provides 5 classes - Agent, ClientRequest, Server, ServerResponseand IncomingMessage. Consider them.

http.Agent

The global instance of the class http.Agentcreated by Node.js is used to manage connections. It is used as the default for all HTTP requests and provides for queuing requests and reusing sockets. In addition, it supports a pool of sockets, which ensures high performance of the network subsystem Node.js. If necessary, you can create your own object http.Agent.

http.ClientRequest

A class object http.ClientRequestrepresenting the running query is created by calling methods http.request()or http.get(). When a response to a request is received, an event is triggered response, in which the response is sent - an instance http.IncomingMessage. The data obtained after the execution of the request can be processed in two ways:

• You can call a method response.read().
• In the event handler, responseyou can configure a listener for the event data, which allows you to work with streaming data.

http.Server

Instances of this class are used to create servers using the command http.createServer(). After we have a server object, we can use its methods:

• The method is listen()used to start the server and organize waiting and processing incoming requests.
• The method close()stops the server.

http.ServerResponse

This object is created by the class http.Serverand passed as the second parameter to the event requestwhen it occurs. Typically, similar objects in the code are assigned a name res:

const server = http.createServer((req, res) => {
  //res - это объект http.ServerResponse
})

In such handlers, after the server’s response is ready to be sent to the client, the method end()that completes the response is called. This method must be called after the completion of the formation of each response.

Here are the methods that are used to work with HTTP headers:

• getHeaderNames() - returns a list of names of installed headers.
• getHeaders() - returns a copy of the installed HTTP headers.
• setHeader('headername', value) - sets the value for the specified header.
• getHeader('headername') - returns the set header.
• removeHeader('headername') - deletes the installed header.
• hasHeader('headername')- returns trueif the response already has a header, the name of which is transferred to this method.
• headersSent()- returns trueif headers have already been sent to the client.

After processing the headers, they can be sent to the client by calling the method response.writeHead(), which, as the first parameter, accepts a status code. As the second and third parameters, you can send a message corresponding to the status code, and headers.

To send data to the client in the response body use the method write(). It sends buffered data to the HTTP response stream.

If headers have not yet been set by the command response.writeHead(), headers will be sent first with a status code and a message that are specified in the request. You can set their values ​​by setting values ​​for properties statusCodeand statusMessage:

response.statusCode = 500response.statusMessage = 'Internal Server Error'

http.IncomingMessage

The class object http.IncomingMessageis created during the work of the following mechanisms:

• http.Server- when processing an event request.
• http.ClientRequest- when processing an event response.

It can be used to work with response data. Namely:

• To find out the status code of the response and the corresponding message, the properties statusCodeand are used statusMessage.
• The response headers can be viewed by referring to the property headersor rawHearders(for a list of raw headers).
• The query method can be found using the property method.
• You can find out what version of HTTP you are using with the property httpVersion.
• To get the URL is a property url.
• The property socketallows you to get the object net.Socketassociated with the connection.

Response data is presented as a stream as the object http.IncomingMessageimplements the interface Readable Stream.

Work with streams in Node.js

Threads are one of the fundamental concepts used in Node.js applications. Flows are tools that allow you to read and write files, organize networking of systems, and, in general, effectively implement data exchange operations.

The concept of threads is not unique to Node.js. They appeared in the OS of the Unix family decades ago. In particular, programs can interact with each other by transmitting data streams using pipelines (using the conveyor symbol - |).

If we imagine, say, reading a file without using streams, then, in the course of executing the corresponding command, the contents of the file will be completely read into memory, after which it will be possible to work with these contents.

Through the use of a streaming mechanism, files can be read and processed in parts, which eliminates the need to store large amounts of data in memory.

The Node.js stream module is the foundation on which all APIs that support streaming are built.

▍About Strengths Strengths

Streams, in comparison with other data processing methods, have the following advantages:

• Efficient use of memory. Working with a stream does not involve storing in memory large amounts of data loaded there in advance, before it becomes possible to process them.
• Save time. The data obtained from the stream can be processed much faster than in the case when, in order to start processing them, you have to wait for them to be fully loaded.

▍Example work with threads

A traditional example of working with streams demonstrates reading a file from disk.

First, consider the code in which the threads are not used. The standard module Node.js fsallows you to read the file, after which it can be sent via HTTP in response to a request received by the HTTP server:

const http = require('http')
const fs = require('fs')
const server = http.createServer(function(req, res){
  fs.readFile(__dirname + '/data.txt', (err, data) => {
    res.end(data)
  })
})
server.listen(3000)

The method readFile()used here allows you to read the entire file. When the reading is completed, it calls the appropriate callback.

The method res.end(data)called in the callback sends the contents of the file to the client.

If the file size is large, then this operation will take a lot of time. Here is the same example rewritten using streams:

const http = require('http')
const fs = require('fs')
const server = http.createServer((req, res) => {
  const stream = fs.createReadStream(__dirname + '/data.txt')
  stream.pipe(res)
})
server.listen(3000)

Instead of waiting for the moment when the file is completely read, we begin to transfer its data to the client immediately after the first portion of this data is ready to be sent.

PipeMethod pipe ()

In the previous example, we used a view construct stream.pipe(res)in which the file stream method is invoked pipe(). This method takes data from its source and sends it to its destination.

It is called for the stream representing the data source. In this case, it is the file stream that is sent to the HTTP response.

The return value of the method pipe()is the target stream. This is very convenient, since it allows you to chain several method calls pipe():

src.pipe(dest1).pipe(dest2)

This is equivalent to this design:

src.pipe(dest1)
dest1.pipe(dest2)

▍API Node.js using threads

Threads are a useful mechanism, with the result that many Node.js core modules provide standard threading capabilities. Here are some of them:

• process.stdin- returns the stream connected to stdin.
• process.stdout- returns the stream connected to stdout.
• process.stderr- returns the stream connected to stderr.
• fs.createReadStream() - creates a readable stream for working with a file.
• fs.createWriteStream()- creates a recordable stream for working with the file.
• net.connect() - initiates a stream based connection.
• http.request()- returns an instance of the class http.ClientRequestthat provides access to the recorded stream.
• zlib.createGzip()- compresses data using an algorithm gzipand sends them to the stream.
• zlib.createGunzip()- performs decompression- gzipflow.
• zlib.createDeflate()- compresses data using an algorithm deflateand sends them to the stream.
• zlib.createInflate()- performs decompression- deflateflow.

▍Different stream types

There are four types of threads:

• A read stream ( Readable) is a stream from which data can be read. You cannot write data to such a stream. When data flows into such a stream, they are buffered until the data consumer starts reading them.
• The stream to write to ( Writable) is the stream to which data can be sent. You cannot read data from it.
• Duplex stream ( Duplex) - you can send data to such a stream and read it from it. Essentially, it is a combination of a stream for reading and a stream for writing.
• Transforming stream ( Transform) - such streams are similar to duplex streams, the difference is that what comes to the input of these streams converts what can be read from them.

▍Create a stream to read

The stream for reading can be created and initialized using the module’s capabilities stream:

const Stream = require('stream')
const readableStream = new Stream.Readable()

Now you can put data into the stream, which the consumer of this data can later read:

readableStream.push('hi!')
readableStream.push('ho!')

▍ Create a stream for recording

In order to create a recordable stream, you need to extend the base object Writableand implement its method _write(). To do this, first create the appropriate thread:

const Stream = require('stream')
const writableStream = new Stream.Writable()

Then we implement its method _write():

writableStream._write = (chunk, encoding, next) => {
    console.log(chunk.toString())
    next()
}

Now it is possible to connect a stream intended for reading to such a stream:

process.stdin.pipe(writableStream)

▍Getting data from the stream for reading

In order to obtain data from a stream intended for reading, we will use a stream for writing:

const Stream = require('stream')
const readableStream = new Stream.Readable()
const writableStream = new Stream.Writable()
writableStream._write = (chunk, encoding, next) => {
    console.log(chunk.toString())
    next()
}
readableStream.pipe(writableStream)
readableStream.push('hi!')
readableStream.push('ho!')
readableStream.push(null)

The team readableStream.push(null)reports the end of the output.

You can work with threads for reading directly by processing the event readable:

readableStream.on('readable', () => {
  console.log(readableStream.read())
})

▍Sending data to stream for writing

To send data to a stream for recording, use the method write():

writableStream.write('hey!\n')

▍Message for recording that data record is complete

In order to tell the stream to write that the data has been written to it, you can use its method end():

writableStream.end()

This method accepts several optional parameters. In particular, it can transfer the last portion of the data to be written to the stream.

The basics of working with MySQL in Node.js

MySQL is one of the most popular DBMS in the world. In the Node.js ecosystem there are several packages that allow you to interact with MySQL databases, that is, to store data in them, retrieve data from databases and perform other operations.

We will use the mysqljs / mysql package . This project, which has existed for a long time, has collected more than 12,000 stars on GitHub. In order to reproduce the following examples, you will need a MySQL server.

▍ Package installation

To install this package use the following command:

npm install mysql

▍ Initialization of database connection

First, connect the package in the program:

const mysql = require('mysql')

After that create a connection:

const options = {
  user: 'the_mysql_user_name',
  password: 'the_mysql_user_password',
  database: 'the_mysql_database_name'
}
const connection = mysql.createConnection(options)

Now try to connect to the database:

connection.connect(err => {
  if (err) {
    console.error('An error occurred while connecting to the DB')
    throw err
  }
}

Соединения Connection Parameters

In the example above, the object optionscontained three connection parameters:

const options = {
  user: 'the_mysql_user_name',
  password: 'the_mysql_user_password',
  database: 'the_mysql_database_name'
}

In fact, these parameters exist much more. Including the following:

• host- the name of the host where the MySQL server is located; the default is localhost.
• port- server port number, by default - 3306.
• socketPath - used to specify a Unix socket instead of a host and port.
• debug - allows you to work in debug mode, by default this feature is disabled.
• trace - allows you to display information about the stack trace when errors occur, by default this feature is enabled.
• ssl - used to configure SSL-connection to the server.

▍ Execute a SELECT query

Now everything is ready to execute SQL queries to the database. To execute queries, use the join method query, which accepts the request and callback. If the operation is completed successfully, the callback will be called with the transfer of data received from the database to it. In case of an error, the corresponding error object will be placed in the callback. Here is what it looks like when executing a data query:

connection.query('SELECT * FROM todos', (error, todos, fields) => {
  if (error) {
    console.error('An error occurred while executing the query')
    throw error
  }
  console.log(todos)
})

When forming the query, you can use the values ​​that will be automatically embedded in the query string:

const id = 223
connection.query('SELECT * FROM todos WHERE id = ?', [id], (error, todos, fields) => {
  if (error) {
    console.error('An error occurred while executing the query')
    throw error
  }
  console.log(todos)
})

To send several values ​​to the request, you can use an array as the second parameter:

const id = 223
const author = 'Flavio'
connection.query('SELECT * FROM todos WHERE id = ? AND author = ?', [id, author], (error, todos, fields) => {
  if (error) {
    console.error('An error occurred while executing the query')
    throw error
  }
  console.log(todos)
})

▍Perform INSERT request

Queries INSERTare used to write data to the database. For example, write to the database object:

const todo = {
  thing: 'Buy the milk'
  author: 'Flavio'
}
connection.query('INSERT INTO todos SET ?', todo, (error, results, fields) => {
  if (error) {
    console.error('An error occurred while executing the query')
    throw error
  }
})

If the table to which data is added has a primary key with a property auto_increment, its value will be returned in the form results.insertId:

const todo = {
  thing: 'Buy the milk'
  author: 'Flavio'
}
connection.query('INSERT INTO todos SET ?', todo, (error, results, fields) => {
  if (error) {
    console.error('An error occurred while executing the query')
    throw error
  }}
  const id = results.resultId
  console.log(id)
)

▍Closing the database connection

After work with the database is completed and it is time to close the connection - use its method end():

connection.end()

This will lead to the correct shutdown of the database.

About the difference between the development environment and the production environment

By creating applications in the Node.js environment, you can use various configurations for the development and production environments.

By default, the Node.js platform runs in the development environment. In order to indicate to her that the code is executed in the production environment, you can set up an environment variable NODE_ENV:

NODE_ENV=production

This is usually done on the command line. In Linux, for example, it looks like this:

export NODE_ENV=production

It is better, however, to put a similar command in a configuration file like .bash_profile(when using Bash), since otherwise such settings are not saved after the system is rebooted.

You can adjust the value of the environment variable using the following construction when starting the application:

NODE_ENV=production node app.js

This environment variable is widely used in external libraries for Node.js. Setting NODE_ENVto a value productionusually means the following:

• Logging is reduced to a minimum.
• More caching levels are used to optimize performance.

For example, Pug - a library for working with templates used by Express, prepares to work in debug mode if the variable is NODE_ENVnot set to a value production. Express views, in design mode, are generated when each request is processed. In production mode, they are cached. There are many other similar examples.

Express provides configuration hooks for each environment. Which one will be called depends on the value NODE_ENV:

app.configure('development', () => {
  //...
})
app.configure('production', () => {
  //...
})
app.configure('production', 'staging', () => {
  //...
})

For example, with their help, you can use different event handlers for different modes:

app.configure('development', () => {
  app.use(express.errorHandler({ dumpExceptions: true, showStack: true }));
})
app.configure('production', () => {
  app.use(express.errorHandler())
})

Т Results

Hopefully, having mastered this guide, you learned a lot about Node.js platform in order to get to work with it. We believe that now, even if you started reading the first article of this cycle, without understanding Node.js at all, you can start writing something of your own, read the code of others with interesting information and use the documentation for Node.js really .

Dear readers! If you have read this series of publications, without having knowledge of the development for Node.js, please tell us how you rate your level now.