Node.js Part 8 Tutorial: HTTP and WebSocket Protocols

Published on October 02, 2018

Node.js Part 8 Tutorial: HTTP and WebSocket Protocols

Original author: Flavio Copes
  • Transfer
Node.js is a server platform. The main task of the server is to process requests from clients, in particular - from browsers, as quickly and efficiently as possible. The eighth part of the translation guide for Node.js, which we publish today, is devoted to the protocols HTTP and WebSocket.




What happens when making HTTP requests?


Let's talk about how browsers perform requests to servers using the HTTP / 1.1 protocol.

If you have ever been interviewed in the IT field, then you might be asked about what happens when you type something in the address bar of the browser and press Enter. Perhaps this is one of the most popular questions that is encountered at such interviews. The one who asks such questions wants to know if you can explain some fairly simple concepts and find out if you understand the principles of the Internet.

This question involves many technologies, to understand the general principles of which is to understand how one of the most complex systems ever built by mankind, which covers the whole world, is arranged.

HTTP HTTP Protocol


Modern browsers are able to distinguish real URLs entered in their address bar from search queries, which are usually processed using the default search engine. We will talk about URLs. If you enter the address of the site in the browser line, it seems flaviocopes.comthat the browser converts this address to a form http://flaviocopes.com, based on the assumption that the HTTP protocol will be used to exchange data with the specified resource. Please note that on Windows, what we are going to talk about here may look a little different than on macOS and Linux.

DNS DNS Lookup Phase


So, the browser, starting work on downloading data from the address requested by users, performs a DNS lookup (DNS Lookup) operation in order to find out the IP address of the corresponding server. Character resource names entered in the address bar are convenient for people, but the Internet device implies the ability to exchange data between computers using IP addresses, which are sets of numbers like 222.324.3.1 (for IPv4).

First, figuring out the IP address of the server, the browser looks in the local DNS cache in order to find out if a similar procedure has been performed recently. The Chrome browser, for example, there is a convenient way to see the DNS-cache by typing in the address bar the following address: chrome://net-internals/#dns.

If nothing is found in the cache, the browser uses the POSIX system call.gethostbyname in order to find out the IP address of the server.

GetGethostbyname feature


The function gethostbynamefirst checks the file hosts, which, in macOS or Linux, can be found at the address /etc/hosts, in order to find out whether it is possible to dispense with local information when figuring out the server address.

If it is not possible to resolve the request for finding the IP address of the server by local means, the system executes the request to the DNS server. Addresses of such servers are stored in the system settings.

Here are a couple of popular DNS servers:

  • 8.8.8.8: Google's DNS server.
  • 1.1.1.1: CloudFlare DNS server.

Most people use the DNS servers provided by their providers. The browser performs DNS queries using the UDP protocol.

TCP and UDP are two basic protocols used in computer networks. They are located at the same conceptual level, but TCP is a connection-oriented protocol, and for exchanging UDP messages, processing of which creates a small additional load on the systems, the procedure of establishing a connection is not required. We will not talk about exactly how data is exchanged via UDP.

The IP address corresponding to the domain name in question may be in the cache of the DNS server. If this is not the case, it will contact the root DNS server. The root DNS server system consists of 13 servers on which the entire Internet depends.

It should be noted that the root DNS server is unknown correspondences between all existing domain names in the world and IP addresses. But such servers know the addresses of top-level DNS servers for domains such as .com, .it, .pizza, and so on.

After receiving the request, the root DNS server redirects it to the DNS server of the top-level domain, to the so-called TLD server (from the Top-Level Domain).

Suppose a browser is looking for an IP address for a server flaviocopes.com. By accessing the root DNS server, the browser will receive from it the address of the TLD server for the .com zone. Now this address will be stored in the cache, as a result, if you need to know the IP address of another URL from the .com zone, the root DNS server will not have to be contacted again.

The TLD servers have the IP addresses of name servers (Name Server, NS), by means of which you can find the IP address using the URL we have. Where do NS servers get this information? The fact is that if you buy a domain, the domain registrar sends data about it to the name servers. A similar procedure is performed and, for example, when changing hosting.

The servers in question are usually owned by hosting providers. As a rule, to protect against failures, several such servers are created. For example, they may have the following addresses:

  • ns1.dreamhost.com
  • ns2.dreamhost.com
  • ns3.dreamhost.com

To find out the IP address at the URL, in the end, refer to such servers. It is they who keep current data on IP addresses.

Now, after we managed to find out the IP address behind the URL entered in the address bar of the browser, we proceed to the next step of our work.

▍ TCP connection establishment


Having learned the IP address of the server, the client can initiate a TCP connection procedure to it. In the process of establishing a TCP connection, the client and the server transmit to each other some service data, after which they will be able to exchange information. This means that after the connection is established, the client will be able to send a request to the server.

ЗапросаSending a request


The request is a structured text according to the rules of the protocol used. It consists of three parts:

  • Query string
  • Request header
  • Body request.

Query string


The query string is a single text string that contains the following information:

  • HTTP method.
  • Resource address
  • Protocol version.

For example, it may look like this:

GET / HTTP/1.1

Request header


The request header is represented by a set of view pairs поле: значение. There are 2 required header fields, one of which is Host, and the second is Connection. The remaining fields are optional.

The headline might look like this:

Host: flaviocopes.com
Connection: close

The field Hostindicates the domain name that the browser is interested in. A field Connectionset to a value closemeans that the connection between the client and the server does not need to be kept open.

Other commonly used query headers include the following:

  • Origin
  • Accept
  • Accept-Encoding
  • Cookie
  • Cache-Control
  • Dnt

In fact, there are many more.

The request header ends with an empty string.

Request body


The request body is optional, it is not used in GET requests. The request body is used in POST requests, as well as in other requests. It may contain, for example, data in JSON format.

Since now we are talking about a GET request, the request body will be empty, we will not work with it.

▍ Answer


After the server receives the request sent by the client, it processes it and sends a response to the client.

The response starts with a status code and a corresponding message. If the request is successful, the beginning of the response will look like this:

200 OK

If something went wrong, there may be other codes. For example, the following:

  • 404 Not Found
  • 403 Forbidden
  • 301 Moved Permanently
  • 500 Internal Server Error
  • 304 Not Modified
  • 401 Unauthorized

The response further contains a list of HTTP headers and the response body (which, since the request is executed by the browser, will be HTML code).

HTML parsing


After the browser receives a response from the server, the body of which contains the HTML code, it begins to parse it, repeating the above process for each resource that is needed to form the page. These resources include, for example, the following:

  • CSS files.
  • Images.
  • Webpage icon (favicon).
  • Javascript files

The way the browser displays the page does not apply to our conversation. The main thing that interests us here is that the above-described process of requesting and receiving data is used not only for the HTML code, but also for any other objects transmitted from the server to the browser using the HTTP protocol.

About creating a simple server using Node.js


Now, after we have analyzed the interaction between the browser and the server, you can take a fresh look at the First Node.js application section from the first part of this series of materials, in which we described the simple server code.

Making HTTP requests using Node.js


To execute HTTP requests using Node.js, the corresponding module is used . The examples below use the https module . The fact is that in modern conditions, whenever possible, it is necessary to use the HTTPS protocol.

▍ GET requests


Here is an example of performing a GET request using Node.js:

const https = require('https')
const options = {
  hostname: 'flaviocopes.com',
  port: 443,
  path: '/todos',
  method: 'GET'
}
const req = https.request(options, (res) => {
  console.log(`statusCode: ${res.statusCode}`)
  res.on('data', (d) => {
    process.stdout.write(d)
  })
})
req.on('error', (error) => {
  console.error(error)
})
req.end()

▍POST request execution


Here's how to perform a POST request from Node.js:

const https = require('https')
const data = JSON.stringify({
  todo: 'Buy the milk'
})
const options = {
  hostname: 'flaviocopes.com',
  port: 443,
  path: '/todos',
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length
  }
}
const req = https.request(options, (res) => {
  console.log(`statusCode: ${res.statusCode}`)
  res.on('data', (d) => {
    process.stdout.write(d)
  })
})
req.on('error', (error) => {
  console.error(error)
})
req.write(data)
req.end()

▍PUT and DELETE requests


The execution of such requests looks the same as the execution of POST requests. The main difference, in addition to the semantic content of such operations, is the value of the property of the methodobject options.

▍Executing HTTP requests in Node.js using the Axios library


Axios is a very popular JavaScript library that also works in a browser (this includes all modern browsers and IE, starting with IE8), and in Node.js, which can be used to perform HTTP requests.

This library is based on promises, it has some advantages over standard mechanisms, in particular, over API Fetch. Among its advantages are the following:

  • Support for older browsers (you need a polyfill to use Fetch).
  • Ability to interrupt requests.
  • Support setting timeouts for requests.
  • Built-in protection against CSRF attacks.
  • Support uploading data providing progress information on this process.
  • Support JSON data conversion.
  • Work at Node.js

Installation


To install the Axios, you can use npm:

npm install axios

The same effect can be achieved when working with yarn:

yarn add axios

You can connect the library to the page using unpkg.com:

<script src="https://unpkg.com/axios/dist/axios.min.js"></script>

API Axios


You can execute an HTTP request using the object axios:

axios({
  url: 'https://dog.ceo/api/breeds/list/all',
  method: 'get',
  data: {
    foo: 'bar'
  }
})

But it is usually more convenient to use special methods:

  • axios.get()
  • axios.post()

This is similar to how jQuery $.ajax()uses $.get()and instead $.post().

Axios offers separate methods for performing other types of HTTP requests that are not as popular as GET and POST, but are still used:

  • axios.delete()
  • axios.put()
  • axios.patch()
  • axios.options()

The library has a method for executing a query designed to get only HTTP headers, without the response body:

  • axios.head()

GET requests


Axios is convenient to use using the modern syntax async / await. In the following sample code for Node.js, the library is used to load a list of dog breeds from the Dog API . Here the method is applied axios.get()and rocks are calculated:

const axios = require('axios')
const getBreeds = async () => {
  try {
    return await axios.get('https://dog.ceo/api/breeds/list/all')
  } catch (error) {
    console.error(error)
  }
}
const countBreeds = async () => {
  const breeds = await getBreeds()
  if (breeds.data.message) {
    console.log(`Got ${Object.entries(breeds.data.message).length} breeds`)
  }
}
countBreeds()

The same can be rewritten without using async / await, using promises:

const axios = require('axios')
const getBreeds = () => {
  try {
    return axios.get('https://dog.ceo/api/breeds/list/all')
  } catch (error) {
    console.error(error)
  }
}
const countBreeds = async () => {
  const breeds = getBreeds()
    .then(response => {
      if (response.data.message) {
        console.log(
          `Got ${Object.entries(response.data.message).length} breeds`
        )
      }
    })
    .catch(error => {
      console.log(error)
    })
}
countBreeds()

Using Parameters in GET Requests


A GET request may contain parameters that look like this in a URL:

https://site.com/?foo=bar

With Axios, this kind of query can be done like this:

axios.get('https://site.com/?foo=bar')

The same effect can be achieved by setting the property paramsin the object with parameters:

axios.get('https://site.com/', {
  params: {
    foo: 'bar'
  }
})

POST requests


The execution of POST requests is very similar to the execution of GET requests, but here, instead of a method axios.get(), the method is used axios.post():

axios.post('https://site.com/')

As the second argument, the method postaccepts an object with query parameters:

axios.post('https://site.com/', {
  foo: 'bar'
})

Using the WebSocket protocol in Node.js


WebSocket is an alternative to HTTP, it can be used to organize data exchange in web applications. This protocol allows you to create long-lived bidirectional communication channels between the client and the server. After the connection is established, the communication channel remains open, which gives the application a very fast connection, characterized by low latency and a small additional load on the system.

The WebSocket protocol is supported by all modern browsers.

▍Difference from HTTP


HTTP and WebSocket are very different protocols that use different approaches to data exchange. HTTP is based on the request-response model: the server sends some data to the client after it has been requested. In the case of WebSocket, everything is different. Namely:

  • The server can send messages to the client on its own initiative, without waiting for a request from the client.
  • Client and server can exchange data simultaneously.
  • When sending a message is used extremely small amount of service data. This, in particular, leads to low data transfer delays.

The WebSocket protocol is very well suited for real-time communication through channels that remain open for a long time. HTTP, in turn, is great for organizing episodic communication sessions initiated by the client. At the same time, it should be noted that, from the point of view of programming, it is much easier to implement data exchange via the HTTP protocol than via the WebSocket protocol.

▍ Secure version of WebSocket protocol


There is an insecure version of the WebSocket protocol (URI scheme ws://), which resembles, in terms of security, the protocol http://. Usage ws://should be avoided, giving preference to the protected version of the protocol wss://.

▍ Creating WebSocket connections


To create a WebSocket connection, you need to use the appropriate constructor :

const url = 'wss://myserver.com/something'
const connection = new WebSocket(url)

After a successful connection, an event is triggered open. You can listen to this event by assigning a callback function to the property of the onopenobject connection:

connection.onopen = () => {
  //...
}

For error handling, an event handler is used onerror:

connection.onerror = error => {
  console.log(`WebSocket error: ${error}`)
}

▍Sending data to server


After opening a WebSocket connection to the server, you can send data to it. This can be done, for example, in a callback onopen:

connection.onopen = () => {
  connection.send('hey')
}

▍Receiving data from the server


To receive data sent from the server using the WebSocket protocol, you can assign a callback onmessageto be called when the event is received message:

connection.onmessage = e => {
  console.log(e.data)
}

▍ Implementing a WebSocket server in Node.js environment


In order to implement a WebSocket server in the Node.js environment, you can use the popular ws library . We will use it for server development, but it is suitable for creating clients and for organizing interaction between two servers.

Install this library by initializing the project:

yarn init
yarn add ws

The WebSocket server code we need to write is rather compact:

constWebSocket = require('ws')
const wss = newWebSocket.Server({ port: 8080 })
wss.on('connection', ws => {
  ws.on('message', message => {
    console.log(`Received message => ${message}`)
  })
  ws.send('ho!')
})

Here we create a new server that listens on port 8080, which is standard for the WebSocket protocol, and we describe a callback that, when the connection is established, sends a message to the client ho!and displays the message received from the client to the console.

Here is a working example of a WebSocket server, and here is a client that can interact with it.

Results


Today we talked about the networking mechanisms supported by the Node.js platform, drawing parallels with similar mechanisms used in browsers. Our next topic will be working with files.

Dear readers! Do you use the WebSocket protocol in your web applications, the server part of which was created using Node.js?