This is not the first introductory article about Impress on Habré, but over the past year I got a lot of questions and gained some experience in explaining the architecture and philosophy of this application server and, I hope, I began to better understand the problems and tasks of developers starting to master it. Yes, and in the server itself there have been enough changes to make the urgency of a completely new introductory article.

Impress Application Server (IAS) is an application server for Node.jswith alternative architecture and philosophy, not like the mainstream of development under a node and designed to simplify and automate a wide range of repeatable typical tasks, raise the level of abstraction of application code, define the framework and structure of applications, optimize both code performance and developer productivity. IAS now covers only server tasks, but it does it in a comprehensive way, for example, you can combine APIs, web sockets, streaming, statics, Server-Sent Events, proxying and URL-rewriting on one port, serve several domains and several applications, as on one server, and on a group of servers working in conjunction, as a whole, as one application server.

Introduction

To begin with, I want to list a number of problems in the generally accepted approach for node.js that prompted me to start developing application server:

1. Application code is often mixed with system code. The fact is that the node, and most of the derived frameworks, are too low-level and each application necessarily contains part of the system code that is not related to the tasks of the subject area. So it happens, for example, that adding an HTTP header becomes a method of the Patient class and is in a single file with the task of routing URLs to this patient and sending events via web sockets. This is monstrous.
2. Noda gives excessive freedom in terms of application architecture , which is difficult to immediately digest not only a beginner, but even an experienced specialist. In addition to the concept of middleware, which, you must admit, is not enough for full development, there are no common architectural patterns. Dividing a project into files, dividing logic into classes, applying patterns, creating internal APIs in applications, allocating layers and even the directory structure are all left to the discretion of the developer. As a result, the structure and architecture of projects is very different for different teams and specialists, which complicates the understanding and docking of the code.
3. In the world of nodes, there is fanatical worship of REST and, as a result, refusal to store state in server memory. This is despite the fact that Node.js applications live in memory for a long time (i.e., they are not loaded with each request and do not end between requests). Weak memory usage ignoring the ability to deploy a model of the problem to be solved there for a long time (due to which, I / O could be reduced to a minimum) is a crime against performance.
4. A large number of modules in npm are garbage modules , and among the few good ones, it’s not easy to find the right one (the number of downloads and stars does not always adequately reflect the quality of the code and the speed of troubleshooting). It is even more difficult to compose a complete application from a set of good, even very good modules. Together, they can behave unstably and unpredictably. The modules are not sufficiently shielded from each other to eliminate integration conflicts (for example, some module can override res.end or send http headers, while others do not expect this behavior).

There are still many minor problems, and the deep grief with catching errors in Node.js is a topic for three volumes, filled with tears, blood and coffee (tea). As a consequence of all of the above, the node is still a cause for concern and, in most cases, is used as an additional tool in conjunction with other server technologies, performing auxiliary work, such as: scripting the assembly of client applications, prototyping, or providing delivery of notifications on web sockets. Very rarely you can meet a large project that has a server part exclusively on a node.

Formulation of the problem

In addition to negative motivation (the listed problems), there were also positive motivating factors for the development of IAS (ideas and tasks):

1. Scaling node.js applications on more than one server, each of which has its own cluster (a cluster of processes connected by IPC interprocess communication).
2. Serving multiple applications within a single process, or a cluster of processes or a server farm with a cluster of processes on each.
3. Automatically replaces code in memory if it changes on disk, even without restarting the application, by monitoring the file system. As soon as the files loaded by the application change, the IAS reads them into memory. At some point in the memory there may be several versions of the code, the old one is unloaded as soon as the processing of all requests that came before the change is completed, and the new one is already used for the following requests.
4. Synchronization of data structures in memory between processes. Of course, not all memory structures, but only the global fragment of the domain model deployed in it. Additive changes and transactions are supported, i.e. if some parameter is incremented in parallel in different processes, then these changes merge, because their order is not important.

Impress Philosophy

1. Maximum memory usage . Faster than asynchronous I / O is only when there is no I / O at all or it is reduced to a minimum and runs in lazy mode, and not during requests.
2. Monolithic architecture and high code coherence, all core modules are integrated, coordinated and optimized to work together. Thanks to this, there are no unnecessary checks, and the behavior in solving typical problems is always predictable.
3. Multiplexer of ports, hosts, IP, protocols, servers, processes, applications, processors and methods. Thus, you can combine statics, APIs, web sockets, SSE, streaming video and large files on one port, process several applications on different domains or multi-domain sites, etc.
4. The principle of an applied virtual machine isolated from the environment using sandboxes. Each application has its own context (scope) into which its libraries and data are loaded. The application architecturally provides places for all kinds of handlers: initialization and finalization, data models and database structure, configuration and installation (first start), updates, migration, etc.
5. Separation of application and system code. In general, separating layers of abstractions (higher and lower levels) in an application is much more important than separating logic with a model and presentation within the same layer of abstractions (sometimes it’s even more efficient to mix them).
6. Mapping URLs to the file system with inheritance and redefinition in the directory tree. But with the ability to programmatically add processors directly to memory and prescribe routing by hand in addition to automatic routing by directory structure.
7. The brevity of the code (see examples below) is achieved thanks to the developed built-in API, which takes care of everything necessary in the vast majority of cases, can be expanded and reused from project to project. Also brevity is promoted by a special style of working with visibility zones and code splitting into files with logical parts of convenient size.

Application area

IAS is designed to create several types of applications:

1. One-page web applications with API and dynamic page changes on the client without rebooting from the server.
2. Multi-page web applications with some degree of dynamics on the pages through the API (the logic is divided into client and server).
3. Multipage applications with page reloading at each event (all logic on the server).
4. Applications with two-way data exchange or a stream of events from the server, interactive applications (usually this is an add-on for options 1 and 2).
5. Network API for server access for native mobile and window applications.

Multiple API creation methods supported

• RPC API - when a URL identifies a network method with a set of parameters, the order of the call is important and the state between the calls is stored on both the client and the server;
• REST API, when a URL identifies a resource, a limited number of operations can be performed on the resource (for example, HTTP verbs or CRUD), requests are atomic, there is no difference in the order of calls, and there is no state between calls;
• Event bus: a unidirectional or bi-directional stream of client-server interaction via WebSockets or SSE (Server-Sent Events) used to notify or synchronize the state of objects between the client and server;
• Or a mixed way.

Handlers

The analog of middleware for IAS is the handler (handler) - this is an asynchronous function that has two parameters (client and callback) and is located in a separate file (from which it is exported). When a callback is called, the IAS application server finds out that the processing has completed. If the function does not cause a callback longer than the timeout, then IAS returns the HTTP status 408 (Request timeout) and fixes the problem in the logs. If an exception occurs when the handler is called, then IAS assumes the answer to the client, catching the error and restoring the work in the optimal way, up to deleting and re-creating the sandbox with damaged or leaked data structures.

Example handler API:

module.exports = function(client, callback) {
  dbAlias.equipment.find({ type: client.fields.type }).toArray(function(err, nodes) {
    if (!err) callback(nodes);
    else callback({ message: 'Equipment of given type not found' }, 404);
  });
}

Each HTTP request can cause the execution of several handlers. For example, if the URL is requested domain.com/api/example/method.jsonand IAS is set to /impress, then execution will start from the directory /impress/appplications/domain.com/app/api/example/method.json/and go through the following steps:

• access rights are checked for the access.js file from this directory, session (if any) and the user account (if there is a session bound),
• the request.js handler from this directory is executed (if found), it is executed when any HTTP method is called (get, post ...),
• one of the handlers corresponding to the HTTP request method is executed, for example, get.js , put.js , post.js , etc. (if found)
• the end.js handler is executed (if found), it will be called with any HTTP method,
• serialization of response data or page standardization will occur (if this is provided by the type of response returned),
• the result of the request is sent to the client,
• after that, when the client received the response and we do not delay it, the lazy.js handler is executed (if found), which can, for example, do pending operations, change / recount or save data in the database,
• at any stage of execution, an error may occur in the application code, causing an unhandled exception, but we do not need to wrap the code in try / catch or create a domain, this is already done in IAS, if an error occurs, the error.js handler will be called (if found).

If the requested directory does not have the right handler, then IAS will look for it one directory above until it reaches /app. If the handler is in the folder, then it can programmatically call the handler from the directory above (or the one closest up the tree) through client.inherited(). Thus, you can use the directory tree to generate inheritance and redefine handlers. For example, you can generate response data in the output /api/example/request.js, and to issue them in three formats /api/example/method.json, /api/example/method.html(contains also templates for output html), /api/example/method.csv(may comprise additional steps, such as forming table header). Or make a common error handler for the entire API in the file/api/error.js. This approach provides more flexibility and reduces the size of the code, however, we pay for this with well-known limitations.

Directory extensions mean automatic return of a certain type of content from them, which means setting certain HTTP headers and converting the result to the desired data format. All of this can be redefined manually, but using extensions reduces the amount of code. Such extensions are supported out of the box: .json, .jsonp, .xml, .ajax, .csv, .ws, .sseand this list is simply extensible using plugins.

Namespaces

The following names are visible inside the handler, through which we can access the IAS functions and connected libraries:

• client - an object containing parsed request fields, links to the original request and response, respectively, in client.reqand client.res, the API for working with the request, links to the session and authorized user;
• application - the object responsible for the application, and containing its configuration, parameters and the corresponding application server API;
• db - namespace containing links to all loaded DBMS drivers and established database connections; you can access them through db [alias] or db.alias;
• api - namespace containing links to all the built-in and external libraries that were allowed from the application configuration. For example api.fs.readFile(...)orapi.async.parallel(...);
• api.impress - link to the application server API;
• System global identifiers , common for JavaScript and Node.js: require, console, Buffer, process, setTimeout, clearTimeout, setInterval, clearInterval, setImmediate, clearImmediate. But we can prohibit the use of some of them in the configuration, for example, by disabling the require application and providing it with only a certain set of libraries automatically loaded into its namespace api.

You do not need to do require in handlers, just install the libraries in the / impress folder via npm install and connect them through the /config/sandbox.js configuration (first in the IAS config, and then locally in the application config). Further libraries are visible in handlers through api.libName, in the same way built-in libraries become visible, for example, api.path.extname(...)etc.

All databases and DBMS drivers are visible through db.name. Connections are configured in /config/databases.js (for each application separately), are established at startup and are automatically restored when communication is lost. The set includes drivers for MongoDB, PostgreSQL and MySQL, wrapped in plug-ins for IAS, if desired, you can wrap any DBMS in driver plug-ins in 30 minutes.

For the html content type, a simple built-in template engine is used, it is needed not for the complete generation of pages on the server side, but for the layout assembly (the main layout and location of the interface pieces), as well as for the substitution of a few values ​​from data structures in html. The template engine contains inlays and iterators, but more complex template needs to be implemented already in the browser using React, Angular, EJS, etc., requesting templates and data separately and collecting them in the browser (using template reuse), which is typical for dynamic web applications. The built-in templating engine starts rendering from a file html.templateand substitutes data from it into it client.context.data. The construction will @fieldName@substitute the value from the field, the construction @[file]@will insert the file file.template, and the construction@[name]@ ... @[/name]@implements an iterator over a hash or an array named name.

For handlers that return serialized data (.json, .jsonp, .csv, etc.), templating is not needed. For them, the data structure is client.context.datasimply serialized in JSON (with recursion clipping). For convenience, you can return the data structure from the handler by the first parameter. callback({ field: "value" });If one handler returned the data to the callback or assigned it to client.context.data, then those following it (until the end of the current HTTP request) can read and modify the data.

Handlers can change the http status code, add their own http headers, but in normal mode they only work with the client object, which has methods of a secure API:client.error(code), client.download(filePath, attachmentName, callback), client.cache(timeout), client.end(output)etc. Starting with version 0.1.157, IAS has implemented partial support for middleware handlers, which have 3 parameters: req, res and next. But this is extremely rare, and code ported from projects to express or connect can usually be rewritten several times shorter and easier.

Create handlers of both types, i.e. handler (with 2nd parameters) and middleware (with 3rd parameters) can be not only from files, but by adding routing manually, through method calls, for example:

application.get('/helloWorld.ajax', function(req, res, next) {
  res.write('
Middleware handler style
');
  next();
});

Application structure

Server code is not limited to processors; an application may also contain a domain model, specialized libraries and utilities used in many processors, and other “places” for placing logic and data. All applications running in IAS are located in the / applications directory and have the following structure:

• / app - the root directory of the handlers corresponding to the root of the site hostname,
• / config - application configuration
• / doc - documentation and supporting materials,
• / files - a directory for placing files uploaded by the user (a 2- or 3-level system of subdirectories is automatically built in it so as not to overload the file system with a large number of files),
• / init - the initialization code that starts when the server starts (here you can programmatically create handlers, prepare data structures in memory, open tcp ports, etc.),
• / lib - a directory for libraries and utilities that are loaded at startup (but after initialization) and are accessible from the entire application code,
• / log - directory for logs (if separate logging for this application is configured in the configuration),
• / model - a directory for domain models (also loaded at startup, but after initialization),
• / setup - the js files located in this directory will be run only 1 time when restarting IAS or the entire server, this place is necessary for update or migration scripts, which are necessary to maintain a full application life cycle during its operation,
• / tasks - a directory for placing scheduled tasks, which are of two types: run periodically, at a certain interval or run at the appointed time,
• / tmp - directory for temporary files.

The following directories will appear in the next versions ( issue # 195 ):

• / client - the directory of source code distribution of the client part,
• / static - the collected clients will then be placed in /static, and as the collector, it will be possible to use several of the most common tools.

IAS functionality

Let this article remain introductory, so, I will not now describe in detail the entire arsenal of IAS and overload the reader. I confine myself to a simple enumeration of the main one: registration with a service (daemon), transparent scaling to many processes and many servers, built-in system of users and sessions (including anonymous and authenticated), support for SSE (Server-Sent Events) and web sockets with the system channels and subscribing to messages, support for proxying requests, URL rewriting, introspection of the network API and issuing directory indexes, controlling access to directories via access.js (analogous to .htaccess), configuring applications, logging, scrolling logs, sending articles and cached in memory, gzip kompessiya support HTTP headers «if-modified-since» and HTTP 304 (Not Modified), HTTPS support, streaming files with support for recoil in parts (from a specified location to a specified location that players usually use, for example, HTML5 video tag via HTTP Content-Range and Accept-Ranges headers), there are scripts for quick server deployment for clean machines (CentOS, Ubuntu , Debian), built-in interprocess communication mechanisms via IPC, HTTP and ZeroMQ, a special API for synchronizing the state between processes, a built-in mechanism for monitoring server health, a pending task launch subsystem, the ability to generate workers (parallel processes), string validation data structures and database schemas, generating data structures from schemas for SQL-compatible DBMSs, automatic error and long stack handling, garbage collection optimization, sandboxes shielding, HTTP basic authentication support,

Conclusion

Impress (IAS) is actively developing, from 4 to 7 minor versions appear every week. Now version 0.1.195 is relevant and version 0.2 is on the way, in which we fix the application structure and the basic API, observing backward compatibility for all 0.2.x versions. In 0.2.x, we will only deal with optimization issues and bug fixes, and expanding functionality will be possible only if this does not require a redesign of applications based on 0.2.x. All major innovations and experiments will be introduced simultaneously in the 0.3.x branch. I invite everyone to develop the project, and for my part, I promise to maintain the code, at least as long as it is relevant. Version 1.0 will only appear when I understand that independent developers are fully able to support the code. Now the documentation is being prepared, which until then was impossible due to the fact that the structure and architecture changed frequently, I will publish a link to it according to the readiness of version 0.2. Prior to this, IAS can be examined in more detail using the examples that are installed with IAS as the default application.

A few figures as of 2015-01-11: downloads from npm yesterday: 1,338, this week: 5,997, last month: 21,223, stars on github: 168, contribution to the repository: 8 people, lines of code: 6 120, the size of the sources: 207 Kb (of which the kernel: 118Kb), the average cyclical complexity of the code: 20, the number of closed issues in github: 151, open issues: 9, the date of the first published version: 2013-06-08, number of builds in Travis CI: 233, github commits: 468.

References

NPM: www.npmjs.com/package/impress
Github: github.com/tshemsedinov/impress