June 17, 2016 at 14:26Data synchronization in real-time applications with Theron
- Tutorial
Sometimes I draw myself a graph of what the architecture of modern systems should look like and find those moments of the development process that can be improved and those practices that can be applied to improve these processes. After another such iteration, I was once again convinced that there are amazing frameworks and methodologies for developing both the server and client parts, but the data synchronization between the client, server and database does not work as modern realities require: quick response to changes in the state of the system, distribution and asynchrony of data processing, reuse of previously processed data.
In recent years, the requirements for modern applications and their development methods have changed significantly. Most of these applications use an asynchronous model consisting of many loosely coupled components (microservices). Users want the application to run smoothly and always up to date (data must be synchronized at any time), in other words, users feel more comfortable when they don’t need to click the Update button or restart the application completely, if something went wrong. Under the cut, there is a bit of theory and practice and a full-fledged open source application with a development stack of React, Redux / Saga, Node, TypeScript and our Theron project.
Rick and Morty. Rick opens up many portals.
I used various services for real-time synchronization and storage of data, most of which are mentioned in this article . But each time, as soon as the application developed into a more complex product, it became obvious that the system is too much dependent on one service provider and it does not have the necessary flexibility provided by the creation of its microarchitecture with many diversified satellite services, the use of classical databases ( SQL and NoSQL) and coding, in return for BaaS designers and control panels. This approach is indeed more complicated at the initial stage of prototype development, but it pays for itself in the future.
The result of my research was Theron. Theron- A service for creating modern real-time applications. Theron Online Data Warehouse continuously broadcasts changes to the database based on a query to it. In just over four months, we implemented a basic architecture with a small team of two developers, the main criteria of which are:
I liked the functional approach back when I became acquainted with one of the oldest functional programming languages, focused on symbolic computing Refal . Later, without realizing it, I started using the reactive programming paradigm, and, over time, most of my work was built on these approaches.
Theron is based on ReactiveX . The fundamental concept at Theron is reactive channels that provide a flexible way to transmit data to different segments of users. Theron uses the classic Pub / Sub design pattern . To create a new channel (the number is unlimited) and stream data, just create a new subscription.
After installation, import Theron and create a new client:
Creating a new client does not establish a new WebSocket connection and does not start data synchronization. A connection is established only when a subscription is created, provided that there is no other active connection. That is, as part of reactive programming, the Theron client and channels are “ cold observable ” objects.
Create a new subscription:
When the channel is no longer needed - unsubscribe:
Sending data to clients subscribed to this channel from the server side (Node.js) is also simple:
Theron uses an exponential backoff (enabled by default) when a connection is lost or if non-critical errors occur (eng.) : Errors when you can re-subscribe to a channel.
The implementation of many algorithms in the framework of reactive programming is elegant and simple, for example, the exponential backoff in the Theron client library looks something like this:
As mentioned above, Theron is a reactive data warehouse: a change notification system that continuously broadcasts updates via secure channels for your application based on a regular SQL query to the database. Theron parses the database query and sends data artifacts that can be used to recreate the original data.
Let's see how this works on the example of the life cycle of a simple list consisting of the first three elements, arranged in alphabetical order:
Before we continue, connect the database to Theron by entering the data to access it in the control panel:
Internal locking device (locking ) Databases are a big topic for a separate article in the future. Theron is a distributed system, so the pool of database connections is limited to 10 (with the possibility of increasing to 20) common connections.
1. Creating a new subscription
Theron works with SQL queries, so your server should return not the result of the query, but the original SQL query. For example, in our case, the JSON server response might look like this:
On the client side, we begin the data translation for our SQL query by creating a new subscription:
Theron will send a '/ todos' GET request to your server, check the validity of the returned SQL query and start translating the initial instructions with the necessary data if the request has not previously been cached on the client side.
Instructions TheronRowArtefact - it is a regular JavaScript object with `payload` the data itself and the type of instruction` type`. The main types of instructions:
Suppose that several elements A , B , C already exist in the database . Then the change in the state of the client can be represented as follows (it was on the left, it became on the right):
Theron instructions for this state:
Each sync block starts and ends with the BEGIN_TRANSACTION and COMMIT_TRANSACTION instructions. To correctly sort items on the client side, Theron additionally sends data about the previous item.
2. The user renames the element A (1) to D (1)
Suppose that the user renames the element A (1) to D (1) . Since the SQL query arranges the elements in alphabetical order, the elements will be sorted, and the state of the client will change as follows:
Theron instructions for this state:
3. The user creates a new element A (4)
Assume that the user creates a new element A (4) . Since our SQL query limits the data to the first three elements, then on the client side, the D (1) element will be deleted , and the client state will change as follows:
Theron instructions for this state:
4. User deletes item D (1)
Suppose the user deletes item D (1) from the database. In this case, Theron will not send new instructions, since this change in the database does not affect the data returned by our SQL query, and therefore does not affect the state of the client:
Processing Client-Side Instructions
Now that we know how Theron works with data, we can implement the logic to recreate client-side data. The algorithm is quite simple: we will use the type of instructions and metadata of the previous element to correctly position the elements in the array. In a real application, you need to use, for example, the Immutable.js library for working with arrays and the scan operator is an example .
Sometimes it’s better to study based on ready-made examples: therefore, here is the promised application published under the MIT license - https://github.com/therondb/figure . Figure is a service for working with HTML forms in static sites; development stack - React, Redux / Saga, Node, TypeScript and, of course, Theron. For example, we use Figure to create a list of subscribers to our blog and documentation site ( https://github.com/therondb/therondb.com ):
In addition to fixing a hypothetical ton of errors and classic writing of client libraries for popular platforms, we are working on the selection of an inverse proxy server and balancer as an independent component. The idea is to be able to create server-side APIs that can be accessed both through regular HTTP requests and through a permanent WebSocket connection. In the next article about Theron architecture, I will write about this in more detail.
Our team is small, but energetic, and we love to experiment. Theron is under active development: there are many ideas and points that need to be implemented and improved. We will listen to any criticism with pleasure, take advice and constructively discuss it.
In recent years, the requirements for modern applications and their development methods have changed significantly. Most of these applications use an asynchronous model consisting of many loosely coupled components (microservices). Users want the application to run smoothly and always up to date (data must be synchronized at any time), in other words, users feel more comfortable when they don’t need to click the Update button or restart the application completely, if something went wrong. Under the cut, there is a bit of theory and practice and a full-fledged open source application with a development stack of React, Redux / Saga, Node, TypeScript and our Theron project.
Rick and Morty. Rick opens up many portals.
I used various services for real-time synchronization and storage of data, most of which are mentioned in this article . But each time, as soon as the application developed into a more complex product, it became obvious that the system is too much dependent on one service provider and it does not have the necessary flexibility provided by the creation of its microarchitecture with many diversified satellite services, the use of classical databases ( SQL and NoSQL) and coding, in return for BaaS designers and control panels. This approach is indeed more complicated at the initial stage of prototype development, but it pays for itself in the future.
The result of my research was Theron. Theron- A service for creating modern real-time applications. Theron Online Data Warehouse continuously broadcasts changes to the database based on a query to it. In just over four months, we implemented a basic architecture with a small team of two developers, the main criteria of which are:
- Quickly create new applications and seamlessly migrate existing ones to Theron.
- The use of modern practices in the creation of asynchronous, distributed and fault-tolerant systems and the isomorphism of system components.
- Distributed low-level integration with databases such as Postgres and Mongo.
- Easy integration with modern frameworks such as React, Angular and their friends: ReactiveX, Redux etc.
- Focus on solving the problem of data synchronization, rather than providing a complete development stack and the subsequent " vendor locking ".
- The basic logic of applications (including authentication and access rights) must be implemented by developers on their side.
Reactive channels
I liked the functional approach back when I became acquainted with one of the oldest functional programming languages, focused on symbolic computing Refal . Later, without realizing it, I started using the reactive programming paradigm, and, over time, most of my work was built on these approaches.
Theron is based on ReactiveX . The fundamental concept at Theron is reactive channels that provide a flexible way to transmit data to different segments of users. Theron uses the classic Pub / Sub design pattern . To create a new channel (the number is unlimited) and stream data, just create a new subscription.
After installation, import Theron and create a new client:
import { Theron } from 'theron';
const theron = new Theron('https://therondb.com', { app: 'YOUR_APP_NAME' });
Creating a new client does not establish a new WebSocket connection and does not start data synchronization. A connection is established only when a subscription is created, provided that there is no other active connection. That is, as part of reactive programming, the Theron client and channels are “ cold observable ” objects.
Create a new subscription:
const channel = theron.join('the-world');
const subscription = channel.subscribe(
message => {
console.log(message.payload);
},
err => {
console.log(err);
},
() => {
console.log('done');
}
);
When the channel is no longer needed - unsubscribe:
subscription.unsubscribe();
Sending data to clients subscribed to this channel from the server side (Node.js) is also simple:
import { Theron } from 'theron';
const theron = new Theron('https://therondb.com', { app: 'YOUR_APP_NAME', secret: 'YOUR_SECRET_KEY' });
theron.publish('the-world', { message: 'Greatings from Cybertron!' }).subscribe(
res => {
console.log(res);
},
err => {
console.log(err);
},
() => {
console.log('done');
},
);
Theron uses an exponential backoff (enabled by default) when a connection is lost or if non-critical errors occur (eng.) : Errors when you can re-subscribe to a channel.
The implementation of many algorithms in the framework of reactive programming is elegant and simple, for example, the exponential backoff in the Theron client library looks something like this:
let attemp = 0;
const rescueChannel = channel.retryWhen(errs =>
errs.switchMap(() => Observable.timer(Math.pow(2, attemp + 1) * 500).do(() => attemp++))
).do(() => attemp = 0);
Database Integration
As mentioned above, Theron is a reactive data warehouse: a change notification system that continuously broadcasts updates via secure channels for your application based on a regular SQL query to the database. Theron parses the database query and sends data artifacts that can be used to recreate the original data.
Theron is currently integrated with Postgres; Integration with Mongo in the development process.
Let's see how this works on the example of the life cycle of a simple list consisting of the first three elements, arranged in alphabetical order:
Before we continue, connect the database to Theron by entering the data to access it in the control panel:
Internal locking device (locking ) Databases are a big topic for a separate article in the future. Theron is a distributed system, so the pool of database connections is limited to 10 (with the possibility of increasing to 20) common connections.
1. Creating a new subscription
Theron works with SQL queries, so your server should return not the result of the query, but the original SQL query. For example, in our case, the JSON server response might look like this:
{ "query": "SELECT * FROM todos ORDER BY name LIMIT 3" }
On the client side, we begin the data translation for our SQL query by creating a new subscription:
import { Theron } from 'theron';
const theron = new Theron('https://therondb.com', { app: 'YOUR_APP_NAME' });
const subscription = theron.watch('/todos').subscribe(
action => {
console.log(action); // Инструкции Theron'а
},
err => {
console.log(err);
},
() => {
console.log('complete');
}
);
Theron will send a '/ todos' GET request to your server, check the validity of the returned SQL query and start translating the initial instructions with the necessary data if the request has not previously been cached on the client side.
Instructions TheronRowArtefact - it is a regular JavaScript object with `payload` the data itself and the type of instruction` type`. The main types of instructions:
- ROW_ADDED - a new element has been added.
- ROW_REMOVED - The item has been deleted.
- ROW_MOVED - The item has been changed.
- ROW_CHANGED - The item has been changed.
- BEGIN_TRANSACTION - new synchronization block.
- COMMIT_TRANSACTION - Synchronization completed successfully.
- ROLLBACK_TRANSACTION - An error occurred while synchronizing.
Suppose that several elements A , B , C already exist in the database . Then the change in the state of the client can be represented as follows (it was on the left, it became on the right):
| Id | Name | Id | Name |
|---|---|---|---|
| 1 | A | ||
| 2 | B | ||
| 3 | C |
Theron instructions for this state:
{ type: BEGIN_TRANSACTION }{ type: ROW_ADDED, payload: { row: { id: 1, name: 'A' }, prevRowId: null } }{ type: ROW_ADDED, payload: { row: { id: 2, name: 'B' }, prevRowId: 1 }{ type: ROW_ADDED, payload: { row: { id: 3, name: 'C' }, prevRowId: 2 } }{ type: BEGIN_TRANSACTION }
Each sync block starts and ends with the BEGIN_TRANSACTION and COMMIT_TRANSACTION instructions. To correctly sort items on the client side, Theron additionally sends data about the previous item.
2. The user renames the element A (1) to D (1)
Suppose that the user renames the element A (1) to D (1) . Since the SQL query arranges the elements in alphabetical order, the elements will be sorted, and the state of the client will change as follows:
| Id | Name | Id | Name |
|---|---|---|---|
| 1 | A | 2 | B |
| 2 | B | 3 | C |
| 3 | C | 1 | D |
Theron instructions for this state:
{ type: BEGIN_TRANSACTION }{ type: ROW_CHANGED, payload: { row: { id: 1, name: 'D' }, prevRowId: 3 } }{ type: ROW_MOVED, payload: { row: { id: 1, name: 'D' }, prevRowId: 3 } }{ type: ROW_MOVED, payload: { row: { id: 2, name: 'B' }, prevRowId: null } }{ type: ROW_MOVED, payload: { row: { id: 3, name: 'C' }, prevRowId: 2 } }{ type: COMMIT_TRANSACTION }
3. The user creates a new element A (4)
Assume that the user creates a new element A (4) . Since our SQL query limits the data to the first three elements, then on the client side, the D (1) element will be deleted , and the client state will change as follows:
| Id | Name | Id | Name |
|---|---|---|---|
| 2 | B | 4 | A |
| 3 | C | 2 | B |
| 1 | D | 3 | C |
Theron instructions for this state:
{ type: BEGIN_TRANSACTION }{ type: ROW_ADDED, payload: { row: { id: 4, name: 'A' }, prevRowId: null } }{ type: ROW_MOVED, payload: { row: { id: 2, name: 'B' }, prevRowId: 4 } }{ type: ROW_MOVED, payload: { row: { id: 3, name: 'C' }, prevRowId: 2 } }{ type: ROW_REMOVED, payload: { row: { id: 1, name: 'D' }, prevRowId: 3 } }{ type: COMMIT_TRANSACTION }
4. User deletes item D (1)
Suppose the user deletes item D (1) from the database. In this case, Theron will not send new instructions, since this change in the database does not affect the data returned by our SQL query, and therefore does not affect the state of the client:
| Id | Name | Id | Name |
|---|---|---|---|
| 4 | A | 4 | A |
| 2 | B | 2 | B |
| 3 | C | 3 | C |
Processing Client-Side Instructions
Now that we know how Theron works with data, we can implement the logic to recreate client-side data. The algorithm is quite simple: we will use the type of instructions and metadata of the previous element to correctly position the elements in the array. In a real application, you need to use, for example, the Immutable.js library for working with arrays and the scan operator is an example .
import { ROW_ADDED, ROW_CHANGED, ROW_MOVED, ROW_REMOVED } from 'theron';
let todos = [];
const subscription = theron.watch('/todos').subscribe(
action => {
switch (action.type) {
case ROW_ADDED:
const index = nextIndexForRow(rows, action.prevRowId)
if (index !== -1) {
rows.splice(index, 0, action.row);
}
break;
case ROW_CHANGED:
const index = indexForRow(rows, action.row.id);
if (index !== -1) {
rows[index] = action.row;
}
break;
case ROW_MOVED:
const index = indexForRow(rows, action.row.id);
if (index !== -1) {
const row = list.splice(curPos, 1)[0];
const newIndex = nextIndexForRow(rows, action.prevRowId);
rows.splice(newIndex, 0, row);
}
break;
case ROW_REMOVED:
const index = indexForRow(rows, action.row.id);
if (index !== -1) {
list.splice(index, 1);
}
break;
}
},
err => {
console.log(err);
}
);
function indexForRow(rows, rowId) {
return rows.findIndex(row => row.id === rowId);
}
function nextIndexForRow(rows, prevRowId) {
if (prevRowId === null) {
return 0;
}
const index = indexForRow(rows, prevRowId);
if (index === -1) {
return rows.length;
} else {
return index + 1;
}
}
Examples time
Sometimes it’s better to study based on ready-made examples: therefore, here is the promised application published under the MIT license - https://github.com/therondb/figure . Figure is a service for working with HTML forms in static sites; development stack - React, Redux / Saga, Node, TypeScript and, of course, Theron. For example, we use Figure to create a list of subscribers to our blog and documentation site ( https://github.com/therondb/therondb.com ):
Conclusion
In addition to fixing a hypothetical ton of errors and classic writing of client libraries for popular platforms, we are working on the selection of an inverse proxy server and balancer as an independent component. The idea is to be able to create server-side APIs that can be accessed both through regular HTTP requests and through a permanent WebSocket connection. In the next article about Theron architecture, I will write about this in more detail.
Our team is small, but energetic, and we love to experiment. Theron is under active development: there are many ideas and points that need to be implemented and improved. We will listen to any criticism with pleasure, take advice and constructively discuss it.