node-seq in a new way (again about asynchrony)
Hello, Habr! I am writing to you as slopok. Indeed, while spaceships with vertical take-off and landing plow the expanses of the oceans, and the most impatient ones use ES6 features in their projects, I brought you another library to make life easier for the asynchronous.
Of course, for a long time already there are a million implementations of promises, and for those who wish, async. There is also a well-known Seq library in narrow circles from the notorious comrade Substack. I started using it almost from the first days of my javascript and used it wherever I could. The approach proposed by this library seems to me more understandable and logical for curbing asynchronous noodles than the approach used, for example, in async. See for yourself:
Everything is so simple and clear that even explaining laziness. Unfortunately, the library has not been developed and maintained for a long time. In addition, during the time of use, a list of bugs was accumulated which I came across one way or another, a list of features I wanted, a list of complaints - because not everything worked as well as in my imagination. Once, once again I stumbled upon the imperfection of the world and decided that This Moment had come. It's time to fork and fix. I often do this, but what this library does seemed to me real Magic.
And here I am happy and determined to do git clone, cd, gvim, and try to understand what is happening here. I do not understand. The author uses a couple of his libraries and for enlightenment, you must first deal with them. After a couple of hours I get bored and I discover a Fatal Flaw. No sooner said than done. I sit down and write from scratch the Dream Library. Oddly enough, there was no magic in all of this. The prototype was ready for the evening. Then, for some time, I finished it already using it in a real project, replacing it with Seq completely. And so it turned out what happened. Let's get acquainted.
YAFF - Yet Another Flow Framework.
Overall, I tried to make it compatible with Seq. And most of the code migrated simply by replacing the import (it was var Seq = require ('seq'); it became var Seq = require ('yaff');). Of course, something else had to be replaced. Seq uses the .catch () method to catch fleas. For example, the above piece of code can be changed like this:
This construction is terrible in that after we “caught” the error we can continue. You can not do it this way. Firstly, it is not clear what to do if parEach (or other similar methods) throw a few errors. To catch only the first? To catch everything? And if we have already gone far down and in some parEach an error popped up in the timer? And if below we already have no catch? But what if the catch below is not prepared for error handling that will pop out from forEach by timer? There are many unanswered questions. Therefore, I decided that in each YAFF there should be only one construct for error handling and it should be at the end. And in order not to violate the tradition of nodejs, let it also process the result. It turns out beauty. Make sure:
If we assume that this is all inside the asynchronous function, then in finally we can just throw the callback provided to us, a la:
Conveniently? I like it too. In general, this library is built with respect to the concept of an array of arguments (it was worth writing about it from the very beginning). And all the methods that are here one way or another change this stack by applying to it or to its individual elements. Say, a flock of functions wrapped in par will take one element from the stack, in the order in which these par are written and only after all par have shot callbacks (and the callback inside all methods is this) YAFF will go to what's next queues. Suppose further that we have several functions wrapped in seq. YAFF will apply the entire stack to them and replace it with something that returns a wrapped function to the callback. Here is the code to make it clear:
Of course, if someone calls his callback with a non-zero first argument (error), then YAFF immediately spits on all the functions that are still there, and goes to do what is written in finally. If you forget to write finally or consider that your error code certainly cannot be YAFF, in case of an error, it will unceremoniously throw an exception. So it’s better that finally be.
Also, there are all kinds of synchronous functions for working with the argument stack as an array: push, pull, filter, set, reduce, flatten, extend, unflatten, empty, shift, unshift, splice, reverse, save (name), load (fromName ) Fuf, like that. The names speak for themselves, but if that - do not hesitate to ask and look at the source. There is one file ( main.js ) and everything is very simple.
Well, of course, for what it was all about, asynchronous functions for processing data arrays: forEach (YAFF will not wait for this block to finish processing, the results of this block will not affect the stack. YAFF will immediately go to the next handler in the chain), seqEach , parEach (YAFF will wait until all functions are shot, but the results will not affect the stack). seqMap, parMap, seqFilter, parFilter - do what you expect; YAFF waits until they work out and the results of these blocks replace those that were on the stack before. In addition, for all methods with the par prefix, you can specify a number after the function. This number is the limit of simultaneously working asynchronous functions. Something like this:
In this example, we asynchronously resize a stack of photos. In order not to overload the server, we resize no more than 10 images at a time for each client. unflatten is needed in order to collect photographs spread across the stack into an array which will be one argument for a callback.
YAFF also has the mseq and mpar methods - this is a curtsy towards async users. These methods accept an array of functions that will be executed sequentially or in parallel. With the same success, you could write a bunch of seq () and par (), but sometimes you want to generate functions dynamically. We still have a functional language, right?
To completely confuse you, I came up with the following example and drew a picture (in the desperate hope that it would clarify everything):
I hope that I was able to clearly explain what I wanted; you are inspired by the idea and I'm not the only one who does not understand why async is needed.
All specific wishes and suggestions are best written in the form of issues (in English, do not be shy - I also don't know him well, we will train literature together) on a github or even in the form of pool requests .
Oh yes, the library is at npmjs.org .
PS I just added a synchronous apply method in a fit of passion - now all other synchronous methods can be thrown away. But I will leave it for convenience and compatibility.
Of course, for a long time already there are a million implementations of promises, and for those who wish, async. There is also a well-known Seq library in narrow circles from the notorious comrade Substack. I started using it almost from the first days of my javascript and used it wherever I could. The approach proposed by this library seems to me more understandable and logical for curbing asynchronous noodles than the approach used, for example, in async. See for yourself:
var fs = require('fs');
var Hash = require('hashish');
var Seq = require('seq');
Seq()
.seq(function () {
fs.readdir(__dirname, this);
})
.flatten()
.parEach(function (file) {
fs.stat(__dirname + '/' + file, this.into(file));
})
.seq(function () {
var sizes = Hash.map(this.vars, function (s) { return s.size })
console.dir(sizes);
});
Everything is so simple and clear that even explaining laziness. Unfortunately, the library has not been developed and maintained for a long time. In addition, during the time of use, a list of bugs was accumulated which I came across one way or another, a list of features I wanted, a list of complaints - because not everything worked as well as in my imagination. Once, once again I stumbled upon the imperfection of the world and decided that This Moment had come. It's time to fork and fix. I often do this, but what this library does seemed to me real Magic.
And here I am happy and determined to do git clone, cd, gvim, and try to understand what is happening here. I do not understand. The author uses a couple of his libraries and for enlightenment, you must first deal with them. After a couple of hours I get bored and I discover a Fatal Flaw. No sooner said than done. I sit down and write from scratch the Dream Library. Oddly enough, there was no magic in all of this. The prototype was ready for the evening. Then, for some time, I finished it already using it in a real project, replacing it with Seq completely. And so it turned out what happened. Let's get acquainted.
YAFF - Yet Another Flow Framework.
Overall, I tried to make it compatible with Seq. And most of the code migrated simply by replacing the import (it was var Seq = require ('seq'); it became var Seq = require ('yaff');). Of course, something else had to be replaced. Seq uses the .catch () method to catch fleas. For example, the above piece of code can be changed like this:
var fs = require('fs');
var Hash = require('hashish');
var Seq = require('seq');
Seq()
.seq(function () {
fs.readdir(__dirname, this);
})
.flatten()
.parEach(function (file) {
fs.stat(__dirname + '/' + file, this.into(file));
})
.catch(function (err)(
console.error(err);
))
.seq(function () {
var sizes = Hash.map(this.vars, function (s) { return s.size })
console.dir(sizes);
});
This construction is terrible in that after we “caught” the error we can continue. You can not do it this way. Firstly, it is not clear what to do if parEach (or other similar methods) throw a few errors. To catch only the first? To catch everything? And if we have already gone far down and in some parEach an error popped up in the timer? And if below we already have no catch? But what if the catch below is not prepared for error handling that will pop out from forEach by timer? There are many unanswered questions. Therefore, I decided that in each YAFF there should be only one construct for error handling and it should be at the end. And in order not to violate the tradition of nodejs, let it also process the result. It turns out beauty. Make sure:
var fs = require('fs');
YAFF(['./', '../'])
.par(function (path) {
fs.readdir(path, this);
})
.par(function (path) {
fs.readdir(path, this);
})
.flatten()
.parMap(function (file) {
fs.stat(__dirname + '/' + file, this);
})
.map(function (stat) {
return stat.size;
})
.unflatten()
.finally(function (e, sizes) {
if (e)
throw e;
log(sizes);
});
If we assume that this is all inside the asynchronous function, then in finally we can just throw the callback provided to us, a la:
var listDirs = function (dirs, cb) {
YAFF(dirs)
[волшебные пузырьки]
.finally(cb);
};
Conveniently? I like it too. In general, this library is built with respect to the concept of an array of arguments (it was worth writing about it from the very beginning). And all the methods that are here one way or another change this stack by applying to it or to its individual elements. Say, a flock of functions wrapped in par will take one element from the stack, in the order in which these par are written and only after all par have shot callbacks (and the callback inside all methods is this) YAFF will go to what's next queues. Suppose further that we have several functions wrapped in seq. YAFF will apply the entire stack to them and replace it with something that returns a wrapped function to the callback. Here is the code to make it clear:
YAFF(['one', 'two', 'three'])
.par(function (one) {
this(null, one);
})
.par(function (two) {
this(null, two);
})
.par(function (three) {
this(null, three);
})
.seq(function (one, two, three) {
this(null, one, three);
})
.seq(function (one, three) {
this(null, null);
})
.seq(function () {
this(null, 'and so on and so forth');
})
Of course, if someone calls his callback with a non-zero first argument (error), then YAFF immediately spits on all the functions that are still there, and goes to do what is written in finally. If you forget to write finally or consider that your error code certainly cannot be YAFF, in case of an error, it will unceremoniously throw an exception. So it’s better that finally be.
Also, there are all kinds of synchronous functions for working with the argument stack as an array: push, pull, filter, set, reduce, flatten, extend, unflatten, empty, shift, unshift, splice, reverse, save (name), load (fromName ) Fuf, like that. The names speak for themselves, but if that - do not hesitate to ask and look at the source. There is one file ( main.js ) and everything is very simple.
Well, of course, for what it was all about, asynchronous functions for processing data arrays: forEach (YAFF will not wait for this block to finish processing, the results of this block will not affect the stack. YAFF will immediately go to the next handler in the chain), seqEach , parEach (YAFF will wait until all functions are shot, but the results will not affect the stack). seqMap, parMap, seqFilter, parFilter - do what you expect; YAFF waits until they work out and the results of these blocks replace those that were on the stack before. In addition, for all methods with the par prefix, you can specify a number after the function. This number is the limit of simultaneously working asynchronous functions. Something like this:
var resizePhotos(photos, cb) {
YAFF(photos)
.parMap(function (photo) {
asyncReisze(photo.image, photo.params, this);
}, 10)
.unaflatten()
.finally(cb);
}
In this example, we asynchronously resize a stack of photos. In order not to overload the server, we resize no more than 10 images at a time for each client. unflatten is needed in order to collect photographs spread across the stack into an array which will be one argument for a callback.
YAFF also has the mseq and mpar methods - this is a curtsy towards async users. These methods accept an array of functions that will be executed sequentially or in parallel. With the same success, you could write a bunch of seq () and par (), but sometimes you want to generate functions dynamically. We still have a functional language, right?
To completely confuse you, I came up with the following example and drew a picture (in the desperate hope that it would clarify everything):
YAFF(['./'])
.mseq([
function (path1) {
fs.readdir(path1, this);
},
function (arg) {
this(null, arg);
}
])
.flatten()
.parMap(function (file) {
fs.stat(__dirname + '/' + file, this);
})
.map(function (stat) {
return stat.size;
})
.unflatten()
.finally(function (e, sizes) {
log(sizes);
});
Big picture
I hope that I was able to clearly explain what I wanted; you are inspired by the idea and I'm not the only one who does not understand why async is needed.
All specific wishes and suggestions are best written in the form of issues (in English, do not be shy - I also don't know him well, we will train literature together) on a github or even in the form of pool requests .
Oh yes, the library is at npmjs.org .
PS I just added a synchronous apply method in a fit of passion - now all other synchronous methods can be thrown away. But I will leave it for convenience and compatibility.