Multithreaded file loader in JS (jQuery)
Good day, colleagues. In this article, I will describe the experience of creating a multi-threaded file downloader (with a limited server load) on JS (jQuery).
Most recently, I had a problem (the solution of which I want to share with you): to make, in the admin panel, the ability to select and upload more than one file at a time. The task seems to be trivial and not difficult, but in the end my solution seemed to me rather interesting, since no analogues were found.
Since the task was simple, and, of course, plenty of solutions have already been written, I turned to Google for help. But, the search did not give positive results - most of the proposed solutions use Flash (which, due to some specifics, did not allow me to use such solutions), or the written libraries in JS were very huge and, most unfortunate, non-working. I had to assemble a bike.
The task was urgent, so an urgent solution was needed. I don’t think for long, I screwed the multiple attribute to the input field (available with HTML5).
This was followed by small changes to the handler to receive not one file, but an array of files - and the problem is solved! (my naivety (inexperience) was not a chapel).
As many have already thought with a laugh, five hundred thirds answered the first, normal, batch of nginx files. We had to think further.
Since the past decision was made beautifully and conveniently for admins, it was decided to build on it. It was necessary to solve the problem of error No. 503, which nginx returned due to lengthy file processing.
Half a minute to think it over and a new solution appears: we will send ajax not all the files at once, but one at a time.
The solution was approximately as follows:
Everything is simple: we sort through an array of files (which is located in the desired input), create an instance of the class for working with files (more on this later) and ajax send a request to the server. It is worth paying attention to the " async: false " parameter - here we set the synchronous execution of the ajax request, since the asynchronous one will create a lot of requests to the server for us, which we will easily put it on ourselves.
The solution works, there is no error, but one problem is that it works slowly. And then I came up with the idea of another solution to the problem.
To speed up the upload of files to the server, you can increase the number of requests in which files will be transferred. Such a solution rests in the solution of two problems:
1). With a large number of requests, I quickly get to hell with my server.
2) A large amount of simultaneously downloaded files will clog us the entire channel.
Judging by the problems, our bootloader should consider the number of running requests and the amount of data transferred, and under some conditions, wait for some requests to finish, for others to start.
Let's get down to the code:
Legend:
dataArr - an array with data to send.
fStek - write timeout identifiers here, to further stop recursion and clear memory from incomplete functions.
vStek - the number of threads invoked .
deley - delay recursion of a function that checks flows and volumes.
debug - debug mode. It is necessary for debugging, but in this example I deleted all its signs.
maxFilesSize - maximum amount of downloaded files
maxThreads - maximum number of threads.
The second considered function FilesUploader.controller () will introduce complete clarity in variables (especially fStek and deley ). In the meantime, let's move on to initializing the class:
It is on this function that the processing of the button click event in the form is hung. The function’s work is simple: we go through the files ( jQuery.each ) entered in the input and add ( FilesUploader.dataArr.push (f) ) an entry about each in the array. Next, we call the controller, which is the most important and complex part of the system:
In the first line of the function, we call the same function asynchronously (after a certain period of time) (i.e., create a recursion), and put the identifier of the called function into a variable, in order to interrupt its execution.
Next is the condition for checking threads.
After receiving the file from the array ( FilesUploader.dataArr.pop () ) we check it for presence.
1. If there is no file, then we “kill” the called functions by their identifier ( clearTimeout (FilesUploader.fStek.pop ()) );
2. If the file exists, we check for the amount of downloaded files, and if it is exceeded, we return the file back to the stack and exit the function, otherwise, if it is not exceeded: we take away the volume, increase the counter of the streamed streams and call the following function (FilesUploader.worker (item) ).
To send a file to the server using ajax, you need to put the file data (file.append ()) in it into an instance of the FormData class.
Next, we call the $ .ajax function , which will transfer our file to the loader on the server. Upon completion of each request (the complete () function), you need to increase the allowable volume and reduce the number of executable threads (which is done in the lines “ FilesUploader.maxFilesSize + = this.fileData.size ” and “ FilesUploader.vStek— “).
And the final touch is the console output function and the closing bracket:
That's all - the class for multithreaded uploading files to the server is ready. Next, you should already set, depending on the server configuration, the permissible number of threads and the volume of simultaneously downloaded files - and you can work.
Most recently, I had a problem (the solution of which I want to share with you): to make, in the admin panel, the ability to select and upload more than one file at a time. The task seems to be trivial and not difficult, but in the end my solution seemed to me rather interesting, since no analogues were found.
Solution No. 1
Since the task was simple, and, of course, plenty of solutions have already been written, I turned to Google for help. But, the search did not give positive results - most of the proposed solutions use Flash (which, due to some specifics, did not allow me to use such solutions), or the written libraries in JS were very huge and, most unfortunate, non-working. I had to assemble a bike.
Decision number 2
The task was urgent, so an urgent solution was needed. I don’t think for long, I screwed the multiple attribute to the input field (available with HTML5).
This was followed by small changes to the handler to receive not one file, but an array of files - and the problem is solved! (my naivety (inexperience) was not a chapel).
As many have already thought with a laugh, five hundred thirds answered the first, normal, batch of nginx files. We had to think further.
Decision No. 3
Since the past decision was made beautifully and conveniently for admins, it was decided to build on it. It was necessary to solve the problem of error No. 503, which nginx returned due to lengthy file processing.
Half a minute to think it over and a new solution appears: we will send ajax not all the files at once, but one at a time.
The solution was approximately as follows:
jQuery.each($('#fileinput')[0].files, function(i, f) {
var file = new FormData();
file.append('file', f);
$.ajax({
url: 'uploader.php',
data: file,
async : false,
contentType: false,
processData: false,
dataType: "JSON",
type: 'POST',
beforeSend: function() {},
complete: function(event, request, settings) {},
success: function(data){ }
});
});
Everything is simple: we sort through an array of files (which is located in the desired input), create an instance of the class for working with files (more on this later) and ajax send a request to the server. It is worth paying attention to the " async: false " parameter - here we set the synchronous execution of the ajax request, since the asynchronous one will create a lot of requests to the server for us, which we will easily put it on ourselves.
The solution works, there is no error, but one problem is that it works slowly. And then I came up with the idea of another solution to the problem.
Decision No. 4
To speed up the upload of files to the server, you can increase the number of requests in which files will be transferred. Such a solution rests in the solution of two problems:
1). With a large number of requests, I quickly get to hell with my server.
2) A large amount of simultaneously downloaded files will clog us the entire channel.
Judging by the problems, our bootloader should consider the number of running requests and the amount of data transferred, and under some conditions, wait for some requests to finish, for others to start.
Let's get down to the code:
FilesUploader = {
dataArr : new Array(),
fStek : new Array(),
vStek : 0,
deley : 100,
debug : true,
maxFilesSize : 1024*1024*10,
maxThreads : 10,
Legend:
dataArr - an array with data to send.
fStek - write timeout identifiers here, to further stop recursion and clear memory from incomplete functions.
vStek - the number of threads invoked .
deley - delay recursion of a function that checks flows and volumes.
debug - debug mode. It is necessary for debugging, but in this example I deleted all its signs.
maxFilesSize - maximum amount of downloaded files
maxThreads - maximum number of threads.
The second considered function FilesUploader.controller () will introduce complete clarity in variables (especially fStek and deley ). In the meantime, let's move on to initializing the class:
run : function() {
jQuery.each($('#fileinput')[0].files, function(i, f) {
FilesUploader.dataArr.push(f);
});
FilesUploader.controller();
},
It is on this function that the processing of the button click event in the form is hung. The function’s work is simple: we go through the files ( jQuery.each ) entered in the input and add ( FilesUploader.dataArr.push (f) ) an entry about each in the array. Next, we call the controller, which is the most important and complex part of the system:
controller : function() {
FilesUploader.fStek.push(setTimeout(FilesUploader.controller, FilesUploader.deley));
if(FilesUploader.vStek>=this.maxThreads) { return; }
item = FilesUploader.dataArr.pop();
if(item) {
if(FilesUploader.maxFilesSize-item.size < 0) {
FilesUploader.dataArr.push(item);
return;
}
FilesUploader.maxFilesSize-=item.size;
FilesUploader.vStek++;
FilesUploader.worker(item);
} else clearTimeout(FilesUploader.fStek.pop());
},
In the first line of the function, we call the same function asynchronously (after a certain period of time) (i.e., create a recursion), and put the identifier of the called function into a variable, in order to interrupt its execution.
Next is the condition for checking threads.
After receiving the file from the array ( FilesUploader.dataArr.pop () ) we check it for presence.
1. If there is no file, then we “kill” the called functions by their identifier ( clearTimeout (FilesUploader.fStek.pop ()) );
2. If the file exists, we check for the amount of downloaded files, and if it is exceeded, we return the file back to the stack and exit the function, otherwise, if it is not exceeded: we take away the volume, increase the counter of the streamed streams and call the following function (FilesUploader.worker (item) ).
worker : function(item) {
var file = new FormData();
file.append('file', item);
$.ajax({
url: 'uploader.php',
data: file,
contentType: false,
processData: false,
dataType: "JSON",
type: 'POST',
beforeSend: function() {},
complete: function(event, request, settings) {
FilesUploader.maxFilesSize+=this.fileData.size;
FilesUploader.vStek--;
},
success: function(data){ }
});
},
To send a file to the server using ajax, you need to put the file data (file.append ()) in it into an instance of the FormData class.
Next, we call the $ .ajax function , which will transfer our file to the loader on the server. Upon completion of each request (the complete () function), you need to increase the allowable volume and reduce the number of executable threads (which is done in the lines “ FilesUploader.maxFilesSize + = this.fileData.size ” and “ FilesUploader.vStek— “).
And the final touch is the console output function and the closing bracket:
out : function(message) {
if(console.log && this.debug) console.log(message);
}
}
That's all - the class for multithreaded uploading files to the server is ready. Next, you should already set, depending on the server configuration, the permissible number of threads and the volume of simultaneously downloaded files - and you can work.