Stream loading and video processing through node.js + php and ffmpeg - part one
I work in an outsourcing company and somehow the task was to download video with the possibility of further processing for the internal needs of the application: resize to the right size, convert to the desired format, pull out audio tracks (if any), video storyboard. In the end, the results need to be stored in the cloud for later use in the online editor. Requirements: scalability, unlimited video size, speed, cross-browser compatibility, visibility.
Since the topic is very extensive, I will divide it into sections:
1. Cross-browser compatibility
In order to ensure this requirement, it was impossible to use HTML5 file upload (it is not implemented in all browsers). Therefore, we went by creating the usual HTML4 form, which was submitted to the server with node.js. In order to get the current progress of loading / processing video, AJAX requests were created. The form included a randomly generated identifier string (for example, 32 characters). Node.js attached information on the current file to it and, with each state request, passed the following JSON:
Based on this, the user sees the download status.
2. User authorization
Since the frontend in PHP (user authorization is also taking place there), it was necessary to somehow transfer the session data to the node.js server. For these purposes, memcached was used. You can read about session substitution in PHP here . The bottom line is simple: sessions are saved in memcached, when loaded through the form, session_id is passed, which is read by node.js. Further node.js addresses memcached in search of the corresponding session, takes user_id, etc. There is one important point: the session_id in the form itself must be set first, because the POST request passes the parameters in the order they appear in HTML. That is, if we do this:
In this case, the file itself will come to node.js first, and then session_id, which is not good. After all, we first need to know who the file is coming from and cancel if the user is not authorized. If in the example above to swap file and session_id, then first we get the session, slow down the file download, and wait until the server checks to see if everything is ok with the user and only then continue to download. In node.js, you can pause request.
3. Streaming video
The problem is that not all videos can be processed right away. For example, according to the mp4 format specification, metadata goes at the end of the file. But we need to immediately know what sizes the video has, what tracks are in the file, duration, etc. Plus, some video formats during transcoding need to be able to access different parts of the source file. Based on this, there are two cases with two options each:
1. At the beginning of the download, when we pull the file information using the ffmpeg command using node.js:
In the variable ffmpeg_stdout we get all the tracks that are contained in the file or an error. If there is an error, then ffmpeg cannot process the file with a stream. In this case, we warn the user about this and download the entire file immediately, and only after that we perform the necessary operations.
2. In this case, speed is important, so we are trying to start the storyboard as soon as the video has just started loading. But if the format does not support streaming processing, then you have to wait until the full download.
It turns out the following algorithm: it is possible to extract metadata, then we try to process the video on the fly, if this turns out, then we are happy, in all other cases we are waiting for the full download. In practice, only a small number of videos need to be downloaded completely in order to begin processing.
That's all for today, tomorrow morning for work.
I would like to conclude on a major note: the possibilities are endless, you can distribute the load on several servers, show the results on the go, for example, pull the last finished frame from the video. It all works on Amazon EC2. In terms of speed, I can say that there was no way to check the download speed (the channel is weak), but not less than 10mbits, I think for sure. And on the load on the server: Middle server alone, one video pulls somewhere around 2Mbps of video, that is, the download has gone far ahead, and processing is not keeping up. Resize takes the most time. Storyboard and pulling audio tracks relatively quickly.
If you like the post, I will describe the implementation and the mechanism of work in more detail next time.
Screenshots of what happened. Immediately make a reservation, this is not a working option, so do not judge strictly:
Since the topic is very extensive, I will divide it into sections:
- Common problems, nuances that I had to face
- Downloading video (I’m probably not going to stop on this topic, since it has already been raised in this and this post.
- Video processing.
- Save to the cloud.
Part one
1. Cross-browser compatibility
In order to ensure this requirement, it was impossible to use HTML5 file upload (it is not implemented in all browsers). Therefore, we went by creating the usual HTML4 form, which was submitted to the server with node.js. In order to get the current progress of loading / processing video, AJAX requests were created. The form included a randomly generated identifier string (for example, 32 characters). Node.js attached information on the current file to it and, with each state request, passed the following JSON:
{
"fileid":"uwrd28a9v71j444d260c55hkj6uli06j", // Идентификатор
"name":"test.FLV", // Название
"progress":"7.80", // Проценты загрузки
"status":"uploading", // Статус
"audio":10.02, // Время аудио
"frame":301, // Текущий фрейм, который обрабатывается
"videoProgress":"7.80", // Процент видео
"audioProgress":"3.73", // Процент аудио
"frameProgress":"3.73", // Процент раскадровки
"messages":[], // Сообщение, например об отсутствии аудио дорожки
"expectedBytes":5389574, // Ожидаемый размер
"uploadedBytes":420139, // Загружено байт
"bps":91773.48186981214, // Скорость байт/сек
"uploadEstimated":54, // Время до окончания загрузки
"videoEstimated":42, // Время до окончания обработки видео
"audioEstimated":75, // Время до окончания обработки аудио
"frameEstimated":71 // Время до окончания раскадровки
}
Based on this, the user sees the download status.
2. User authorization
Since the frontend in PHP (user authorization is also taking place there), it was necessary to somehow transfer the session data to the node.js server. For these purposes, memcached was used. You can read about session substitution in PHP here . The bottom line is simple: sessions are saved in memcached, when loaded through the form, session_id is passed, which is read by node.js. Further node.js addresses memcached in search of the corresponding session, takes user_id, etc. There is one important point: the session_id in the form itself must be set first, because the POST request passes the parameters in the order they appear in HTML. That is, if we do this:
In this case, the file itself will come to node.js first, and then session_id, which is not good. After all, we first need to know who the file is coming from and cancel if the user is not authorized. If in the example above to swap file and session_id, then first we get the session, slow down the file download, and wait until the server checks to see if everything is ok with the user and only then continue to download. In node.js, you can pause request.
3. Streaming video
The problem is that not all videos can be processed right away. For example, according to the mp4 format specification, metadata goes at the end of the file. But we need to immediately know what sizes the video has, what tracks are in the file, duration, etc. Plus, some video formats during transcoding need to be able to access different parts of the source file. Based on this, there are two cases with two options each:
1. At the beginning of the download, when we pull the file information using the ffmpeg command using node.js:
var spawn = require('child_process').spawn;
var ffmpeg = spawn('ffmpeg', ['-i', '-']);
var ffmpeg_stdout = ''
ffmpeg.stderr.on('data', function(buffer)
{
ffmpeg_stdout += buffer;
// Получаем информацию о файле
});
// file - берется с нашей формы
file.on('data', function(buffer)
{
ffmpeg.stdin.write(buffer);
});
In the variable ffmpeg_stdout we get all the tracks that are contained in the file or an error. If there is an error, then ffmpeg cannot process the file with a stream. In this case, we warn the user about this and download the entire file immediately, and only after that we perform the necessary operations.
2. In this case, speed is important, so we are trying to start the storyboard as soon as the video has just started loading. But if the format does not support streaming processing, then you have to wait until the full download.
It turns out the following algorithm: it is possible to extract metadata, then we try to process the video on the fly, if this turns out, then we are happy, in all other cases we are waiting for the full download. In practice, only a small number of videos need to be downloaded completely in order to begin processing.
That's all for today, tomorrow morning for work.
I would like to conclude on a major note: the possibilities are endless, you can distribute the load on several servers, show the results on the go, for example, pull the last finished frame from the video. It all works on Amazon EC2. In terms of speed, I can say that there was no way to check the download speed (the channel is weak), but not less than 10mbits, I think for sure. And on the load on the server: Middle server alone, one video pulls somewhere around 2Mbps of video, that is, the download has gone far ahead, and processing is not keeping up. Resize takes the most time. Storyboard and pulling audio tracks relatively quickly.
If you like the post, I will describe the implementation and the mechanism of work in more detail next time.
Screenshots of what happened. Immediately make a reservation, this is not a working option, so do not judge strictly: