Linux task parallelization
It took me to transcode a number of video files. For this, I wrote the following script:
It would seem that everything was ready, but I noticed that only one processor out of two was loaded, which means that this process can be accelerated two times.
You can set mencoder options. For x264 codec it is possible to specify the number of threads:
The method is good, but not universal and a deterioration in the quality of the result is possible.
The method is simple, we start processes in parallel, for example as follows:
The disadvantage of this method is that due to the fact that the files are of different sizes, the time for processing them can vary significantly and as a result, one processor can be unloaded much earlier than the other.
I wanted to somehow improve the scenario so that two tasks always worked in parallel and new ones started as they were completed. Thinking about how to do this, I remembered that there was a wonderful build utility that could do exactly what I needed. It turned out the following:
Makefile:
It turned out surprisingly simple and short. At the beginning, a list of all the files that I want to receive is listed. It is followed by the path to the source files and the build rule. Run with the command “make -j 2” so that 2 processes work simultaneously.
(The first article on a habr, do not judge strictly)
#!/bin/bash
recode() {
mencoder -o $2 $1 -ovc x264 -x264encopts bitrate=22000:keyint=50 -oac mp3lame -lameopts vbr=3:br=320 -fps 50
}
recode input/00108.mts 00108.avi
recode input/00109.mts 00109.avi
...
...
It would seem that everything was ready, but I noticed that only one processor out of two was loaded, which means that this process can be accelerated two times.
The first method: parallelization with mencoder
You can set mencoder options. For x264 codec it is possible to specify the number of threads:
- threads = <0-16>
Use threads to encode on multiple processors simultaneously (default: 1). This slightly degrades the quality of the encoding. 0 or 'auto' - automatically determine the number of processors and use the appropriate number of threads.
The method is good, but not universal and a deterioration in the quality of the result is possible.
The second method: parallelization using Bash
The method is simple, we start processes in parallel, for example as follows:
(recode input/00108.mts 00108.avi
recode input/00109.mts 00109.avi
...
...) &
(recode input/00108.mts 00110.avi
recode input/00109.mts 00111.avi
...
...)
The disadvantage of this method is that due to the fact that the files are of different sizes, the time for processing them can vary significantly and as a result, one processor can be unloaded much earlier than the other.
Third way: parallelizing with GNU make
I wanted to somehow improve the scenario so that two tasks always worked in parallel and new ones started as they were completed. Thinking about how to do this, I remembered that there was a wonderful build utility that could do exactly what I needed. It turned out the following:
Makefile:
all: 00108.avi 00109.avi 00110.avi 00111.avi 00118.avi 00119.avi 00120.avi 00123.avi
VPATH = input:.
%.avi: %.mts
mencoder -o $@ $< -ovc x264 -x264encopts bitrate=22000:keyint=50 -oac mp3lame -lameopts vbr=3:br=320 -fps 50
It turned out surprisingly simple and short. At the beginning, a list of all the files that I want to receive is listed. It is followed by the path to the source files and the build rule. Run with the command “make -j 2” so that 2 processes work simultaneously.
(The first article on a habr, do not judge strictly)