Matlab is a well-known medium for numerical computing, widely used by the scientific community. It works on Windows, nix-systems and on poppies. Matlab language is optimized for working with matrices and multidimensional arrays; in addition, there are a huge number of extensions (officially called Toolboxes) for solving optimization problems, statistical calculations, processing signals and images, etc. Plus, there is a large amount of scientific code already written in Matlab, which further spurs its popularity.

In this article, I will briefly describe the capabilities of Matlab for working with video. For those who are not familiar with matlab syntasis, a brief overview of the main features in Russian can be found here .

Video format

The first thing to do to work with the video is to upload it to Matlab. The main difficulty that arose with me is related to codecs. By default, the list of supported video formats is small ( English documentation ) and whether the video will open your video depends on the codecs installed in the system. The general rule is that if your built-in player (in Windows it is Windows Media Player) cannot open the file, then Matlab most likely will not open it.

Accordingly, the ability to record video strongly depends on the installed codecs, although Matlab can create uncompressed video on any system without installed codecs.

I solved the problems with reading and recording video by installing KLite Codec Pack Full in the “Lots of stuff” option:

Different ways to read video

There are several ways to read videos in matlab. Two of them are described below.

Reading a video using mmreader and read methods

The first way to read video is to use the method mmreaderto open a file and then use the method readto read frames:

% open the file
video = mmreader ('d: \ test \ 1.avi');
% this is how you can learn about some properties of an open video
width = video.Width;
height = video.Height;
frameRate = video.FrameRate;
numOfFrames = video.NumberOfFrames;
% read frame
% frames are numbered from 1
frameNo = 50;
frame = read (video, frameNo);
% frame can be read and like this
frame = video.read (frameNo);
% read the range of frames
% 50 - number of the first read frame
% 100 - number of the last read frame
framesRange = [50 100];
frames = read (video, framesRange);
% read all videos
% operation will not be performed if the video is large and does not fit into memory
entireVideo = read (video);
% In matlab there is no special function for closing a video file.
% Accordingly, when you finish working with the video
% no additional actions need to be performed.

All frame reading functions return a multidimensional matrix as the result of execution. For example, here is the result of executing several commands for working with video in the interactive console of the matlab:

>> video.Width
ans =
   704
>> video.Height
ans =
   576
>> frame = read (video, 5);
>> size (frame)
ans =
   576 704 3
>> frames = read (video, [5 10]);
>> size (frames)
ans =
   576 704 3 6

Thus, as a result of reading one frame, a matrix of size Height x Width x 3 is obtained, whose elements are 8-bit unsigned integers. As a result of reading a range of several frames, a matrix of size Height x Width x 3 x NumOfFrames is obtained.

Accordingly, when you uploaded a frame, you can work with it like a regular Matlab matrix, as well as apply all sorts of functions for image processing included in the Image Processing Toolbox to it.

Reading videos with System Objects - Matlab 2010a innovations

In matlaba version 2010a, the concept of system objects was introduced. The Matlab manual says that system objects provide an object-oriented implementation of algorithms. Compared to ordinary Matlab functions, they automatically control their state and are well suited for streaming information processing. In general, the main point is that according to the developer, it is more convenient to use them for streaming information processing. At the expense of convenience, you can bet, but the code is getting a little shorter.

Here is an example of frame-by-frame reading of videos using system objects:

% create an object to read the video
reader = video.MultimediaFileReader ('Filename', 'd: \ test \ 1.avi');
% read all the videos frame by frame
% step function will return a matrix of size Height x Width x 3
% true elements of this matrix will already be of type single - 4 byte floating-point numbers
while ~ isDone (reader)
   frame = step (reader);
end
% here, as in the previous case, no closing operations are necessary
% at the end of the video

A significant drawback of this approach is that there is no random access to video - frames can only be read sequentially. Also, according to subjective sensations, opening a video and reading frames using system objects takes slightly longer than using mmreaderand read.

Playing video

In addition to the fact that the video can be read and processed, it can also be played by means built into the matlab.

Play with movie function

The movie function allows you to play a video. The peculiarity of using this function is that you need to prepare in advance the entire video, which must be played. In addition, the video must be stored in memory in a format different from that obtained when reading the video. The following is an example of how you can use this function to play the first 200 frames of a video:

%% video reading
filename = 'd: \ test \ 1.avi';
video = mmreader (filename);
nFrames = 200;
% prepare the structure into which the information will be read
mov (1: nFrames) = struct ('cdata', [], 'colormap', []);
% Read frame by frame
for k = 1: nFrames
    mov (k) .cdata = read (video, k);
end
%% play video
hf = figure;
set (hf, 'position', [150 150 video.Width video.Height])
movie (hf, mov, 1, video.FrameRate);

In this code, a movmovie is prepared in the variable . This variable stores an array of structures. Each structure has two fields: cdataand colormap. The field cdatastores a bitmap image of the frame, and the field colormapneeds to store a color palette for videos with indexed color. In this case, this is a full-fledged RGB video and the field is colormapempty.

Play with system objects

Compared with the previous example, the use of system objects can significantly reduce the code:

% create an object to read the video
reader = video.MultimediaFileReader ('Filename', 'd: \ test \ 1.avi');
% create an object to play the video
player = video.VideoPlayer;
% read frame by frame and play video
while ~ isDone (reader)
   frame = step (reader);
   step (player, frame);
end

Video recording

There are several ways to record video. Here I will analyze 3 methods: frame-by-frame recording, video recording in one piece and the use of system objects.

I note in advance that if you are recording video in more than one piece, you must definitely close the video file at the end of the operation. If the file is not closed, then until you exit the matlab you can’t access, delete or overwrite it.

Record video frame by frame

%% prepare variables
filename = 'd: \ test \ 1.avi';
result_file = 'd: \ test \ result.avi';
nFrames = 200;
video = mmreader (filename);
% open the file for writing
% compressor install - XVID
writer = avifile (result_file, 'compression', 'XVID');
%% in the loop read-modify-write
% structural element for dilatation
se = strel ('rectangle', [3 3]);
for k = 1: nFrames
    % read frame
    frame = read (video, k);
    % perform frame dilatation
    frame = imdilate (frame, se);
    % write frame
    writer = addframe (writer, frame);
end
% be sure to close writer
writer = close (writer);

Here I note the way in which function arguments are set avifile. The first argument is the file name, then the arguments go in pairs - first the name of the parameter, then the value. In the above example, avifilein addition to the file name, the compression parameter with the value XVID is also passed.

You can specify one of the identifiers as the compression parameter: 'None', 'MSVC', 'RLE', 'Cinepak', 'Indeo3' or 'Indeo5'. On * nix systems, only the 'None' option is available. In addition to the above options, you can also specify FourCC of the desired compressor (as in the example). Here it all depends on the codecs installed in the system for video encoding. It is worth noting that not all of the installed codecs are suitable for both decoding and video encoding. On my system of several tried and tested options, only XVID earned.

One-piece video recording

In addition to frame-by-frame recording, you can also trim an array of frames and write them all at once to a file. An array of frames should be prepared in the same form as in the example of using the function movieto play video. More nicely: the array should consist of structures, one of the structure members is the cdata field that stores the bitmap of the image, and the second is the colormap field that stores the palette for the image with indexed color:

%% video reading
filename = 'd: \ test \ 1.avi';
video = mmreader (filename);
nFrames = 200;
% prepare the structure into which the information will be read
mov (1: nFrames) = struct ('cdata', [], 'colormap', []);
% Read and modify the first 200 frames of video
se = strel ('rectangle', [3 3]);
for k = 1: nFrames
    frame = read (video, k)
    mov (k) .cdata = imdilate (frame, se);
end
%% video recording
movie2avi (mov, 'd: \ test \ result.avi', 'compression', 'XVID', 'fps', 25);

The compression parameter for the function movie2aviis the same as in the previous section.

Record video using system objects

The following code demonstrates the use of system objects to record video:

reader = video.MultimediaFileReader ('Filename', 'd: \ test \ 1.avi');
% open the video for recording,
% filename = d: \ test \ result.avi
% audio - no, video - is
% video codec = MJPEG
writer = video.MultimediaFileWriter ('Filename', 'd: \ test \ result.avi', ...
    'AudioInputPort', false, 'VideoInputPort', true, ...
    'VideoCompressor', 'MJPEG Compressor');
% structural element for dilatation
se = strel ('rectangle', [3 3]);
while ~ isDone (reader)
   % read frame
   frame = step (reader);
   % perform frame dilatation
   frame = imdilate (frame, se);
   % write frame
   step (writer, frame);
end
% be sure to close writer
close (writer);

As arguments to the function video.MultimediaFileWriter, pairs 'parameter name', parameter value are specified. Even the file name is transmitted in this way (i.e. the pair 'Filename', 'file name' is specified).

The possible values ​​of the VideoCompressor parameter here are already different from those that can be specified for the avifileand functions movie2avi. As VideoCompressor can be specified 'None (uncompressed)', I also earned 'MJPEG Compressor'. If the VideoCompressor parameter is omitted altogether in the argument list, this will be equivalent to specifying 'None (uncompressed)'.

A free alternative in the face of GNU Octave

In general, the Matlab product is quite expensive, but there are partially free alternatives compatible with it. One of them is GNU Octave .

GNU Octave is actively developing and the syntax of the language itself is almost identical to Matlab. The differences in the syntax are mainly due to the fact that some feature has already appeared in the matlab, but has not yet been added to Octave.

With a set of libraries, things are worse. Matlab toolboxes are missing from GNU Octave, although replacements have been written for some functions.

Install GNU Octave on Ubuntu 10.04

I worked with Octave in Ubuntu, so for those who want to repeat my experience I describe the installation instructions:

1. Using the package manager, you need to install the octave3.2 Linux packages; octave3.2-doc; octave3.2-headers with all dependencies.
2. Octave, unlike matlab, consists only of an interpreter. Accordingly, you may need a development environment with a graphical interface. I installed QtOctave in Ubuntu 10.04 (also through the package manager).
3. If you plan to work with video, then you need to install the Linux packages libavformat-dev, libavcodec-dev, libavutil-dev, libswscale-dev
4.Octave itself also has the concept of a package. Octave packages include a variety of features that extend the functionality of Octave. You can download Octave packages from the Octave-Forge website .
Octave need to run with administrative rights to install Octave-package: sudo octave. And then in Octave execute the command. pkg install /path/to/package.tar.gz
To work with images and videos, Octave-packages image and video will be understood.

Actually work with video in Octave

In general, loading and saving video in Octave is done in a similar way to how it is done in matlab. The following is the code that dilates the first 25 frames of the video:

filename = '/home/alexey/octave/1.avi';
result_file = '/home/alexey/octave/result.avi';
% uncomment the next line to access the video properties
% info = aviinfo (filename);
% width = info.Width
% height = info.Height
% create object recording video
writer = avifile (result_file);
se = ones ([3 3 3]);
for i = 1: 25
  % read frame number i from video
  frame = aviread (filename, i);
  % perform frame dilatation
  frame = imdilate (frame, se);
  % write frame
  addframe (writer, frame);
end

Here, the function avifile(filename, frameNo)reads a frame with a number frameNofrom a video file with a name filename. You do not need to create any objects in advance and open the file. This function is also present in matlab, but it is deprecated.

If you need to find out information about the video, then this can be done using the function aviinfo(filename). As a result, a structure will be returned that stores information about the size and format of the video, the number of frames, etc. In matlab this function is deprecated.

Video recording is performed using the functions avifileand addframe, but in the matlab and Octave the syntax of these functions is slightly different. In addition, in Octave you do not need to close the object that records the video (moreover, there is not even a method that would close it).

As a result, in order for the above code for Octave to work in matlab, it needs to be modified as follows:

%% video reading
filename = 'd: \ test \ result.avi';
result_file = 'd: \ test \ result2.avi';
% info = aviinfo (filename);
% width = info.Width;
% height = info.Height;
writer = avifile (result_file, 'compression', 'xvid');
se = strel ('rectangle', [3 3]);
for i = 1: 25
   frame = aviread (filename, i);
   frame.cdata = imdilate (frame.cdata, se);
   writer = addframe (writer, frame.cdata);
end
writer = close (writer);

Thus, in terms of video and picture processing, there is currently a difference in the syntax of the Matlab functions and GNU Octave functions.