April 30, 2014 at 16:23Writing a plugin for GStreamer on MS Visual Studio
I have always been interested in real-time video processing applications. On Habré I read a series of articles about multimedia framework GStreamer:
I really wanted to do something with its use. But, as usually happens, current tasks completely exhausted the free time resource.
And once, in the process of working on a project, I needed to organize a video surveillance system at the facility and integrate it into the accounting system. This task was successfully solved by the specialists of our company and is not worthy of the attention of the general public. The accounting system runs on Microsoft .NET, all cameras issue an H264 RTSP stream. To capture the video, the commercial MediaSuite library from Streamcoders is used.
As a bonus, I allowed myself to conduct a series of experiments with GStreamer to capture and process video.
To begin with, I decided to try to capture the stream from one of the cameras aimed at the scales for weighing vehicles, transfer the frame to the license plate recognition subsystem and apply the recognition result to the video. In the above example, I will not call the LPR functional, just intercept the frame and draw a rectangle on it.
So, given:
You must get:
Solution Method:
GStreamer provides the developer with its class hierarchy . For our purposes, we can inherit our plugin from the GstElement class . GstElement is the base abstract class needed to develop an element that can be used in processing GStreamer threads.
Download and install GStreamer gstreamer-1.0-xxx-1.2.4.msi. You can take it here .
For development, we will also need the gstreamer-1.0-devel-xxx.msi distribution, for example, gstreamer-1.0-devel-x86-1.2.4.msi. In the installation process, we select the necessary options:
Looking ahead, I will say it would be better to install the Windows Device Driver Kit 7.1.0 . By default, it is set to C: \ WinDDK \ 7600.16385.1 and that is where Visual Studio will look for it when building the project. If you already have DDK installed in a different way, this can be fixed later, directly in the project settings.
The GStreamer project website, among other documentation, has a guide for plugin developers. You can read the manual to write all the code yourself, but you can download the plug-in project template from the repository .
The template contains C ++ source files and a script for generating the plugin code. As the manual says, after deploying the template from the repository, you need to go to the gst-template / gst-plugin / src directory and run the ../tools/make_element utility. The make_element utility has two parameters: the name of the plugin (dummy), the name of the source file to be used (gstplugin by default).
As a result of execution, we get two files: gstdummy.c and gstdummy.h. Inside there will be a skeleton of the dummy plugin, which is still stupid and does nothing, but can already be built into the framework plugin system.
A small remark: all of the above is true for Linux, Unix machines, but what about the mournful owners of Windows? Cmd.exe will not execute make_element. If you look inside make_element, it becomes clear that he is not doing anything complicated, and with the help of the stream editor, sed generates target sources based on the parameters given to it. You can do it yourself. Just in case, I created a repository where I will put my test project in the course of development: github.com/nostrum-service/gst .
After we did the preliminary work, the time came to form the project directly in MS Visual Studio 2010. Fortunately, the GStreamer developers took care of the Visual Studio users and put everything necessary for creating the project into the distribution kit. You just need to correctly place the files in Visual Studio directories.
Everything you need is in the gstreamer \ 1.0 \ x86 \ share \ vs \ 2010 directory.
We carry out:
We start Visual Studio (or restart so that it sees new settings), create a new project, and select Visual C ++ \ gst-dk-template from the installed templates.
If everything went fine, an empty project with the necessary settings will be created. Since we want to create a plugin, go to the project settings and change in Project Details Configuration Type from Application (.exe) to Dynamic Library (.dll).
In the Property Manager window, we observe the following picture (you can enable Property Manager View-> Other Windows-> Property Manager):
In the created empty project, you must include the files that were previously created using the make_element utility.
Compile, if everything is correct, we get a ready-made DLL that must be copied to the GStreamer plugin directory (I have C: \ gstreamer \ 1.0 \ x86 \ lib \ gstreamer-1.0 \).
Just in case, check: gst-inspect-1.0 dummy. Here we will see what GStreamer learned about our plugin.
In order to embed the plugin in the chain, we need to implement the following life cycle:
The function
static void gst_dummy_class_init (GstdummyClass * klass) is responsible for providing metadata about the plugin.
In our example, the dummy element has a Silent property of type Boolean, which is responsible for displaying text when processing a stream.
With this code, we inform the GStreamer environment that the plugin has the Silent property, it has the Boolean type, the delegate gst_dummy_set_property is responsible for its installation, reading is gst_dummy_get_property, it is read and write accessible, the default value is FALSE. Next, we register the connection points to the plugin - pads.
We define the input pad sink, which is always available - GST_PAD_ALWAYS and accepts any format GST_STATIC_CAPS ("ANY").
We determine the output pad src, which is always available - GST_PAD_ALWAYS and produces any format GST_STATIC_CAPS ("ANY").
During the pipeline construction process, the GStreamer environment calls the plugin instance initialization function:
at the entrance of which the structure to be filled is supplied:
In order to handle events occurring at the plugin input, you must specify the appropriate delegate:
When events occur at the plugin input, in our case the function will be called:
which does nothing yet.
Now you can try to run - gst-launch-1.0 videotestsrc! dummy! autovideosink –v. This command transfers the test video stream generated by videotestsrc to our plugin, which transfers it further without changes to the autovideosink video player. Key v allows you to see how the entire chain is processed.
If we saw a test picture, it means that our plugin successfully transferred data from its input to output.
For our case, the output of the processing will contain the following:
It follows that an instance of the GstVideoTestSrc class with the name videotestsrc0 provided pad src, issuing a video / x-raw stream in I420 format with frame sizes of 320 by 240, etc.
Since our plugin can accept any format, its input was associated with the output of videotestsrc0:
Having found out that the mechanics are working, we will try to take several steps to our goal. To illustrate the evolution of the solution, I added another painter plugin to the project, which will draw a rectangle over the image.
You can draw a rectangle without involving any libraries, but I wanted to continue experimenting using OpenCV.
When adding a new plugin, there is a problem of its registration, because The registrar should be one per project. In this regard, the GstTestLib.cpp file was introduced, which contains the registration code of both plugins.
For painter, I created more stringent restrictions on the input stream format. Now it looks like this:
This means that streaming video in BGRx format with any resolution and frame rate can be input. OpenCV uses the BGR scheme by default. You can read about color schemes and transformations here .
Since the camera that interests me gives out the RTSP H264 stream, we need to decode it and feed it to the input of the videoconvert converter. Our test case will look like this:
(10.10.0.15 - my internal address is taken as an example).
Now, at the input, the buffer will be in BGRx format. If you do not know how to build a pipeline, you can use the universal decodebin container. He will try to select and connect the appropriate plugins:
the –v switch is needed for diagnostic output, and there you will see how the pipeline is formed.
Back to our plugin. All our processing will consist in forming a new image based on the input image and drawing a rectangle over it. The working function has the form:
In the sink pad event handler we can find out the input stream information
The problem is solved in the forehead. The experiment was a success. As GStreamer develops, new opportunities open up, which, as I advance and have interest in the topic, I will talk more about.
GStreamer
object hierarchy GStreamer
plugin development guide GStreamer
OpenCV
plugin template repository
my sources
- Introducing GStreamer: Introduction
- Introducing GStreamer: Data Sources
- Introducing GStreamer: Output Devices
- GStreamer: Linux-flavored codecs
I really wanted to do something with its use. But, as usually happens, current tasks completely exhausted the free time resource.
And once, in the process of working on a project, I needed to organize a video surveillance system at the facility and integrate it into the accounting system. This task was successfully solved by the specialists of our company and is not worthy of the attention of the general public. The accounting system runs on Microsoft .NET, all cameras issue an H264 RTSP stream. To capture the video, the commercial MediaSuite library from Streamcoders is used.
As a bonus, I allowed myself to conduct a series of experiments with GStreamer to capture and process video.
To begin with, I decided to try to capture the stream from one of the cameras aimed at the scales for weighing vehicles, transfer the frame to the license plate recognition subsystem and apply the recognition result to the video. In the above example, I will not call the LPR functional, just intercept the frame and draw a rectangle on it.
So, given:
- RTSP H264 source
- Windows 7 32-bit system
- MS Visual Studio 2010
- C ++ language
- Gstreamer 1.0
You must get:
- video stream containing processing results
Solution Method:
- Plugin development for GStreamer 1.0
GStreamer provides the developer with its class hierarchy . For our purposes, we can inherit our plugin from the GstElement class . GstElement is the base abstract class needed to develop an element that can be used in processing GStreamer threads.
Install GStreamer
Download and install GStreamer gstreamer-1.0-xxx-1.2.4.msi. You can take it here .
For development, we will also need the gstreamer-1.0-devel-xxx.msi distribution, for example, gstreamer-1.0-devel-x86-1.2.4.msi. In the installation process, we select the necessary options:
Looking ahead, I will say it would be better to install the Windows Device Driver Kit 7.1.0 . By default, it is set to C: \ WinDDK \ 7600.16385.1 and that is where Visual Studio will look for it when building the project. If you already have DDK installed in a different way, this can be fixed later, directly in the project settings.
The GStreamer project website, among other documentation, has a guide for plugin developers. You can read the manual to write all the code yourself, but you can download the plug-in project template from the repository .
The template contains C ++ source files and a script for generating the plugin code. As the manual says, after deploying the template from the repository, you need to go to the gst-template / gst-plugin / src directory and run the ../tools/make_element utility. The make_element utility has two parameters: the name of the plugin (dummy), the name of the source file to be used (gstplugin by default).
As a result of execution, we get two files: gstdummy.c and gstdummy.h. Inside there will be a skeleton of the dummy plugin, which is still stupid and does nothing, but can already be built into the framework plugin system.
A small remark: all of the above is true for Linux, Unix machines, but what about the mournful owners of Windows? Cmd.exe will not execute make_element. If you look inside make_element, it becomes clear that he is not doing anything complicated, and with the help of the stream editor, sed generates target sources based on the parameters given to it. You can do it yourself. Just in case, I created a repository where I will put my test project in the course of development: github.com/nostrum-service/gst .
After we did the preliminary work, the time came to form the project directly in MS Visual Studio 2010. Fortunately, the GStreamer developers took care of the Visual Studio users and put everything necessary for creating the project into the distribution kit. You just need to correctly place the files in Visual Studio directories.
Everything you need is in the gstreamer \ 1.0 \ x86 \ share \ vs \ 2010 directory.
We carry out:
xcopy c:\gstreamer\1.0\x86\share\vs\2010\gst-template\*.* "C:\Program Files\Microsoft Visual Studio 10.0\VC\VCWizards\gst-template\*.*" /s /e /c
xcopy C:\gstreamer\1.0\x86\share\vs\2010\wizard\*.* "C:\Program Files\Microsoft Visual Studio 10.0\VC\vcprojects\"
We start Visual Studio (or restart so that it sees new settings), create a new project, and select Visual C ++ \ gst-dk-template from the installed templates.
If everything went fine, an empty project with the necessary settings will be created. Since we want to create a plugin, go to the project settings and change in Project Details Configuration Type from Application (.exe) to Dynamic Library (.dll).
In the Property Manager window, we observe the following picture (you can enable Property Manager View-> Other Windows-> Property Manager):
In the created empty project, you must include the files that were previously created using the make_element utility.
Compile, if everything is correct, we get a ready-made DLL that must be copied to the GStreamer plugin directory (I have C: \ gstreamer \ 1.0 \ x86 \ lib \ gstreamer-1.0 \).
Just in case, check: gst-inspect-1.0 dummy. Here we will see what GStreamer learned about our plugin.
Minimum plugin features
In order to embed the plugin in the chain, we need to implement the following life cycle:
- tell the framework about yourself (initialize the class)
- initialize plugin instance
- process thread connection process
- do something with the input and pass it to the output
- at the end, clear the occupied resources
Metadata
The function
static void gst_dummy_class_init (GstdummyClass * klass) is responsible for providing metadata about the plugin.
In our example, the dummy element has a Silent property of type Boolean, which is responsible for displaying text when processing a stream.
gobject_class->set_property = gst_dummy_set_property;
gobject_class->get_property = gst_dummy_get_property;
g_object_class_install_property (gobject_class, PROP_SILENT,
g_param_spec_boolean ("silent", "Silent", "Produce verbose output ?",
FALSE, (GParamFlags)G_PARAM_READWRITE));
With this code, we inform the GStreamer environment that the plugin has the Silent property, it has the Boolean type, the delegate gst_dummy_set_property is responsible for its installation, reading is gst_dummy_get_property, it is read and write accessible, the default value is FALSE. Next, we register the connection points to the plugin - pads.
gst_element_class_add_pad_template (gstelement_class,
gst_static_pad_template_get (&src_factory));
gst_element_class_add_pad_template (gstelement_class,
gst_static_pad_template_get (&sink_factory));
We define the input pad sink, which is always available - GST_PAD_ALWAYS and accepts any format GST_STATIC_CAPS ("ANY").
static GstStaticPadTemplate sink_factory = GST_STATIC_PAD_TEMPLATE ("sink",
GST_PAD_SINK,
GST_PAD_ALWAYS,
GST_STATIC_CAPS ("ANY")
);
We determine the output pad src, which is always available - GST_PAD_ALWAYS and produces any format GST_STATIC_CAPS ("ANY").
static GstStaticPadTemplate src_factory = GST_STATIC_PAD_TEMPLATE ("src",
GST_PAD_SRC,
GST_PAD_ALWAYS,
GST_STATIC_CAPS ("ANY")
);
Instance initialization
During the pipeline construction process, the GStreamer environment calls the plugin instance initialization function:
static void
gst_dummy_init (Gstdummy * filter)
at the entrance of which the structure to be filled is supplied:
struct _Gstdummy
{
GstElement element;
GstPad *sinkpad, *srcpad;
gboolean silent;
};
In order to handle events occurring at the plugin input, you must specify the appropriate delegate:
gst_pad_set_event_function (filter->sinkpad, GST_DEBUG_FUNCPTR(gst_dummy_sink_event));
Stream Connection
When events occur at the plugin input, in our case the function will be called:
static gboolean
gst_dummy_sink_event (GstPad * pad, GstObject * parent, GstEvent * event)
which does nothing yet.
Trial run
Now you can try to run - gst-launch-1.0 videotestsrc! dummy! autovideosink –v. This command transfers the test video stream generated by videotestsrc to our plugin, which transfers it further without changes to the autovideosink video player. Key v allows you to see how the entire chain is processed.
If we saw a test picture, it means that our plugin successfully transferred data from its input to output.
For our case, the output of the processing will contain the following:
/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = video/x-raw, format=(string)I420, width=(int)320, height=(int)240, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
It follows that an instance of the GstVideoTestSrc class with the name videotestsrc0 provided pad src, issuing a video / x-raw stream in I420 format with frame sizes of 320 by 240, etc.
Since our plugin can accept any format, its input was associated with the output of videotestsrc0:
/GstPipeline:pipeline0/Gstdummy:dummy0.GstPad:sink: caps = video/x-raw, format=(string)I420, width=(int)320, height=(int)240, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
Further dive
Having found out that the mechanics are working, we will try to take several steps to our goal. To illustrate the evolution of the solution, I added another painter plugin to the project, which will draw a rectangle over the image.
You can draw a rectangle without involving any libraries, but I wanted to continue experimenting using OpenCV.
When adding a new plugin, there is a problem of its registration, because The registrar should be one per project. In this regard, the GstTestLib.cpp file was introduced, which contains the registration code of both plugins.
struct _elements_entry
{
const gchar *name;
GType (*type) (void);
};
static const struct _elements_entry _elements[] = {
{"dummy", gst_dummy_get_type},
{"painter", gst_painter_get_type},
{NULL, 0},
};
static gboolean
plugin_init (GstPlugin * plugin)
{
gint i = 0;
while (_elements[i].name) {
if (!gst_element_register (plugin, _elements[i].name,
GST_RANK_NONE, (_elements[i].type) ()))
return FALSE;
i++;
}
return TRUE;
}
For painter, I created more stringent restrictions on the input stream format. Now it looks like this:
static GstStaticPadTemplate sink_factory = GST_STATIC_PAD_TEMPLATE ("sink",
GST_PAD_SINK,
GST_PAD_ALWAYS,
GST_STATIC_CAPS ( GST_VIDEO_CAPS_MAKE ("{ BGRx }") )
);
This means that streaming video in BGRx format with any resolution and frame rate can be input. OpenCV uses the BGR scheme by default. You can read about color schemes and transformations here .
Since the camera that interests me gives out the RTSP H264 stream, we need to decode it and feed it to the input of the videoconvert converter. Our test case will look like this:
gst-launch-1.0 -v rtspsrc location=rtsp://10.10.0.15 ! rtph264depay ! avdec_h264 ! videoconvert ! painter ! videoconvert ! autovideosink
(10.10.0.15 - my internal address is taken as an example).
Now, at the input, the buffer will be in BGRx format. If you do not know how to build a pipeline, you can use the universal decodebin container. He will try to select and connect the appropriate plugins:
gst-launch-1.0 -v rtspsrc location=rtsp://10.10.0.15 ! decodebin ! autovideosink
the –v switch is needed for diagnostic output, and there you will see how the pipeline is formed.
Back to our plugin. All our processing will consist in forming a new image based on the input image and drawing a rectangle over it. The working function has the form:
static GstBuffer *
gst_painter_process_data (Gstpainter * filter, GstBuffer * buf)
{
// объединяем все части буфера в непрерывную область для чтения
GstMapInfo srcmapinfo;
gst_buffer_map (buf, &srcmapinfo, GST_MAP_READ);
// формируем новый буфер
GstBuffer * outbuf = gst_buffer_new ();
// создаем заголовок на исходное изображение
IplImage * dst = cvCreateImageHeader (cvSize (filter->width, filter->height), IPL_DEPTH_8U, 4);
// выделяем память для нового изображения
GstMemory * memory = gst_allocator_alloc (NULL, dst->imageSize, NULL);
GstMapInfo dstmapinfo;
if (gst_memory_map(memory, &dstmapinfo, GST_MAP_WRITE)) {
// копируем исходное изображение в новую область памяти
memcpy (dstmapinfo.data, srcmapinfo.data, srcmapinfo.size);
dst->imageData = (char*)dstmapinfo.data;
// рисуем прямоугольник
cvRectangle (dst, cvPoint(10,10), cvPoint(100, 100), CV_RGB(0, 255, 0), 1, 0);
// добавляем память с модифицированным изображением в выходной буфер
gst_buffer_insert_memory (outbuf, -1, memory);
gst_memory_unmap(memory, &dstmapinfo);
}
cvReleaseImageHeader(&dst);
gst_buffer_unmap(buf, &srcmapinfo);
return outbuf;
}
In the sink pad event handler we can find out the input stream information
static gboolean
gst_painter_sink_event (GstPad * pad, GstObject * parent, GstEvent * event)
{
gboolean ret;
Gstpainter *filter;
filter = GST_PAINTER (parent);
switch (GST_EVENT_TYPE (event)) {
case GST_EVENT_CAPS:
{
GstCaps * caps;
gst_event_parse_caps (event, &caps);
//получаем структуру
GstStructure *structure = gst_caps_get_structure (caps, 0);
//читаем параметы
gst_structure_get_int (structure, "width", &filter->width);
gst_structure_get_int (structure, "height", &filter->height);
filter->format = gst_structure_get_string (structure, "format");
ret = gst_pad_event_default (pad, parent, event);
break;
}
default:
ret = gst_pad_event_default (pad, parent, event);
break;
}
return ret;
}
The problem is solved in the forehead. The experiment was a success. As GStreamer develops, new opportunities open up, which, as I advance and have interest in the topic, I will talk more about.
Sources
GStreamer
object hierarchy GStreamer
plugin development guide GStreamer
OpenCV
plugin template repository
my sources