Broadcast h264 video without transcoding from the Logitech C920

Tutorial

The delay was no more than 0.3 seconds, taking into account the fact that the camera is connected to a weak BeagleBone Blue with a processor load of no more than 30% and transferring the video stream via a wifi router to the laptop.

Initially, the task was to transmit video from the robot to a large computer without loading the processor on the robot and without delays. In the process of googling cameras and programs for this task, it was found that there are cameras with hardware video encoding on the camera itself , after which the Logitech C920 camera was acquired. And the tests and experiments began, during which two variants of translation were found without recoding.

All further actions are checked on Ubuntu 16.04, the second option may work in Windows

Broadcast hardware-encoded video from the Logitech C920 camera via cvlc and v4l2

clvc - server video broadcast via vlc from the command line.

Hardware video encoding

Set pixelformat cameras to H264

v4l2-ctl --device=/dev/video1 --set-fmt-video=width=800,height=600,pixelformat=1

Test H264 using vlc so it will not reset the settings:

cvlc v4l2:///dev/video1 --demux h264

If you try this, vlc will reset the settings before playback:

cvlc v4l2:///dev/video1:chroma=h264

Playback works. We get the H264 stream from the camera. Now let's transmit over the network via HTTP:

cvlc v4l2:///dev/video1:chroma=h264:width=800:height=600 --sout '#standard{access=http,mux=ts,dst=localhost:8080,name=stream,mime=video/ts}' -vvv

(on the network, you need to use the IP address of your local device instead of localhost)

Starting the receiver:

mplayer http://localhost:8080/

So MPEG-TS allows mplayer to connect to the stream at any time with a slight delay (probably waiting for a keyframe or sth).

If you use mux = avi, mime = video / avi, you need to run mplayer while you are running vlc, otherwise mplayer will not start playing.

Ffmpeg error

If you try ffmpeg instead of vlc for playback, you will probably encounter this problem: ffmpeg.org/trac/ffmpeg/ticket/1387
As soon as you start ffmpeg, it will switch to “Video: rawvideo (YUY2 / 0x32595559)”
While he is taking MJPEG, he will not work with H.264
The same thing happens when using qv4l2: the beginning of the capture in H264 mode actually gives you the YUY2 video window

Playback hardware decoded video

Mplayer is able to play the framebuffer, make sure that X is not working and enter:

mplayer http://192.168.1.100:8080/ -fs -framedrop -vo fbdev

You can also play in the X window if you want to: run X from the command line:

Xorg -retro &> /dev/zero

There are a lot of fbdev error messages that we silently reject in / dev / zero.

Playing a video stream from a LAN vlc source (see the section above):

DISPLAY=":0" mplayer http://192.168.1.100:8080/ -fs -framedrop

If we do this, we will notice that we are not using Xvideo or any hardware acceleration yet.
linux-sunxi.org/CedarX seems to provide us with a VPU library, which, unfortunately, is the unprotected thing Allwinner
But there is a hacked VLC that can use it: linux-sunxi.org/VLC
Compiled successfully

Raspberry pi

In case you experience artifacts in the picture, and streams from Raspberry Pi use:

cvlc v4l2:///dev/video0:chroma=h264 ...

You may need to update the firmware:

apt-get install rpi-update raspi-config
rpi-update
reboot

Summary of cvlc and v4l2 translation

Tests showed that the processor load on Beaglebone Blue is no more than 30%, which is very good for a weak processor, but it is not clear how to push this video into ROS.

If the wifi bandwidth is not enough, the video delay is greatly increased, in theory it can be solved by switching from tcp to udp, the cvlc documentation mentions the possibility of broadcasting by udp.

Broadcasting hardware-encoded video from a Logitech C920 camera via Gstreamer

An original developer article supporting this method in Gstreamer.
Install gstreamer on Ubuntu 16.04

sudo apt-get install libgstreamer1.0-0 gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-doc gstreamer1.0-tools
sudo apt-get install libgstreamer-plugins-base1.0-dev

Source uvch264src allows you to:

At the same time capture hardware-encoded h264 video stream and video stream finder.
Adjust hardware video encoding (bitrate, keyframes, etc.)

The secondary viewfinder source is an interesting concept. In addition to the main H.264 stream, you get a low-resolution secondary stream in MJPG or YUV format. I'm not sure if this is a feature of the camera, driver, or gstreamer source block, but I hope it will be useful for our Beaglebone cameras, provided the function does not use a processor that is too large to convert between raw and jpg.

Below is an example of a pipeline that displays high-resolution H.264 stream and viewfinder stream:

gst-launch-1.0 -v -e uvch264src device=/dev/video0 name=src auto-start=true src.vfsrc ! queue ! video/x-raw,format=YUY2,width=320,height=240,framerate=10/1 ! xvimagesink sync=false src.vidsrc ! queue ! video/x-h264,width=1280,height=720,framerate=30/1 ! h264parse ! avdec_h264 ! xvimagesink sync=false

The H.264 primary stream is decoded using the avdec_h264 block, available through the gst-libav package, see notes on the oz9aec wiki . The viewfinder panel can be connected to the fake link if it is not needed, but I do not think that it can be turned off (you can turn off all the same). At least, this is the impression that I read the original KaKaRoTo blog post .

You are likely to be overloaded when you see a large number of parameters that can be configured for this source block:

gst-inspect-1.0 uvch264src

List of parameters uvch264src

Factory Details:
Rank none (0)
Long-name UVC H264 Source
Klass Source/Video
Description UVC H264 Encoding camera source
Author Youness Alaoui <youness.alaoui@collabora.co.uk>

Plugin Details:
Name uvch264
Description UVC compliant H264 encoding cameras plugin
Filename /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstuvch264.so
Version 1.8.3
License LGPL
Source module gst-plugins-bad
Source release date 2016-08-19
Binary package GStreamer Bad Plugins (Ubuntu)
Origin URL launchpad.net/distros/ubuntu/+source/gst-plugins-bad1.0

GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstBin
+----GstBaseCameraSrc
+----GstUvcH264Src

Реализованные интерфейсы:
GstChildProxy

Pad Templates:
SRC template: 'vidsrc'
Availability: Always
Capabilities:
video/x-raw
format: { I420, YV12, YUY2, UYVY, AYUV, RGBx, BGRx, xRGB, xBGR, RGBA, BGRA, ARGB, ABGR, RGB, BGR, Y41B, Y42B, YVYU, Y444, v210, v216, NV12, NV21, NV16, NV61, NV24, GRAY8, GRAY16_BE, GRAY16_LE, v308, RGB16, BGR16, RGB15, BGR15, UYVP, A420, RGB8P, YUV9, YVU9, IYU1, ARGB64, AYUV64, r210, I420_10LE, I420_10BE, I422_10LE, I422_10BE, Y444_10LE, Y444_10BE, GBR, GBR_10LE, GBR_10BE, NV12_64Z32, A420_10LE, A420_10BE, A422_10LE, A422_10BE, A444_10LE, A444_10BE }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
image/jpeg
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
video/x-h264
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
stream-format: { byte-stream, avc }
alignment: au
profile: { high, main, baseline, constrained-baseline }

SRC template: 'imgsrc'
Availability: Always
Capabilities:
EMPTY

SRC template: 'vfsrc'
Availability: Always
Capabilities:
video/x-raw
format: { I420, YV12, YUY2, UYVY, AYUV, RGBx, BGRx, xRGB, xBGR, RGBA, BGRA, ARGB, ABGR, RGB, BGR, Y41B, Y42B, YVYU, Y444, v210, v216, NV12, NV21, NV16, NV61, NV24, GRAY8, GRAY16_BE, GRAY16_LE, v308, RGB16, BGR16, RGB15, BGR15, UYVP, A420, RGB8P, YUV9, YVU9, IYU1, ARGB64, AYUV64, r210, I420_10LE, I420_10BE, I422_10LE, I422_10BE, Y444_10LE, Y444_10BE, GBR, GBR_10LE, GBR_10BE, NV12_64Z32, A420_10LE, A420_10BE, A422_10LE, A422_10BE, A444_10LE, A444_10BE }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
image/jpeg
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]

Element Flags:
no flags set

Bin Flags:
no flags set

Element Implementation:
Has change_state() function: 0x7ff438f22ba0

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
SRC: 'vfsrc'
SRC: 'imgsrc'
SRC: 'vidsrc'

Element Properties:
name: The name of the object
flags: считываемый, записываемый
String. Default: «uvch264src0»
parent: The parent of the object
flags: считываемый, записываемый
Object of type «GstObject»
async-handling: The bin will handle Asynchronous state changes
flags: считываемый, записываемый
Boolean. Default: false
message-forward: Forwards all children messages
flags: считываемый, записываемый
Boolean. Default: false
mode: The capture mode (still image capture or video recording)
flags: считываемый, записываемый
Enum «GstCameraBin2Mode» Default: 2, «mode-video»
(1): mode-image — Still image capture (default)
(2): mode-video — Video recording
zoom: Digital zoom factor (e.g. 1.5 means 1.5x)
flags: считываемый, записываемый
Float. Range: 1 — 3,402823e+38 Default: 1
max-zoom: Digital zoom factor (e.g. 1.5 means 1.5x)
flags: считываемый
Float. Range: 1 — 3,402823e+38 Default: 10
ready-for-capture: Informs this element is ready for starting another capture
flags: считываемый
Boolean. Default: true
post-previews: If capture preview images should be posted to the bus
flags: считываемый, записываемый
Boolean. Default: true
preview-caps: The caps of the preview image to be posted (NULL means ANY)
flags: считываемый, записываемый
ANY

preview-filter: A custom preview filter to process preview image data
flags: считываемый, записываемый
Object of type «GstElement»
auto-start: Automatically starts capture when going to the PAUSED state
flags: считываемый, записываемый
Boolean. Default: false
colorspace-name: The name of the colorspace element
flags: считываемый, записываемый, может быть равно только состоянию NULL или READY
String. Default: «videoconvert»
jpeg-decoder-name: The name of the jpeg decoder element
flags: считываемый, записываемый, может быть равно только состоянию NULL или READY
String. Default: «jpegdec»
num-clock-samples: Number of clock samples to gather for the PTS synchronization (-1 = unlimited)
flags: считываемый, записываемый, может быть равно состоянию NULL, READY, PAUSED или PLAYING
Integer. Range: 0 — 2147483647 Default: 0
num-buffers: Number of buffers to output before sending EOS (-1 = unlimited)
flags: считываемый, записываемый
Integer. Range: -1 — 2147483647 Default: -1
device: Device location
flags: считываемый, записываемый
String. Default: "/dev/video0"
device-name: Name of the device
flags: считываемый
String. Default: ""
initial-bitrate: Initial bitrate in bits/second (static control)
flags: считываемый, записываемый, может быть равно только состоянию NULL или READY
Unsigned Integer. Range: 0 — 4294967295 Default: 3000000
slice-units: Slice units (static control)
flags: считываемый, записываемый, может быть равно только состоянию NULL или READY
Unsigned Integer. Range: 0 — 65535 Default: 4
slice-mode: Defines the unit of the slice-units property (static control)
flags: считываемый, записываемый, может быть равно только состоянию NULL или READY
Enum «UvcH264SliceMode» Default: 3, «slice/frame»
(0): ignored — Ignored
(1): bits/slice — Bits per slice
(2): MBs/slice — MBs per Slice
(3): slice/frame — Slice Per Frame
iframe-period: Time between IDR frames in milliseconds (static control)
flags: считываемый, записываемый, может быть равно только состоянию NULL или READY
Unsigned Integer. Range: 0 — 65535 Default: 10000
usage-type: The usage type (static control)
flags: считываемый, записываемый, может быть равно только состоянию NULL или READY
Enum «UvcH264UsageType» Default: 1, «realtime»
(1): realtime — Realtime (video conferencing)
(2): broadcast — Broadcast
(3): storage — Storage
(4): ucconfig0 — UCConfig 0
(5): ucconfig1 — UCConfig 1
(6): ucconfig2q — UCConfig 2Q
(7): ucconfig2s — UCConfig 2S
(8): ucconfig3 — UCConfig 3
entropy: Entropy (static control)
flags: считываемый, записываемый, может быть равно только состоянию NULL или READY
Enum «UvcH264Entropy» Default: 0, «cavlc»
(0): cavlc — CAVLC
(1): cabac — CABAC
enable-sei: Enable SEI picture timing (static control)
flags: считываемый, записываемый, может быть равно только состоянию NULL или READY
Boolean. Default: false
num-reorder-frames: Number of B frames between the references frames (static control)
flags: считываемый, записываемый, может быть равно только состоянию NULL или READY
Unsigned Integer. Range: 0 — 255 Default: 0
preview-flipped: Horizontal flipped image for non H.264 streams (static control)
flags: считываемый, записываемый, может быть равно только состоянию NULL или READY
Boolean. Default: false
leaky-bucket-size: Size of the leaky bucket size in milliseconds (static control)
flags: считываемый, записываемый, может быть равно только состоянию NULL или READY
Unsigned Integer. Range: 0 — 65535 Default: 1000
rate-control: Rate control mode (static & dynamic control)
flags: считываемый, записываемый, может быть равно состоянию NULL, READY, PAUSED или PLAYING
Enum «UvcH264RateControl» Default: 1, «cbr»
(1): cbr — Constant bit rate
(2): vbr — Variable bit rate
(3): qp — Constant QP
fixed-framerate: Fixed framerate (static & dynamic control)
flags: считываемый, записываемый, может быть равно состоянию NULL, READY, PAUSED или PLAYING
Boolean. Default: false
max-mbps: The number of macroblocks per second for the maximum processing rate
flags: считываемый
Unsigned Integer. Range: 0 — 4294967295 Default: 0
level-idc: Level IDC (dynamic control)
flags: считываемый, записываемый, может быть равно состоянию NULL, READY, PAUSED или PLAYING
Unsigned Integer. Range: 0 — 255 Default: 40
peak-bitrate: The peak bitrate in bits/second (dynamic control)
flags: считываемый, записываемый, может быть равно состоянию NULL, READY, PAUSED или PLAYING
Unsigned Integer. Range: 0 — 4294967295 Default: 3000000
average-bitrate: The average bitrate in bits/second (dynamic control)
flags: считываемый, записываемый, может быть равно состоянию NULL, READY, PAUSED или PLAYING
Unsigned Integer. Range: 0 — 4294967295 Default: 3000000
min-iframe-qp: The minimum Quantization step size for I frames (dynamic control)
flags: считываемый, записываемый, может быть равно состоянию NULL, READY, PAUSED или PLAYING
Integer. Range: -127 — 127 Default: 10
max-iframe-qp: The minimum Quantization step size for I frames (dynamic control)
flags: считываемый, записываемый, может быть равно состоянию NULL, READY, PAUSED или PLAYING
Integer. Range: -127 — 127 Default: 46
min-pframe-qp: The minimum Quantization step size for P frames (dynamic control)
flags: считываемый, записываемый, может быть равно состоянию NULL, READY, PAUSED или PLAYING
Integer. Range: -127 — 127 Default: 10
max-pframe-qp: The minimum Quantization step size for P frames (dynamic control)
flags: считываемый, записываемый, может быть равно состоянию NULL, READY, PAUSED или PLAYING
Integer. Range: -127 — 127 Default: 46
min-bframe-qp: The minimum Quantization step size for B frames (dynamic control)
flags: считываемый, записываемый, может быть равно состоянию NULL, READY, PAUSED или PLAYING
Integer. Range: -127 — 127 Default: 10
max-bframe-qp: The minimum Quantization step size for B frames (dynamic control)
flags: считываемый, записываемый, может быть равно состоянию NULL, READY, PAUSED или PLAYING
Integer. Range: -127 — 127 Default: 46
ltr-buffer-size: Total number of Long-Term Reference frames (dynamic control)
flags: считываемый, записываемый, может быть равно состоянию NULL, READY, PAUSED или PLAYING
Integer. Range: 0 — 255 Default: 0
ltr-encoder-control: Number of LTR frames the device can control (dynamic control)
flags: считываемый, записываемый, может быть равно состоянию NULL, READY, PAUSED или PLAYING
Integer. Range: 0 — 255 Default: 0

Element Actions:
«get-enum-setting»: gboolean user_function (GstElement* object,
gchararray arg0,
gpointer arg1,
gpointer arg2);
«get-boolean-setting»: gboolean user_function (GstElement* object,
gchararray arg0,
gpointer arg1,
gpointer arg2);
«get-int-setting»: gboolean user_function (GstElement* object,
gchararray arg0,
gpointer arg1,
gpointer arg2,
gpointer arg3);
«start-capture»: void user_function (GstElement* object);
«stop-capture»: void user_function (GstElement* object);

Many of these parameters can be used to configure parameters in H.264 compression. I find it quite surprising, given that the compression is performed by a chip inside the camera. The following example increases the default data transfer rate from 3 to 5 Mbps and reduces the default keyframe interval from 10 to 3 seconds:

gst-launch-1.0 -v -e uvch264src initial-bitrate=5000000 average-bitrate=5000000 iframe-period=3000 device=/dev/video0 name=src auto-start=true src.vfsrc ! queue ! video/x-raw,format=YUY2,width=320,height=240,framerate=10/1 ! xvimagesink sync=false src.vidsrc ! queue ! video/x-h264,width=1280,height=720,framerate=30/1 ! h264parse ! avdec_h264 ! xvimagesink sync=false

On the other hand, there are no parameters for adjusting the usual webcam parameters, such as contrast, brightness, focus, etc. Thus, we still need external tools, such as v4l2.ctl, or the variant mentioned in next section.

v4l2src

It looks like we can also use the good old v4l2src to capture H.264 encoded streams from the Logitech C920:

gst-launch-1.0 -v -e v4l2src device=/dev/video1 ! queue ! video/x-h264,width=1280,height=720,framerate=30/1 ! h264parse ! avdec_h264 ! xvimagesink sync=false

This is probably due to the evolution of both gstreamer and V4L2. This option is easier if you do not need to change the H.264 compression settings, and this gives us access to camera settings, such as brightness and contrast. It also has the advantage of being in a good package of plug-ins, which may be better preserved in the future.

Network Broadcast via Gstreamer

On the receiving computer, launch the receiver:

gst-launch-1.0 -v udpsrc port=6666 ! application/x-rtp, encoding-name=H264 ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! autovideosink

On the device with the camera, we launch the transfer of hardware-encoded video:

gst-launch-1.0 uvch264src name=src auto-start=true src.vidsrc ! video/x-h264,width=160,height=120,framerate=30/1 ! h264parse ! rtph264pay ! udpsink host=192.168.1.196 port=6666

For help in writing this code, a special thanks to Amomum

Summary of delayed video broadcast via Gstreamer

Even with a resolution of 1920 * 1080, the processor load on Beaglebone Blue did not exceed 30%, and the video delay was no more than 0.3 seconds. Yes, since the video in this version was broadcast on udp, then with the loss of packets, the picture falls apart to the next keyframe, but we can configure it to send keyframes more often.

ROS tasks

It remains now only to stuff the received video in ROS, if anyone can help with this, write in a personal.

Tags: