We broadcast WebRTC, RTSP and RTMP streams to Media Source Extensions via Websocket protocol
Media source extensions
Media Source Extensions (hereinafter MSE) is a browser API that allows you to play audio and video through the corresponding HTML5 tags <audio /> and <video />.
To play a piece of audio or video, you need to feed this chunk to this element through the MSE API. Based on MSE built HLS-players. HLS fragments are transmitted to the MSE and displayed in the player.
Let's look at its Can I Use in more detail.
As you can see from the table, MSE is supported in all recent browsers, except iOS Safari and Opera Mini. In this regard, it is very similar to WebRTC, and even bypasses WebRTC in availability in Safari and IE. I must say that Safari 11 already has WebRTC, but at the time of this writing, it has not yet been released.
It is worth noting that MSE and WebRTC are technologies of different weight categories. Roughly speaking, MSE is just a player, and WebRTC is both a player and streamer and calls (real-time streaming with low latency).
Thus, if you only need a player and there are no realtime requirements (delays of less than 1 second), then MSE is a good choice for playing video streams. If you need realtime without plugins, then there are no options. Need to take WebRTC.
Media source extensions | WebRTC |
| Player only | Player and Streamer |
| Average delay | Real time (delay less than 1 second) |
WebRTC to MSE
Suppose WebRTC acts as a streamer. This means that the browser or mobile application sends the video stream to the server.
The fastest way to pick up a video stream in MSE is to connect to the server using the Websocket protocol and deliver the stream to the browser. In the player, parse the received stream and transfer it to the MSE for playback.
Broadcast Scheme:
- Broadcast browser sends H.264 + Opus stream to WebRTC server
- WebRTC server distributes the stream using the Websocket H.264 + AAC protocol
- The viewer browser opens the stream and gives the H.264 and AAC frames for playback in the MSE.
- The player plays audio and video.
Thus, when using Media Source Extensions as a player, the video part of the WebRTC stream with the H.264 codec passes to the player without transcoding (proxied), which helps to keep the CPU usage on the server low.
The Opus audio codec is transcoded to AAC for successful reading in MSE, but audio transcoding is already much less resource-intensive than video.
Examples
These streamer and MSE player examples use Web Call Server 5 as the server that converts the WebRTC video stream for use in Media Source Extensions.
We use the Two Way Streaming web application as a streamer , which sends the WebRTC video stream to the server.
As a player, we use a test player with support for Websockets and MSE.
The MSE player code works through a high-level API.
Create a div - an element on an HTML page
<divid="myMSE"style="width:640;height:360"></div>And then we call the playback of the video stream.
We pass the div - the element with the display parameter and the MSE player will be inserted into this div block.
session.createStream({name:"stream22",display:document.getElementById("myMSE")}).play();Similarly, publishing a WebRTC stream from another page works.
The stream is assigned a unique name stream22, which will be indicated during playback.
session.createStream({name:"stream22",display:document.getElementById("myWebCam")}).publish();
The full code for the tape drive and player can be found on github.
- streamer
- player
RTSP to MSE
Another case of using MSE over Websockets is to play video from an IP camera or other system that provides video stream over RTSP.
The IP camera, as a rule, natively supports H.264 and AAC codecs, so the codecs are exactly the same as those used on MSE. This helps to avoid transcoding that consumes CPU resources.
The translation scheme is as follows:
- The browser requests an RTSP stream.
- The server establishes a connection with the camera and requests this stream from the camera via RTSP.
- The camera gives the RTSP stream. Streaming begins.
- The RTSP stream is converted to Websockets on the server side and down to the browser.
- The browser streams the stream to the MSE player for playback.
To work with the RTSP stream, the same player is used, which we showed above. It simply sets the full URL of the RTSP stream instead of the name.
At the time of writing, we have tested the MSE player in Chrome, Firefox, Safari.
RTMP to MSE
RTMP is actually the standard for delivering live content from a user to a CDN.
Therefore, you need to make sure that standard RTMP streams with H.264 and AAC codecs will play Media Source Extensions correctly.
This can be quickly checked using an encoder such as Wirecast (alternatively, you can use FMLE, ffmpeg, OBS encoder and others).
1. Open Wirecast and the Output menu. Set the server address and the name of the RTMP stream as stream44.
2. Stream the stream to the server.
3. Play the stream in the MSE player.
Comparison with other playback technologies
Now let's look at MSE in comparison with other video playback technologies in the browser.
For comparison, we take 4 technologies:
- WebRTC
- Flash
- Hls
- Canvas + Web Audio
MSE | WebRTC | Flash | Hls | Canvas | |
| Support Chrome, Firefox | Yes | Yes | Yes | Yes | Yes |
| IOS Safari Support | Not | Not | Not | Yes | Yes |
| Mac Safari Support | Yes | Not | Yes | Yes | Yes |
| Delay | 3 | 0.5 | 3 | 20 | 3 |
| Full HD Resolution | Yes | Yes | Yes | Yes | Not |
Thus, if you need to play a live stream in a browser with a slight delay, MSE is a good solution, replacing Flash in recent browsers.
If the delay is absolutely not important, it is worth using HLS, as the most universal option.
If low latency is critical, you need to use a combination of WebRTC, Flash, Canvas, MSE to cover most browsers.
The delay should be less than 1 second and without plug-ins, we use WebRTC.
References
Two Way Streaming - demo translation of WebRTC stream to
Source server - source code for broadcasting WebRTC stream
Player - demo player with Media Source Extension Extension
Source - source code of player with MSE support
Web Call Server 5 - server that supports WebRTC and RTSP broadcasts with stream conversion for Media Source Extensions
Can I Use MSE - browsers that support MSE
Can I Use WebRTC - browsers that support WebRTC