Why did we do VOD on WebRTC
VOD is about video on demand, i.e. playing regular videos, as is done on YouTube or another streaming service. WebRTC is a low latency realtime video. You may ask - how can these two things be connected? Come under the cat for details.
Support and bugfix
It all started, as usual, with a support. We were approached by a female programmer, presumably from an outsourcing company in India, which participated in the development of a mobile application for telemedicine. One of the client’s Wishlist was recording WebRTC video chat from an iOS application, followed by playback in the same iOS application. This recording worked, but when played back using standard iOS SDK tools, green artifacts were detected. More precisely, not even artifacts, but quite clear rectangular areas of green color, occupying ¾ of the screen. Of course, this was no good, and we began to study the problem.
Play recorded WebRTC video through AVPlayerViewController
In the mobile video playback application, we used standard MPMoviePlayer or AVPlayerViewController components that can play mp4 video if you specify its http URL, for example http: //host/sample.mp4 . These components played back MP4 video normally, but in the case of broadcasts recorded from the iOS application, the green area did not disappear anywhere and spoiled everything.
WebRTC dynamically changes the resolution of the stream
It turned out that the green artifacts in the recording were due to the fact that for WebRTC it is normal to change the resolution of the video on the fly. At the same time, in the mp4 recording file, we have wonderful frames of different sizes, more precisely, first a sequence of one size 640x480, then another 320x240, etc. Such tricks are excellently played out, for example, by VLC, without any artifacts, and the built-in video playback components in iOS via HTTP give green artifacts when changing the video resolution in the bitstream.
Let's start the broadcast from the iOS application and make sure that this is true. For testing, you can use our mobile application for iOS - Two Way Streaming based on the iOS SDK and the demo server WCS5-EU .
This is the application that streams the stream to the server:
And it looks like the change in resolution of the video sent from the mobile application in time:
It can be seen from the video stream monitoring graph that the resolution of the picture dynamically changes depending on the capabilities of the mobile device to compress video, work with the network, etc. WebRTC changes resolution to target low latency.
WebRTC VOD as a solution
A way out of this situation would be rescaling with transcoding, i.e. decoding frames, converting to a single resolution, for example 640x480 and recording in this resolution. But if you do this with every stream published to the server, the CPU resources will quickly run out on 10-20 video streams. Therefore, we needed some kind of solution that did not involve transcoding.
We thought - if WebRTC streams a video with such changes in resolution, then it should be able to and play back the video recorded in this way. It turns out that if we read the mp4 file and feed it via the WebRTC browser or mobile application, then everything should be in order and the green rectangles on the iOS application screen should go away.
It remains to implement the read-out of recorded mp4 and forwarding to the Web Call Server engine for further conversion to WebRTC. The first tests showed good results - the green rectangles disappeared.
So we got VOD with playback not only via WebRTC, but also across all supported protocols and technologies: RTMP, RTMFP, RTSP, HTML5 Canvas, Media Source, HLS, WebRTC.
WebRTC VOD - Live Broadcast
Then the question arose - “What if users want to broadcast the video as a stream, all at once and at the same time?”.
As a result, I had to make two types of VOD.
The first is a personal VOD . For each user who wants to play the video, a separate channel is created through which the video is played from the very beginning of the video.
The second is live VOD . If the user started playing the video, and the second user connected later, they will watch the video as a live broadcast in real time, i.e. the server will stream exactly one stream from the video, to which both users will be connected and will be able to simultaneously watch, for example, a football match and comment on the actions of players.
In our player and in the API, in order to play a stream, you need to know its name.
For VOD, we introduced the following schemes:
If you want to play the video personally, we transfer the stream name like this:
vod://sample.mp4If you want to make a full-fledged online broadcast from the video, then the stream name will be like this:
vod-live://sample.mp4The sample.mp4 file itself must be in the WCS_HOME / media server folder and have the MP4 / H.264 + AAC format.
What happened to the iOS application? - you ask.
All is well. iOS application plays VOD video on WebRTC without green rectangles.
WebRTC VOD in Web Player
Now let's see how WebRTC VOD looks on the web. To do this, on the server side, copy the mp4 file to the / usr / local / FlashphonerWebCallServer / media folder. Let it be the famous Big Buck Bunny, sample.mp4.
We open the page with the test player , indicate the name of the stream vod: //sample.mp4 and click Test Now.
The player starts playing the stream through WebRTC. In chrome: // webrtc-internals you can see playback graphs:
In the end, the story ended well. We fixed a bug with a green screen when playing an mp4 recording of a broadcast in an iOS application and made a WebRTC-VOD function for web browsers and mobile devices running iOS and Android.
References
- Two Way Streaming live streaming iOS app
- Web player - an example of a player for playing VOD - WebRTC streams
- Server Web Call Server 5 , the dealer mp4 videos on WebRTC
- WebRTC SDK for iOS - library for developing streaming applications for iOS