Interserver WebRTC
WebRTC can operate Peer-to-Peer and Peer-to-Server, where a peer, as a rule, is a browser or a mobile application. In this article, we will talk about how WebRTC works in Server-to-Server mode, why it is needed and how it works.
Scaling, Origin-Edge
Where cross-server WebRTC might be needed? The Origin-Edge pattern immediately comes to mind, which is used to scale broadcast to a large audience.
- The user sends a WebRTC video stream to the Origin-WebRTC server from a browser or mobile device.
- Origin server streams to several Edge servers.
- Edge servers distribute the stream to end users on their browsers and mobile applications.
In modern CDNs, the RTMP protocol is actively used for video delivery when publishing a stream to an Origin server and when distributing a stream to an Edge server, and end users receive the picture via HTTP.
An advantage of WebRTC over this approach can be guaranteed low latency of translation, which cannot be achieved by RTMP / HTTP delivery means, especially if the nodes are geographically separated.
However, for us, the cross-server WebRTC did not start with scaling.
Stress tests
They were always in one form or another. Automatic and semi-automatic, synthetic and close to what users do. We used and use load testing to catch multi-threaded bugs, control resource leaks, optimizations, and many other things that cannot be caught in ordinary testing.
In recent tests, we raised Linux servers without a GUI in the cloud and ran scripts that started many Google Chrome browser processes on the X11 virtual desktop. Thus, real WebRTC browsers were launched, which pulled and played WebRTC video streams from the Web Call Server, thereby creating a load as close as possible to the real one. For the server, it looked like a real user opened the browser and took the video stream, fully utilizing the browser’s WebRTC stack, including video decoding and rendering.
The disadvantage of this testing method was the performance of the test system. It’s hard enough to run many Chrome processes on the same Linux system, even when using a powerful server with memory megameters and CPU. As a result, we had to raise many servers and somehow manage / monitor them.
Another limitation was the lack of flexibility - we could not control the processes of chromium. There were only two operations:
- Raise the process and open the desired URL
- Kill the process. When the url was opened, the HTML page was loaded and there was an automatic connection to the server, already using JavaScript, Websocket + WebRTC. So the audience load was modeled.
A flexible tool for load testing was required, which would allow to give a server load close to real, control the testing programmatically and ensure high testing performance.
WebRTC Server-Server Testing
We came to the conclusion that the nodes of our server themselves can become load generators if they are correctly hooked to the tested servers.
This idea was implemented as WebRTC pulling . One WCS server can pull a stream from another WCS server via WebRTC. To do this, we introduced an internal abstraction of WebRTCAgent - an object that rises on a test node and pulls a WebRTC stream from the tested node, connecting to the tested node via Websocket + WebRTC.
After that, we took out the management of WebRTCAgent on REST. As a result, load testing was reduced to calling / pull - methods on the REST interface of the testing node.
When using cross-server WebRTC, we were able to increase the load testing performance by about 7 times and significantly reduce the use of resources, compared with the scheme when we started the Google Chrome processes.
So, we managed to pull WebRTC streams from other servers. The testing server connected to the test via Websocket, and as a decent browser, it established an ICE connection, DTLS and pulled SRTP streams to itself - it turned out true WebRTC pulling.
There was very little left to get a full Origin-Edge model. To do this, it was necessary to forward such pulling to the WCS server engine as a published stream, i.e. make it look like a stream from a webcam, and WCS can already distribute such streams using all available protocols: WebRTC, RTMP, RTMFP, Websocket Canvas, Websocket MSE, RTSP, HLS.
Origin-Edge on WebRTC
It turned out that we did cross-server WebRTC for load testing, but as a result we implemented the Origin-Edge scheme for scaling WebRTC broadcasts, and here is how it works:
The green line shows how video traffic flows.
1. A user from a browser or mobile application, using a webcam, sends a WebRTC video stream named stream1 to the WCS1 server - Origin. The process of sending a video stream using the Two Way Streaming web example looks like this:
And this is the JavaScript code that is responsible for publishing the video stream through the Web API (Web SDK):
session.createStream({name:'steram1',display:document.getElementById('myVideoDiv')}).publish();2. Next, we turn to the WCS2 - Edge server using the REST / HTTP API and give the command to pick up the video stream from the Origin server.
/rest-api/pull/startup
{
uri: wss://wcs1-origin.com:8443
remoteStreamName: stream1,
localStreamName: stream1_edge,
}The WCS2 server connects via web sockets to the WCS1 server at wss: //wcs1-origin.com: 8443 and receives a stream named stream1 via WebRTC.
After that, you can run the REST command
/rest-api/pull/find_allto display all current pull connections.
Or a team
/rest-api/pull/terminateto close the pull connection to the Origin WebRTC server.
3. And finally, we take the stream from the Edge server via WebRTC in the player . Enter the name of the stream stream1_edge and start playing it.
WCS server supports several playback methods. To change the technology, simply drag up MSE or WSPlayer to play the stream not by WebRTC, but by MSE (Media Source Extensions) or in WSPlayer (Websocket - player for iOS Safari).
Thus, the Orign-Edge scheme worked and we got the scalability of the WebRTC server with low latency.
It should be noted that scalability worked earlier on RTMP. The WCS server can re-publish incoming WebRTC streams to one or more servers using the RTMP protocol. In this case, support for cross-server WebRTC was a step towards reducing the overall system latency.
And again about load testing
For normal load testing, it remains to write a Web interface in the form of a REST client that manages pull sessions. This interface was named Console and took the following form:
Using the console, you can pull individual WebRTC streams using the current node as an Edge server. Through the same interface, you can add one or more nodes and run load tests on them with an assessment of performance.
Much remains to be done and debased. In particular, it is interesting to work with dynamic bitrates on the server-to-server WebRTC channel and compare server-to-server throughput with RTMP. But now we already have Orign-Edge on WebRTC and the right load tests, giving close to the real load, which is good news!
References
WCS5 - WebRTC server Web Call Server 5
Two Way Streaming - example of broadcasting a video stream from a browser
WebRTC Player - example of playing a video stream in a browser with the possibility of changing WebRTC, MSE, Flash, WSPlayer technologies
