We pull the YouTube video and distribute it in real time via WebRTC
The task is as follows. Conduct a joint viewing of the video with YouTube in real time by several users. Viewers should receive video at the same time, with a minimum delay.
Roller like a stream
It is clear that if each of the viewers simply starts playing the video, the goal will not be achieved, because one will receive the video faster, the other slower. Uncontrolled spread will occur.
In order to prevent scatter, you need to distribute this video to everyone at the same time. This can be realized if you wrap the movie in Live-stream. We will show how to do this using the link of this library with ffmpeg.
We need to implement the scheme described above. Namely, ydl connects to YouTube and starts downloading the video. FFmpeg picks up the downloaded movie, wraps it in an RTMP stream and sends it to the server. The server distributes the received stream as WebRTC in real time.
Youtube-dl installation
We start by installing youtube-dl. The installation process on Linux is extremely simple and described in detail in the Readme under Lin and under Win .
1. Download.
curl -L https://yt-dl.org/downloads/latest/youtube-dl -o /usr/local/bin/youtube-dl2. We give launch rights
chmod a+rx /usr/local/bin/youtube-dlThat's all. YouTube download is ready to go.
Take a YouTube video and look at its meta data:
youtube-dl --list-formats https://www.youtube.com/watch?v=9cQT4urTlXMThe result will be like this:
[youtube] 9cQT4urTlXM: Downloading webpage
[youtube] 9cQT4urTlXM: Downloading video info webpage
[youtube] 9cQT4urTlXM: Extracting video information
[youtube] 9cQT4urTlXM: Downloading MPD manifest
[info] Available formats for 9cQT4urTlXM:
format code extension resolution note
171 webm audio only DASH audio 8k , vorbis@128k, 540.24KiB
249 webm audio only DASH audio 10k , opus @ 50k, 797.30KiB
250 webm audio only DASH audio 10k , opus @ 70k, 797.30KiB
251 webm audio only DASH audio 10k , opus @160k, 797.30KiB
139 m4a audio only DASH audio 53k , m4a_dash container, mp4a.40.5@ 48k (22050Hz), 10.36MiB
140 m4a audio only DASH audio 137k , m4a_dash container, mp4a.40.2@128k (44100Hz), 27.56MiB
278 webm 256x144 144p 41k , webm container, vp9, 30fps, video only, 6.54MiB
242 webm 426x240 240p 70k , vp9, 30fps, video only, 13.42MiB
243 webm 640x360 360p 101k , vp9, 30fps, video only, 20.55MiB
160 mp4 256x144 DASH video 123k , avc1.4d400c, 15fps, video only, 24.83MiB
134 mp4 640x360 DASH video 138k , avc1.4d401e, 30fps, video only, 28.07MiB
244 webm 854x480 480p 149k , vp9, 30fps, video only, 30.55MiB
135 mp4 854x480 DASH video 209k , avc1.4d401f, 30fps, video only, 42.42MiB
133 mp4 426x240 DASH video 274k , avc1.4d4015, 30fps, video only, 57.63MiB
247 webm 1280x720 720p 298k , vp9, 30fps, video only, 59.25MiB
136 mp4 1280x720 DASH video 307k , avc1.4d401f, 30fps, video only, 62.58MiB
17 3gp 176x144 small , mp4v.20.3, mp4a.40.2@ 24k
36 3gp 320x180 small , mp4v.20.3, mp4a.40.2
43 webm 640x360 medium , vp8.0, vorbis@128k
18 mp4 640x360 medium , avc1.42001E, mp4a.40.2@ 96k
22 mp4 1280x720 hd720 , avc1.64001F, mp4a.40.2@192k (best)Ffmpeg installation
Next, set ffmpeg with standard spells:
wget http://ffmpeg.org/releases/ffmpeg-3.3.4.tar.bz2
tar -xvjf ffmpeg-3.3.4.tar.bz2
cd ffmpeg-3.3.4
./configure --enable-shared --disable-logging --enable-gpl --enable-pthreads --enable-libx264 --enable-librtmp
make
make installCheck what happened
ffmpeg -vNow the fun part. The youtube-dl library is for download. It's called YouTube Download. Those. You can download the youtube movie in its entirety and after that just stream it through ffmpeg as a file.
But imagine such a user case. Marketer, manager and programmer are sitting in a web conference. The marketer wants to show everyone in real time a YouTube video that weighs, say, 300 megabytes. Agree, there will be some awkwardness, if you need to deflate the entire movie before you start showing it.
- The marketer says - “And now, colleagues, let's see this video with cats, it fully meets our strategy for entering the market”, and presses the “show everyone video” button.
- The preloader appears on the screen: “Wait, the video with the cats is downloading. It takes no more than 10 minutes. ”
- The manager is going to drink coffee, and the programmer is reading habr.
In order not to make people wait, we need real time. You need to pick up the video directly during download, on the fly to wrap in a stream and distribute in real time. Next we will show how to do it.
Transferring data from youtube-dl to ffmpeg
The youtube-dl grabber saves the stream to the file system. You need to connect to this stream and organize the reading from the ffmpeg file as it is downloaded using youtube-dl.
To combine these two processes: downloading and streaming ffmpeg, we need a small binding script.
#!/usr/bin/python
import subprocess
import sys
def show_help():
print 'Usage: '
print './streamer.py url streamName destination'
print './streamer.py https://www.youtube.com/watch?v=9cQT4urTlXM streamName rtmp://192.168.88.59:1935/live'
return
def streamer() :
url = sys.argv[1]
if not url :
print 'Error: url is empty'
return
stream_id = sys.argv[2]
if not stream_id:
print 'Error: stream name is empty'
return
destination = sys.argv[3]
if not destination:
print 'Error: destination is empty'
return
_youtube_process = subprocess.Popen(('youtube-dl','-f','','--prefer-ffmpeg', '--no-color', '--no-cache-dir', '--no-progress','-o', '-', '-f', '22/18', url, '--reject-title', stream_id),stdout=subprocess.PIPE)
_ffmpeg_process = subprocess.Popen(('ffmpeg','-re','-i', '-','-preset', 'ultrafast','-vcodec', 'copy', '-acodec', 'copy','-threads','1', '-f', 'flv',destination + "/" + stream_id), stdin=_youtube_process.stdout)
return
if len(sys.argv) < 4:
show_help()
else:
streamer()This python script does the following:
- Creates a _youtube_process sub- process for reading a movie by the youtube-dl library
- Creates the second _ffmpeg_process subprocess , which receives data from the first through pipe. This process already creates an RTMP stream and sends it to the server at the specified address.
Script testing
To run the script, you need to install python. You can download it here .
When testing, we used version 2.6.6. Most likely, any version is suitable, because the script is quite simple and its task is to transfer from one process to another.
Script run:
python streamer.py https://www.youtube.com/watch?v=9cQT4urTlXM stream1 rtmp://192.168.88.59:1935/liveAs you can see, three arguments are passed:
- YouTube address of the video.
www.youtube.com/watch?v=9cQT4urTlXM - The name of the stream with which the RTMP broadcast will take place.
stream1 - The address of the RTMP server.
rtmp: //192.168.88.59: 1935 / live
For testing, we will use Web Call Server . He is able to receive RTMP streams and distribute them via WebRTC. Here you can download and install WCS5 on your VPS or local Linux test server.
Testing scheme with Web Call Server:
Below we use one of the demo servers for the test:
rtmp://wcs5-eu.flashphoner.com:1935/liveThis is the RTMP address that you need to pass to the streamer.py script to quickly test the broadcast with our demo server.
The launch should look like this:
python streamer.py https://www.youtube.com/watch?v=9cQT4urTlXM stream1 rtmp://wcs5-eu.flashphoner.com:1935/liveIn the stdout console, we will see the following output:
# python streamer.py https://www.youtube.com/watch?v=9cQT4urTlXM stream1 rtmp://wcs5-eu.flashphoner.com:1935/live
ffmpeg version 3.2.3 Copyright (c) 2000-2017 the FFmpeg developers
built with gcc 4.4.7 (GCC) 20120313 (Red Hat 4.4.7-11)
configuration: --enable-shared --disable-logging --enable-gpl --enable-pthreads --enable-libx264 --enable-librtmp --disable-yasm
libavutil 55. 34.101 / 55. 34.101
libavcodec 57. 64.101 / 57. 64.101
libavformat 57. 56.101 / 57. 56.101
libavdevice 57. 1.100 / 57. 1.100
libavfilter 6. 65.100 / 6. 65.100
libswscale 4. 2.100 / 4. 2.100
libswresample 2. 3.100 / 2. 3.100
libpostproc 54. 1.100 / 54. 1.100
]# [youtube] 9cQT4urTlXM: Downloading webpage
[youtube] 9cQT4urTlXM: Downloading video info webpage
[youtube] 9cQT4urTlXM: Extracting video information
[youtube] 9cQT4urTlXM: Downloading MPD manifest
[download] Destination: -
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'pipe:':
Metadata:
major_brand : mp42
minor_version : 0
compatible_brands: isommp42
creation_time : 2016-08-23T12:21:06.000000Z
Duration: 00:29:59.99, start: 0.000000, bitrate: N/A
Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 288 kb/s, 30 fps, 30 tbr, 90k tbn, 60 tbc (default)
Metadata:
creation_time : 2016-08-23T12:21:06.000000Z
handler_name : ISO Media file produced by Google Inc.
Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 125 kb/s (default)
Metadata:
creation_time : 2016-08-23T12:21:06.000000Z
handler_name : ISO Media file produced by Google Inc.
Output #0, flv, to 'rtmp://192.168.88.59:1935/live/stream1':
Metadata:
major_brand : mp42
minor_version : 0
compatible_brands: isommp42
encoder : Lavf57.56.101
Stream #0:0(und): Video: h264 (Main) ([7][0][0][0] / 0x0007), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 288 kb/s, 30 fps, 30 tbr, 1k tbn, 90k tbc (default)
Metadata:
creation_time : 2016-08-23T12:21:06.000000Z
handler_name : ISO Media file produced by Google Inc.
Stream #0:1(und): Audio: aac (LC) ([10][0][0][0] / 0x000A), 44100 Hz, stereo, 125 kb/s (default)
Metadata:
creation_time : 2016-08-23T12:21:06.000000Z
handler_name : ISO Media file produced by Google Inc.
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Stream #0:1 -> #0:1 (copy)
frame= 383 fps= 30 q=-1.0 size= 654kB time=00:00:12.70 bitrate= 421.8kbits/s speed= 1xIf you quickly run through this log, you can understand that the following happens:
- The video page opens.
- Data on video formats is retrieved.
- Download mp4 video 1280x720, H.264 + AAC
- It starts ffmpeg, picks up downloaded data and streams over RTMP with a bitrate of 421 kbps. Such a meager bitrate is explained by the selected clip with a timer. A normal video will give an order of magnitude greater bitrate.
After the streaming process has started, we try to play the stream in the WebRTC player . The stream name is specified in the Stream field, and the server address in the Server field. Connection to the server occurs via the Websocket (wss) protocol, and the stream arrives at the player via WebRTC (UDP).
We specifically took this particular video on YouTube in order to be able to test the realtime stream, because our ultimate goal was to deliver the YouTube stream to all viewers at the same time, with a minimum delay and time spread. A clip with a millisecond timer is perfectly suited for such a test.
The test itself is very simple. We open two browser tabs (we simulate two viewers), play this stream with a timer according to our scheme, and take several screenshots to capture the difference in video arrival time. Next, we compare the milliseconds and see who received the video earlier, and who later and how much.
We get the following results:
Test 1
Test 2
Test 3
As you can see, each of the viewers sees the same video, with a spread of no more than 130 milliseconds.
Thus, the task of real-time broadcast of the video from YouTube to WebRTC is solved. Viewers received the stream almost simultaneously. The manager didn’t leave to drink coffee, the programmer read the habr, and the marketer successfully showed everyone the video clip.
Have a good streaming!
References
youtube-dl - a library for downloading videos from YouTube
ffmpeg - RTMP encoder
Web Call Server - a server that can split an RTMP stream by WebRTC
streamer.py - a script for integrating youtube-dl and ffmpeg with sending an RTMP stream