Progressive Streaming technology, or how to watch 4k video over the network, without friezes
Today, one cannot surprise anyone with the speed of the Internet of 100 Mbit / s, but there is a problem how to use it. All basic operations do not fully load the network. At the same time, the heavy 4k-8k audio and video formats that you want to watch online are more popular. And looking at the high speed of the Internet, a logical question arises - why not? How to master the full speed provided by the provider? Both on the client side and on the service side. Consider all these issues in the article.
I made a really good software product and I want to tell in more detail how everything is arranged there. There are many new technologies. Perhaps because no one has invented anything in this industry for a long time, and the time has come. This article is not only for developers, but also for ordinary people. I tried to explain everything as simple as possible.
1. Let's start with the basic foundations of today's data transfer technologies.
There is a big gap in the knowledge of many people on high-speed data transfer algorithms - more than 10 Mbit / s ... Let's fill in these gaps:
Space number 1. The problem is that the totality of radio transmission technologies is now focused on continuous data transmission, as the best scenario.
Everything that is transmitted via wi-fi \ 3G \ 4G and even by wire will be transmitted most quickly if the transmission is constant, even at a lower speed than the maximum. This will be much faster than transmitting intermittently, but at maximum speed.
- to resume communication, more service information is added to the traffic;
- when resuming communication, the server can lower the client rating and give data at a lower speed (possibly due to the appearance of other clients), or not at all. Even in a home wi-fi network, the router can lower the rating, for example, due to a wi-fi vacuum cleaner. Those. this applies to all networks where there are more than 2 clients. In general, almost to everyone :-)
Space # 2. When copying data, any program uses a kind of “bucket” that draws data from the source and transfers it to the destination. So this bucket, depending on the data transfer rate, must be of different sizes . It’s easy to understand: if you scoop a bucket of water with a teaspoon, it will take much longer than scooping a mug.
Space # 3. To increase the speed of downloading data over the network, RAM should be used. Any slightest milliseconds of delay when recording incoming data are then poured into seconds, minutes and hours. To prevent this from happening, you need to write data first to RAM, and then to a more voluminous “bucket” on a permanent medium (hard disk). Otherwise, the data transfer speed will dramatically fall.
This is enough to just copy the file.
2. These gaps are just the tip of the iceberg. With normal file transfer this is enough, but what if our file is multimedia, and we must start playing it online. A modern multimedia file cannot completely fit into RAM, so you must consider saving it to disk.
The best strategy is built from the above about spaces:
- data loading should occur regardless of playback, continuously;
- to ensure rewinding, it is necessary to create another stream of data loading;
- to overcome the frieze at startup, due to the receipt of technical data (codecs, etc.), preloading is necessary. Empirically, I found the formula: the size of the entire file is * 0.002 or 0.2%.
This is enough to play Flac files.
3. It turns out for online streaming, which will rationally and fully use our communication channel, you need RAM and disk space. Without this, the channel will not be fully utilized.
And here begins the branching of data loading algorithms! Without technical details, I will give them in an approximate volume:
- in order for data to not be downloaded twice, it is necessary to make a collision of streams. Those. if we started the track and quickly rewound to the middle, then the first stream, reaching the middle, should interrupt its work.
- so that data is not downloaded twice, a stream should not be created if the data is already loaded.
- For normal playback, a complex logic of interaction between the player stream and the streams that download data is necessary.
I did all this and a little more. It turned out absolutely fabulous. Data flies and playback is not interrupted. Flac files were downloaded completely at 1-3 seconds of playback. And that was enough to play Full HD video.
4. The problem is that our media files are very different in size. And with 4k Blu-ray movies, which weigh about 80-120 GB, nothing worked. The player created 15 threads at the start, and they all shared the speed, which, of course, was not enough for the main thread, which the player was waiting for. The data was loaded at maximum speed, yes ... they downloaded the full channel, but 4k video hung and played slower than the slideshow. It turns out a lot of streams is harmful for 4k video, but where is the border after which the benefits in Full HD turned into harm in 4k ?!
As a result, everything rested on the speed of the channel. In order to optimize the work of streams, you need to know two things:
a) The necessary speed for playback, which is calculated: (file size / duration in seconds) * 8.
b) The speed of downloading data from the main stream that the player is currently working with.
If we manage the loading of data, then we definitely have the opportunity to measure the download speed. Now each thread knew its speed (in Mbps) and this does not impose additional costs on performance. Be sure to identify which stream the player is currently working on. It all starts in the usual, multi-threaded mode. But as soon as the player calculates the duration of the media file in seconds and transmits this data, then all streams get a fixed, necessary speed for playback. Immediately after this, all flows (except the main one) compare the necessary speed with the speed of the main stream and if its speed is lower, then they pause. Further, the main thread slowly builds up its speed, and as soon as it exceeds the necessary speed twice (as I did), it pauses all secondary flows from pause.
According to observations, then, after some time, the speed of the main stream drops again and all the secondary streams go into pause again, and so until the data is downloaded or playback is completed. Such a flexible strategy, when the data loading is parallelized and, if necessary, compressed into one stream, fully downloads the communication channel and at the same time ensures the fastest possible download of the data needed for data reproduction. This strategy works equally well on both 10 MB and 100 GB files. To play over the network without loss, it is impossible to come up with anything better. If you have any suggestions, I will be glad to discuss them in the comments.
Suitable for playing media files of any volume 4k-8k.
Progressive streaming is the fastest type of streaming when transmitting data in its original quality. To increase the data transfer rate, it uses RAM and multi-threading. Data is loaded asynchronously to the playback stream, but with active interaction with it. During playback, the speed of data acquisition is measured, and the number of active streams is adapted to the available data channel.Right now in the world there is a big tendency to increase the number of high-resolution videos, and the popularity of devices for playing it is growing.
Bitrate 4k video is large, but not sky-high. The most efficient VP9 codec today compresses 4k video at 15 Mbps. with audio comes out about 15.5. A 100 GB Blu-ray movie has a bitrate of about 60 Mb / s ... Anyone in the world who wants to watch 4k videos has these Internet speeds . This means 4k videos can be watched online now!
Despite the simplicity of the algorithm described above, the implementation looks very complicated. Streaming technologies with deterioration in the quality of video and audio are forced to spoil content due to the lack of the implementation of the correct data transfer algorithm. I assume that many people and companies have (above) knowledge gaps and, of course, difficulties in implementing this algorithm. Therefore, he wrote an article to simplify the understanding of this method of streaming.
It will be a little more difficult to make this algorithm using copyright, but in general it is also possible. When saving to disk, and reading, it is necessary to encrypt data. It looks like an enemy action against the user, but what to do. Some companies are doing this.
Now let's look at the disadvantages of conventional streaming, compared to progressive:
- it is impossible to preload the next file \ to predict and prepare the next action;
- even with a temporary disconnection of the Internet / interruptions in speed, a noticeable frieze of sound / image will appear;
- the communication channel is not fully used, and half the playback time is idle, at a time when there is still a lot of data to load;
- the inability to play 4k-8k video, even via wi-fi without friezes. Constant gusts of communication and speed jumps to the maximum can not stand it, even home wi-fi throughout the film is 2 hours or more;
- this imposes an even greater burden when playing 4k content, since the player needs to keep in RAM from 200-300 MB of video data (at the required speed of 60 Mbps). When playing through a progressive streaming algorithm, this is no longer necessary, since playback is from a disk, not via a network.
According to these points, it is clear what to stream, as before, is no longer possible. Of course, you can increase the speed \ memory \ cache ten times, but why, if today's speeds are already enough and the problem lies in the algorithms. Bad algorithms sooner or later come to a standstill. Today, progressive streaming is the only technology that allows you to stably and comfortably watch 4k 100 Gigabyte movies online.
As an example of how the technology works, I wrote a multimedia application - the Media Library player . It supports all formats. To demonstrate the capabilities you need to run it. Open the demo site and go to the "Films (TOP) / 4k movies" directory tree. There you can select any movie and watch it. To support all formats, you should switch the player module from ExoPlayer to VlcPlayer.
In this scenario, all of the above data acquisition algorithms will be used. My application supports the following protocols: nmdc \ http \ ftp \ samba, as well as the Mega.nz cloud.
4k video files weighing 100 GB are played with minimal delays and downloaded to memory at the highest possible speed. You can preload more data by simply pausing, but this is usually not required. Playback is always stable, if not stable, the channel speed is near the necessary speed for playback.
Due to the above-described gaps, the policies of the largest companies are not developing in the right way, which slows down progress and often prevents the use of available devices and communication channels in full.
The application provides for saving the cache to external media, but due to the strange complication of access to external media from Google, it is currently supported mainly by Samsung devices and Android version 5.0+. In the future, full support for external usb media will appear.
In general, the algorithm significantly improves the quality and speed of downloading data when streaming, both small files 30-50 MB and large 50-120 GB.
In perspective it can be used:
- for high-quality playback of 4k-8k media content on mobile phones, televisions and other devices;
- in virtual reality for displaying volumetric scenes in high resolution;
- in combination with torrent-like peer-to-peer protocols;
- for high-quality music playback in high resolution on any device. Experimental testing showed that 2 Mbps of an unstable communication channel is enough for this.