July 18, 2019 at 20:42Mad Converter GIF'ok to animated stickers for Telegram
Instead of a thousand words...
xZibit is also happy, because here GIFs are inserted into stickers to be inserted into GIFs for KDPV!
And now about the implementation details.
It all started with a discussion in the Telegram-developers' chat room about the upcoming feature:
![Bohdan Horbeshko, [04/04/19 20:21] Hmm, but the bot will probably only accept gifs, and then convert ... | Vitaly, [07/04/19 20:22] from the gif to Jason? I would have looked :))) | Bohdan Horbeshko, [04.07.19 20:22] And why not? There’s a PlayCanvas model editor in JSON. | Vitaly, [04/04/19 8:23 p.m.] and how is the gif? export pixel by pixel? :) | Bohdan Horbeshko, [04/04/19 20:24] Of course, the IT world has seen and will not tolerate such distortions.](https://habrastorage.org/webt/zu/s5/x1/zus5x1ydd89sejezp98ztbnygto.png)
The guy said - the guy did! The first prototype in Pillow and svgwrite, parsing GIFs into pixels and converting them into vector squares with previews in SVG, was written in one day off.
The fun began further ...
Until now, with the formats in Telegram, they kept tricking. Made support for GIF-animations - in fact, they are converted to MP4-video. Made support for stickers - they are uploaded to PNG, but converted to WebP. This time everything is more honest: that at the entrance, then at the exit.
For animated stickers, Telegram does not use GIFs, videos, or even some well-established vector graphics format such as SVG, or, God forbid Cthulhu! - Flash. It uses a newfangled format that has emerged from under the wing of Airbnb - Lottie. Until now, he had some fame among mobile developers, but thanks to Telegram, it may gain more popularity.
In essence, Lottie files are JSON-serialized Adobe After Effects projects that maximize all the features of this program. With display, alas,everything is not so rosy . Although there are many ready-made "official" implementations of the library for rendering Lottie, just for the platforms covered by Telegram: Android, iOS, Qt and Web - only part of the format’s capabilities is implemented in all of them. Telegram went even further and limited the list of supported features, and also “came up” with its own format, which differs from the usual Lottie in just the GZip package and parameter
And if everything is pretty good with the documentation for libraries for different platforms, then, alas, it was not possible to find at least some description of the format device - only a JSON-schemein the sources of lottie-web. I had to tinker along the way in existing animations in order to understand the general concepts of the format. There were also discrepancies between real files and the scheme: in particular, in layers of type 4 , according to the scheme, nested objects are stored in the property
The clarified nuances of the format:
The first problem directly follows from here: redundancy . Although default values for transformation parameters have recently been added to the JSON scheme, they are not implemented in libraries. So asking them explicitly is still necessary.
It would seem that this is not a problem at all? Even a simple GZip does a good job of compressing blatantly repetitive data, and 1 MB of raw JSON magically turns into a couple of tens of kilobytes, which quietly creep into the stated limit of 64 kB. There it was!
I’m downloading, which means a chubby animation that lottie-web is quietly displayed in Telegram - and here instead of a conditionally beautiful pixel art, the static blurry look at me is this:
What is it ?! But it turned out that there is clearly not specified on the decompressed data1 MB limit . A representative of the Telegram team quickly confirmed it and announced the upcoming increase in the limit to 2 MB.
Even if they solve these problems, stickers that go beyond 1 MB of uncompressed data and do not contain transformations will be inaccessible to users of older versions of Telegram. So it will probably have to comply with the restrictions in the future.
Pillow, along with OpenCV, can be called the industry standard for image processing in Python. Moreover, it is also well-sharpened for GIF features: it supports indexed colors and gives access to the palette. It supports the conversion of a pixel map into a NumPy array, which is important for productive processing. Even collects statistics on colors! But there were also disadvantages:
Therefore, I had to look for a replacement. The gif2numpy module acted as it . It is “sharpened” for GIF features and provides access to all the technical properties of both the image and individual frames, including GCE . Thus, he solves the problem of reading delays.
Transparency, as it turned out, gif2numpy does not support it at all: colors are immediately converted to three channels with bit depth in bytes, without taking into account bit depth and saving color indices. Fortunately, the module consists of one file, so it was not difficult to include it in the project and modify it by reserving color for transparency
The partial frame problem was not trivial to solve. gif2numpy is trying to imposesuch frames to the previous one, however, does not check the blending parameters, which is why the correct result also does not always come out. In order not to mess around with flags, the preliminary processing of images with the help of
Squares are good, but with such restrictions you need to show more imagination, otherwise even miniature GIFs will not “creep” into Telegram.
Something similar to RLE was used first : horizontally adjacent squares of the same color are combined into one rectangle.
Next up is the exploitation of Lottie features. Since each layer has an arbitrary start and end time, you can apply a technique that has long been used by video codecs, and partly in GIF itself: the squares that remain in the same place for several frames can be merged into one layer, during which display changes several others. Which is implemented so far only for pairs of adjacent layers.
The ideas that can be applied here are in bulk:
xZibit is also happy, because here GIFs are inserted into stickers to be inserted into GIFs for KDPV!
And now about the implementation details.
It all started with a discussion in the Telegram-developers' chat room about the upcoming feature:
The guy said - the guy did! The first prototype in Pillow and svgwrite, parsing GIFs into pixels and converting them into vector squares with previews in SVG, was written in one day off.
The fun began further ...
JSON is an open format, they said ...
Until now, with the formats in Telegram, they kept tricking. Made support for GIF-animations - in fact, they are converted to MP4-video. Made support for stickers - they are uploaded to PNG, but converted to WebP. This time everything is more honest: that at the entrance, then at the exit.
For animated stickers, Telegram does not use GIFs, videos, or even some well-established vector graphics format such as SVG, or, God forbid Cthulhu! - Flash. It uses a newfangled format that has emerged from under the wing of Airbnb - Lottie. Until now, he had some fame among mobile developers, but thanks to Telegram, it may gain more popularity.
In essence, Lottie files are JSON-serialized Adobe After Effects projects that maximize all the features of this program. With display, alas,everything is not so rosy . Although there are many ready-made "official" implementations of the library for rendering Lottie, just for the platforms covered by Telegram: Android, iOS, Qt and Web - only part of the format’s capabilities is implemented in all of them. Telegram went even further and limited the list of supported features, and also “came up” with its own format, which differs from the usual Lottie in just the GZip package and parameter
"tgs": 1. I think I know where Denis Popov is working now! :-) And if everything is pretty good with the documentation for libraries for different platforms, then, alas, it was not possible to find at least some description of the format device - only a JSON-schemein the sources of lottie-web. I had to tinker along the way in existing animations in order to understand the general concepts of the format. There were also discrepancies between real files and the scheme: in particular, in layers of type 4 , according to the scheme, nested objects are stored in the property
"it" - however, the key is called in real files "shapes", but "it"does not work. The clarified nuances of the format:
- A file consists of layers. Unlike GIF, here each layer can have an arbitrary start and end time for the display. Various transformations can be applied to the layer (more precisely, it is necessary ): scaling, rotation, changing transparency, etc. Layers can even be three-dimensional (prohibited for Telegram).
- A layer consists of “shapes”. They have many types , some can not be used in Telegram. In practice, for a layer to appear, it should include three shapes: a path (in finished animations, this is usually a type
"sh"— Bezier curves; the converter so far uses only type"rc"— rectangles), fill (type"fl"), and transformation (type"tr"). - You can even include raster elements, create text layers, and establish relationships between layer and shape parameters through expressions. All this goodies is also prohibited on Telegram.
The first problem directly follows from here: redundancy . Although default values for transformation parameters have recently been added to the JSON scheme, they are not implemented in libraries. So asking them explicitly is still necessary.
It would seem that this is not a problem at all? Even a simple GZip does a good job of compressing blatantly repetitive data, and 1 MB of raw JSON magically turns into a couple of tens of kilobytes, which quietly creep into the stated limit of 64 kB. There it was!
I’m downloading, which means a chubby animation that lottie-web is quietly displayed in Telegram - and here instead of a conditionally beautiful pixel art, the static blurry look at me is this:
What is it ?! But it turned out that there is clearly not specified on the decompressed data1 MB limit . A representative of the Telegram team quickly confirmed it and announced the upcoming increase in the limit to 2 MB.
Even if they solve these problems, stickers that go beyond 1 MB of uncompressed data and do not contain transformations will be inaccessible to users of older versions of Telegram. So it will probably have to comply with the restrictions in the future.
Transparency is important.
Pillow, along with OpenCV, can be called the industry standard for image processing in Python. Moreover, it is also well-sharpened for GIF features: it supports indexed colors and gives access to the palette. It supports the conversion of a pixel map into a NumPy array, which is important for productive processing. Even collects statistics on colors! But there were also disadvantages:
- There was no documented way to get the transparent color index. I had to assume as a temporary solution that transparent color is the most common, but in real GIFs this is not always the case.
- The same thing with a delay between frames: Pillow gives only the frames themselves as a sequence of images, without information about the delays.
- Sometimes partial frames are incorrectly superimposed.
Therefore, I had to look for a replacement. The gif2numpy module acted as it . It is “sharpened” for GIF features and provides access to all the technical properties of both the image and individual frames, including GCE . Thus, he solves the problem of reading delays.
Transparency, as it turned out, gif2numpy does not support it at all: colors are immediately converted to three channels with bit depth in bytes, without taking into account bit depth and saving color indices. Fortunately, the module consists of one file, so it was not difficult to include it in the project and modify it by reserving color for transparency
#FE00FE. The partial frame problem was not trivial to solve. gif2numpy is trying to imposesuch frames to the previous one, however, does not check the blending parameters, which is why the correct result also does not always come out. In order not to mess around with flags, the preliminary processing of images with the help of
gifsiclethe key --unoptimize is added - it converts partial frames to full ones. And at the same time it leads them to use the global palette, which eliminated the need to separately process the transparent color when using your own frame palette.Squeeze me harder
Squares are good, but with such restrictions you need to show more imagination, otherwise even miniature GIFs will not “creep” into Telegram.
Something similar to RLE was used first : horizontally adjacent squares of the same color are combined into one rectangle.
Next up is the exploitation of Lottie features. Since each layer has an arbitrary start and end time, you can apply a technique that has long been used by video codecs, and partly in GIF itself: the squares that remain in the same place for several frames can be merged into one layer, during which display changes several others. Which is implemented so far only for pairs of adjacent layers.
Development plans
The ideas that can be applied here are in bulk:
- Recognize single-color areas of any size. You can break them into a set of rectangles, for which there is a good algorithm . It is also advisable to convert them into a contour, but this is overshadowed by the need to indicate all points of the Bezier curves in Lottie - rectangles in some cases may be more advantageous.
- Recognize movement. The technique, again, has long been used in video codecs. If the same contour does not change shape from frame to frame, but only the coordinates - instead of duplicating it on several layers, place it on one layer with transformation.
- Recognize the “covering” of one area with another. Example:
...... .O..O. ...... .OOOO. ......
Pixels of a different color are superimposed on a rectangle of one color. Instead of breaking this rectangle into a bunch of small ones, or transforming it into a contour of complex shape - you can simply overlay them on top of the whole rectangle. - Vectorization of curves and ellipses, recognition of gradients. This will spoil the pixel charm, but it will improve the compressibility of some GIFs by orders of magnitude. Gradients are even in the antediluvian "Koloboks", I guarantee it!
- Lossy compression. First of all, the elimination of dithering, and even in excessively smooth pictures will not hinder the moderation of color. The aforementioned gifsicle will probably handle this.
References
- Sources . Scary in places.
- The channel on which I upload packs of successfully converted GIFs.