How Discord resizes 150 million images every day with Go and C ++

Although Discord is a voice and text chat application, more than one hundred million images pass through it every day. Of course, we would like the task to be simple: just redirect pictures to your friends on all channels. But in reality, the delivery of these images creates quite big technical problems. A direct link to the pictures will give the host with the picture the IP addresses of users, and large images consume a lot of traffic. To avoid these problems, an intermediate service is required that will receive images for users and resize them to save traffic.

Meet Image Proxy

To do this, we created the Python service and creatively named it Image Proxy . It downloads images from remote URLs, and then performs the resource-intensive task of resizing with the pillow-simd package . This package works surprisingly fast, using wherever possible to speed up the resizing of x86 SSE instructions . Image Proxy will receive an HTTP request containing a URL to upload, resize, and finally display the final image.

On top of it, we set up a caching layer that saves resized images in memory and tries, if possible, to issue it directly from memory. The HAProxy layer redirects requests based on the URL hash to the Nginx caching layer. Cache executesCombining queries to minimize the number of transformations required to resize images. The combination of cache and proxy allowed us to scale our resizing service to millions of users.



As Discord grew, Image Proxy began to show signs of overload. The biggest problem was that the load was distributed unevenly, which caused bandwidth to suffer. Requests were executed at completely different times, up to several seconds. We could solve this problem in the existing Image Proxy, but at that time we were just experimenting with the additional use of Go, but here it seemed to be a great place to use Go.

So there was Media Proxy

Rewriting a service that is already running has become a difficult decision . Fortunately, Image Proxy is relatively simple and it was easy to compare the results from it and the new alternative. In addition to faster query execution, the new service also has some new features, including the ability to extract the first frames of .mp4 and .webm videos - therefore, let's call it Media Proxy.

We started by measuring performanceexisting packages for resizing images on Go and quickly became discouraged. Although Go is generally a faster language than Python, none of the packages found reliably outperformed Pillow-simd. The main part of Image Proxy's work was transcoding and resizing images, so that would definitely become a bottleneck in the performance of Media Proxy. The Go language can be a little faster when processing HTTP, but if it is not able to quickly resize pictures, then the additional gain in speed is leveled by the additional time for resizing.

We decided to double the bet and build our own image resizing library on Go. Some promising results showed one Go package based on OpenCV, but he did not support all the functions we needed. We created our Go resizer called Lilliput with our own Cgo wrapper on top of OpenCV. When creating, we carefully watched so as not to generate excess garbage in Go. Lilliput wrapper does almost everything we need, although it took a little fork of OpenCV for our needs. In particular, we wanted to be able to check the headers before deciding whether to decompress images: this way you can instantly refuse resizing too large images.

Lilliput uses existing and proven C libraries (for example, libjpeg-turbo for JPEG, libpng for PNG) and vectorized OpenCV code for quick resizing for compression and decompression. We added fasthttpto meet our requirements for parallel HTTP client and server. As a result, this combination allowed us to launch a service that steadily outperformed Image Proxy in synthetic benchmarks. In addition, lilliput worked no worse or better than pillow-simd in the tasks we needed.



The first code was not without problems. Initially, Media Proxy leaked 16 bytes per request. This is small enough to immediately notice, especially when testing on a small volume of requests. To solve the problem, Media Proxy installed large static pixel buffers for resizing purposes. It uses two such buffers per CPU, so on a 32-core machine it immediately takes 32 gigabytes of memory. During testing, restarting Media Proxy took several hours, since it uses absolutely all the memory. This is a long enough time to make it difficult to clarify the situation: either we really have a memory leak, or simply exceeding the limit during operation.

In the end, we decided that there should still be some kind of memory leak. We were not sure whether this leak was in Go or C ++, and studying the code did not give an answer. Fortunately, Xcode comes with an excellent memory profiler - the  Leaks tool in the Instruments menu . This tool showed the size of the leak and the approximate place where it occurs. Such a hint was enough for a more thorough study to determine the source and correct the leak.

In Media Proxy, we ran into another bug that caused us to interrupt. Sometimes he gave out strangely spoiled pictures, where one half remained normal and the other half was “buggy”. We suspected that we were partially encoding an image somewhere or somehow incorrectly calling OpenCV. The bug showed up infrequently and was difficult to diagnose.

To advance the investigation, we developed a high-performance query simulator that returned URLs with a link to an HTTP server in the simulator, so that this simulator worked both as a requesting client and a host server. He randomly inserted delays in the responses to provoke such image corruption in Media Proxy. Having reliably reproduced the problem, we managed to isolate the components of Media Proxy and find the race state in the output buffer containing the resized image. One image was written to this buffer, and then another, before the first returned to the system. Glitchy pictures in reality were two JPEGs, recorded one on top of the other.


Real buggy JPEG generated by Media Proxy

Another way to look for bugs in complex systems is fuzzing, when random input data is generated and sent to the system. In this case, the system may exhibit strange behavior or collapse. Since our system must be resistant to any input data, we decided to apply this important technique in the testing process. AFL is an exceptionally good fuzzer , so we selected it and set it on Lilliput, which allowed us to identify several failures due to uninitialized variables.

After fixing these bugs, our confidence grew enough to roll Media Proxy into production - and we were happy to see that our efforts were worth it. Media Proxy required60% fewer server instances to handle the same number of requests as Image Proxy, executing these requests in much smaller time scatter. Profiling showed that more than 90% of the CPU time in a new service is spent on decompression, resizing and compression. These libraries are already significantly optimized, that is, it would not be easy to achieve additional growth. In addition, the service almost did not generate garbage in the process.

Now Media Proxy performs image resizing with a median time of 25 ms, and the response delay is median 85 ms. It resizes over 150 million pictures every day. Media Proxy runs on the n1-standard-16 auto-scalable GCE host group , with a peak of 12 instances on a typical day.

Download media in Media Proxy

After the service worked successfully on static images, we wanted to connect support for resizing animated GIFs to it, and OpenCV will not do this work for us. We decided to add another Cgo wrapper on top of giflib to Lilliput, so that Lilliput could resize the animated GIF as a whole, as well as produce the first frame in PNG format.

GIF resizing was not so simple, since the GIF standard provides for the use of 256 color palettes in each frame, and the resizing module works in RGB space. We decided to save the palette of each frame, rather than calculate new palettes. To convert RGB back to palette indices, we provided Lilliput with a simple lookup table that took some RGB bits and used the result as a key in the palette index table. This worked well and kept the original colors, although this approach does not mean that Lilliput is able to create GIFs only from source files of this format.

We also patched giflibto make it easier to decode only one frame at a time. This allowed you to decode the frame, resize it, then encode and compress it before moving on to the next one. This reduces the memory consumption of the GIF resizing module. This complicates Lilliput a bit because it needs to save some GIF states frame by frame, but the more predictable memory usage in Media Proxy seems like a clear advantage.

The giflib wrapper in Lilliput fixes some of the problems that were in the Image Proxy GIF resizer, as giflib gives full control over the resizing process. A considerable number of Nitro usersDownload GIF animated avatars that are buggy or have transparency errors after resizing in Image Proxy, but work fine after processing Media Proxy. In general, as it turned out, resizing programs have problems with some aspects of the GIF format, so they give out visual glitches for frames with transparency or partial frames. Creating your own wrapper allowed us to solve the problems that we encountered.

Finally, Cgo wrapper on libavcodec was added to Lilliputso that he was able to stop the video and receive the first frame of the MP4 and WEBM clips. This functionality will allow Media Proxy to generate previews for user-posted video files, so that other people can decide whether to launch the video based on this preview. The extraction of the first frame remained one of the last factors that stopped us from adding a built-in video player to the client for published files and links.

More Open Source

Now that we are satisfied with the work of Media Proxy, we publish Lilliput under the MIT license. We hope the package is useful to those who need a productive image resizing service, and this article will encourage others to create new Go packages.