Krita: levels of detail or how to draw with a brush with a diameter of 1k on a canvas of 10k pixels



    A little less than a week is left until the end of the campaign on Kickstarter , but the Krita graphic editor has already raised funds for two main tasks of this year: animation and drawing huge images. And if animation is becoming less or less clear, then with large images, questions arise. How to calculate and display an image of 100 million pixels? How can I ensure that a 1 million pixel brush is drawn 500 times per second without delay? I will try to answer these questions in this article.

    Where do drawing delays come from?


    First you need to understand how drawing is done in a modern graphics editor. Any brush is an image (“brushstroke” or “dab”), which is either loaded directly by the user or generated parametrically. When the user makes a brush stroke, this image is sequentially superimposed on the canvas with a certain step (spacing) (usually 10-20% of the size of the brush). The resulting image falls into the rendering pipeline, where it merges with all layers and is transferred to the user interface, where it is already drawn on the monitor screen. From the outside it looks simple, but in fact, even for a simple brush, the stroke made by the user during the pipeline will be processed about 7 (!) Times.
    Detailed conveyor structure
    When painting with a brush, each smear goes through the following stages:
    1. The brush mask is filled with color and the smear itself is formed.
    2. A brushstroke is drawn on top of a temporary canvas that allows strokes to not overlap each other (indirect drawing mode or “Wash Mode”)
    3. A temporary canvas is drawn over the contents of the layer.
    4. All layers merge into one image.
    5. The image is copied to the user interface.
    6. The interface converts the color space of the image to match the color space of the monitor
    7. The final image is loaded into the openGL texture and rendered on the screen.



    Example


    So, each “smear” undergoes a minimum of 7 transformations. Is it a lot or a little? Let's look at a simple example. Imagine that we paint with a brush 300x300 pixels (300 * 300 * 4 = 312 KB) on a canvas of A4 format 300dpi (3508x2480 pixels).

    The speed with which the artist can comfortably move the tablet stylus (taking into account the zoom) is about 18 pixels per millisecond. Then (with a brush step of 10%), the average speed with which we must manage to draw a brush on the canvas will be 600 “strokes” per second.

    Drawing speed distribution
    Graph of the speed of calculation of “strokes” depending on the speed of the mouse.
    Brush: 300 pixels
    Image: A4 300dpi (3508x2480 pixels), 25% zoom
    CPU: Core i7 4700MQ



    Given the size of the brush, it turns out that at each stage of the pipeline, the editor needs to process about 187 MB per second, which is more than 1.2 GB / s (!) For the entire pipeline. And this is not even considering the fact that at almost all stages the pipeline does not just convert one area of ​​300x300 size, but takes two images, calculates their composition (at least one division by pixel operation) and writes the result back to memory. It turns out that even with such relatively small sizes of the brush and image, we are quite close to the theoretical limits of the speed of RAM (10-20GB / s).

    “WTF ?!” the attentive reader will ask. “How then does it work at all ?!” Of course, at each stage of the pipeline, many optimizations are applied. The whole area is divided into several threads, which, moreover, are executed in parallel, they also use SSE / AVX vector instructions, which allow processing up to 8 pixels at a time. In addition, in some special cases (for example, one of the pixels is completely transparent or opaque), the composition degenerates into a simple copy of bytes.

    However, all these measures will help very weakly if we start talking about brushes with a size of 1000 pixels or more. After all, with an increase in the size of the brush by 3 times, the amount of processed data will increase not by 3, but by 9 times! Handle 12 GB per second? Well, I do not! So what to do?

    MIP Texturing and Levels of Detail




    In three-dimensional graphics, there is a well-known technique that allows you to increase the speed and quality of texturing objects that are far from the camera. The fact is that when an object moves away from the observer, it becomes smaller in size and, accordingly, its texture should also scale. To speed up this process, MIP texturing technology was invented. Its meaning is that along with the texture itself, a lot of its reduced copies are stored: in 2, 4, 8, 16, etc. time. And when the GPU needs to draw a smaller version of the texture, it no longer scales the original, but simply takes a pre-prepared copy and works with it. This not only increases the speed of drawing objects, but also greatly improves their quality,

    Levels of Detail in Crete


    Here it is worth considering one observation that if a user decides to draw on an image 10k pixels wide, then most of the time he will use a scale of 20-15%. Otherwise, this image will not fit technically on the screen of its Full HD monitor, with a width of just under 2k. We will use this fact!

    At the beginning of this year, we made a prototype of the system of deferred image rendering for Krita. When a user paints a canvas with a brush, Crete is in no hurry to calculate all his actions. Instead, she takes a small copy of the image and draws all the strokes on it. Since the reduced copy is 2-4 times smaller than the original, drawing on it is 4-16 times faster, and therefore there are no delays that distract the artist from his creative process. And since the artist cannot draw all 100% of the time, Krita will still have plenty of time when it is possible in the background, without rushing, to calculate the strokes on the original image.

    A video showing a 1k brush on an 8k image. Notice how, a few seconds after the completion of the stroke, the second wave of updates arrives.



    As a result, we solved the main problem. Now we are not trying to process gigabytes of data in real time. In real time, we need to calculate only a preview that will allow the user to work comfortably. And you can do the original later.

    Preliminary findings


    At the moment, we have a ready-made prototype of a system of work with levels of detail. Only one brush engine works in it, and even with not all parameters. However, it already allows us to draw some conclusions:

    1. We have solved the main problem: the user sees that he is drawing
    2. Quality preview worthy. Problems arise only at the boundaries of update areas, where the interpolation algorithm used in openGl shaders changes. Need to decide.
    3. As a bonus, openGL 3.0 and higher allows you to load / read information directly from a certain level of detail (GLSL 1.3: textureLod (). That is, we do not need to keep copies of all textures, just update a certain level, tell the shader about it, and it reads directly
    4. The main drawback of the approach is that this system has led to a serious complication of the Crete task scheduler. It needs to solve a lot of problems. For example, two copies of an image (original and small copy) need to be regularly synchronized. And this is compounded by the fact that not all actions in Crete can be performed on a reduced copy. Many actions (so-called legacy actions) require complete control over the image. They work like barriers: before they are launched, all “bifurcated” actions must be completed, and after their completion, copies of the image must be synchronized.
    5. Accordingly, if a user launches a legacy action, then he will have to wait until all background processing is completed. Which may, of course, not entirely convenient. The only solution to this problem can only be reducing the number of legacy actions ...


    Our Kickstarter project has already reached its minimum goal, so the next few months we will spend on the implementation of a full-fledged system of work with levels of detail. And very soon anyone will be able to test drawing with huge brushes in Crete!

    References


    Project page on Kickstarter: link
    Group of Russian-speaking users in VK: http://vk.com/ilovefreeart
    Official website: krita.org

    Also popular now: