Adventures in a separate thread. Yandex Report

    How to work with images on the client, while maintaining a smooth UI? Interface developer Pavel Smirnov spoke about this on the basis of the experience of developing a search for photographs on the Market. From the report you can learn how to properly use Web Workers and OffscreenCanvas. - For this half hour we’ll talk about adventures. I will tell you about my adventure and I really hope that my report will inspire you and you will take and do the same at home. First, I wanted to talk about some new or not very new technologies that our browsers give us that allow us to do cool things. But it seems to me that this would not be very fun, because everyone can go to MDN and read something. Therefore, I’ll tell the story of one feature that I did with the Market team.







    Let's introduce myself again first. My name is Pasha, I’m an interface developer in the Market team.



    I mainly deal with mobile interfaces - map search, offer card. I also rewrite the code from the old stack to the new one, and then from the new to an even newer stack. And I try to make my interfaces good. Here it is worth saying what a good interface is.

    Good interfaces have different characteristics. Firstly, it is convenient; secondly, it is beautiful; thirdly, it is affordable. But one of the characteristics that I want to talk about today is speed. And speed often manifests itself in the smoothness of his work. Even small friezes can greatly change the user experience of our interfaces.



    Let's move on to the plan for my conversation today. First, we’ll talk about the task that I did: finding a picture on the Market. Next, I’ll tell you what problems I had to solve in order to implement this functionality. Here we recall a little how your script works in the browser, and look at the technologies that helped me. Small spoiler: these are Web Workers and OffscreenCanvas.

    Let's get back to the task. A few months ago, Luba, our product manager, approached me. Lyuba deals with the problems of choosing a product on the Market. Now we have several options for finding goods. One of them is to enter something into the search bar.



    For example, "red iPhone X buy in Samara." And we will find something. Or we can use the catalog tree. In this catalog we have categories and subcategories.

    But what if I want to find something on the Market, not knowing what it's called, but either I have a picture of this thing, or I see it at someone's party?



    I will tell a real case. I once went with my friends to a cafe. We ordered lemonade there, you know, in such a jug, and this jug had such a strange thing. I even kept a photo. It was intended so that when you pour lemonade into a glass, ice does not get into it. We thought it was a cool thing, but we had different opinions about what this thing is called and, in general, what it was intended for. Therefore, we found it on Yandex.Pictures.

    But I thought - it would be cool if I could not only search for this thing, but also buy it right away or at least find out the price, read reviews, specifications, etc. At this point, our dreams coincided with Any, and we decided make such functionality on the Market.

    What is this functionality like? It allows the user to upload a photo or picture, you can even immediately take a photo and send it to the Market. We analyze this photo using Yandex search technologies, find the product on it and show the user the results with these products. It seems to sound simple, but if it were so simple, I would not make my report. So that you can see what kind of feature this is, let me show it.

    Watch the first demo

    I will show on production. Let's first I upload the very thing we were looking for and see what happens.

    We found some goods and specifically this thing. This thing is called a strainer. To find something else, I photographed one book at a colleague’s desk yesterday, let's look for it. Here is such a book, perhaps someone read it. It is called "Perfect Code." He also finds it somehow, and for some reason with a limit of 18+. This is probably a little strange.

    Let's get back to our report. What problems have I encountered? The first problem is that the user starts downloading anything, including huge pictures. For example, my phone takes pictures three to four megabytes in size, which is quite a lot. Sending such photos to the backend is inefficient. It takes a long time, it takes a long time to analyze them, so you need to do something about it. But here everything is simple - we will crop, compress, resize this photo on the client.



    How will we do this? We have a file. And we will somehow read this file. We will read using the FileReader API. I will briefly tell you what it is.



    This is such a browser API that allows us to read the downloaded file and do something with it. You can read in different ways, we will look at it now. Here are its features, and we have some kind of object that returned to us from input by the change event. Let's try to read it.



    The code will look like this. There is nothing complicated here yet. We have a Reader object created from the FileReader constructor, on which we attach the developer of the load event. Next we will read this file as DataURL. DataURL - a string that represents the contents of the file encoded through Base64. Like we read, we have to cut it somehow. First, let's load it all into a picture. We have a tag or img element, and we load it right there.



    The code will look something like this. We create an img element, by the load Reader event we load our line into the src attribute and we will do everything further when our line is finished loading into img.

    We will do what we wanted to - crop the image. We will compress it, and here such a thing as Canvas will help us, a very powerful tool. It allows you to do a lot. But here we just draw our picture on this Canvas, and if the picture sizes exceed the maximum allowable, we will fit them a little. We can also pick up this picture with Canvas of the desired compression ratio.



    Like that. Another small disclaimer: the code here is greatly simplified, I do not specify everything. We have error handling and other things, but so that everything fits on the slide and is clear and understandable in the report, I omitted some details.

    We have picture sizes, we just look at them. There are some constants allowed to us. If the size of the pictures exceeds our constants, we just trim them and set our Canvas to these sizes.

    Next we will draw our picture on this Canvas.



    Take the 2d context, we need a 2d image, and try to draw using the drawImage method. DrawImage is an interesting method that accepts, if I'm not mistaken, nine parameters. But they are not all mandatory, we will use only five. We take Image and those two zeros, this is offset or indentation of the picture. We need the top left point. Draw with the dimensions we need.

    Further, from this Canvas, we will take our DataURL encoded Base64 string in exactly the same way and turn it into blob - a special object that is convenient for us to send to the server. It seems to be all. Everything works. The picture is cropped, the picture is sent, the picture is recognized.

    But then I began to notice something. When I tested this solution, when I uploaded a picture, especially on weak devices, my interface slowed down a bit. Either the button was not pressed, then the element did not scroll so. Did you get the feeling that your code works in 99% of cases and works well, but sometimes it just doesn't work? And you can give it for testing, and probably no one will notice. And users, probably, will not notice, especially on weak devices.

    This has never happened to me, and I decided to fix it. This turned out to be a problem. If the image is large, then during the manipulations with cropping, compression, it took us some time, and in this small, small time, our interface was unresponsive.

    First, I figured out why this happens. Here it’s worth a little bit to remember how JavaScript works in the browser. I will not go into details, this is a topic for a big report. Just remember some points.



    We have JavaScript running in a single thread, let's call it main. And we have such a thing in the browser as an event loop. Here we immediately say that this is a model. In some browsers, the event loop is organized differently, but as the name implies, in general it is a loop. It processes certain tasks in the queue in order.

    An unpleasant moment: until he processes one task, he will not proceed to the next. I will show the demo that I saw, she shows it. She is a classic.

    Watch the second demo

    I have a GIF image and CSS animation done in different ways: one using translatex, the other using position: relative left, the third using JavaScript, namely requestAnimationFrame. This is where the hedgehog is spinning. What will i do?

    I will block the main thread for five seconds. You know, usually tough guys calculate the nth Fibonacci number, but I wrote an endless loop with a break in five seconds.

    What will happen? You immediately noticed that the hedgehog stopped spinning, and the lower cat, which is animated using translatex, also stopped riding. But let's see the same demo in another browser, for example Safari. The GIF cat stopped running.

    Why am I showing all this? Firstly, browsers are different, you have to consider this. Secondly, when our flow is blocked by something, some things will stop working. For example - JavaScript animation. Or let’s show that the text will no longer stand out for us, the buttons will no longer be pressed.

    This is a very abstract example. Let's not block the flow for five seconds, but take our task, upload a photo, crop it, squeeze it and draw it here. We will not send it anywhere, it will not be very revealing.

    Watch the third demo

    I have a powerful MacBook here, and to make everything look more convincing, we will slow down the processor by six times. This allows you to do DevTools. Upload our photo. The Perfect Code will help us again. As we see, the same thing happens as when blocking the main thread.

    Let us then return to our task and think about how we will deal with this.



    By the way, if you look at the profiler, we will see this. In the red frame is our microtask, which blocks the main thread. We see that he blocks it for almost five seconds. It is on a rather powerful computer, and on weaker devices it will be even more noticeable.

    Let's move on to the solution. I’ll say right away what I used and what I did, and then we will analyze all these things. Firstly, I used Web Workers. They allow us to put some tasks in a separate thread. And secondly, in the context of Web Workers, the DOM is not available to us. To deal with this situation, we will use other tools. Image will not be available to us, the classic Canvas is available, and therefore we use Canvas and some other tricks.



    Let's quickly remember what Workers are, what they are for. They allow you to run JavaScript in a separate thread, not mainly. And the Workers stream does not interfere with the rendering flow of the main interface. Therefore, we can perform some complex computational tasks without slowing down our interface.

    We have a tool that allows you to transfer something to Workers and return something from Workers. Let's see an example.



    So we create our Worker using the constructor. There you need to transfer the path to the file. We can even pass blob. And we have a Message event handler. In this case, it will simply display something on the screen. Then we can send some data to our Worker.



    What is the support? Everything is fine here. Workers is a well-known tool, not new, but many of my friends think that they are not always supported. This is not true.



    Now let's look at OffscreenCanvas. As we have already seen, Canvas is a very powerful tool, but, unfortunately, it is not available to us in the context of Web Workers, so we will use an alternative. This is a fairly new thing called OffscreenCanvas. It allows you to do about the same things as Canvas, only already off-screen, that is, in the context of Web Workers. Of course, we can do this in the context of window, but now we will not.



    What is there with support? As you can see, there is a lot of red. OffscreenCanvas is normally only supported in Chrome. There is also an option with Firefox, but so far there is a flag, and Canvas only works with WebGL context. Here you can ask - why am I talking about such a cool thing like OffscreenCanvas, which does not work anywhere?



    A small digression. We have some levels of browser support in the Market. And we have two quantities. One value characterizes the browser, which we do not support at all. This is about half the percentage of browser popularity.

    And there is a second quantity. It includes those browsers that we support, but only critical functionality. Here, without Workers, all the search functionality works, but with small friezes. I think it’s ok, and our team believes that it’s ok. Let's see how we will implement this.



    Here is a diagram of what we will do. We even have files that we will read through FileReader. But in the main stream we will send it to Web Workers, where it will be cut, compressed and will return back to us, and we will already send it to the server.



    Let's see the code of our Worker. First, we create an OffscreenCanvas instance with the width and height we need.

    Further, as I said, the Image element is not available to us in the Workers context, so here we use the createImageBitmap method, which will make us the data structure that characterizes our picture.

    From the interesting: we see here self. Those who are not familiar with Web Workers, this thing points to the execution context. It doesn’t matter to us here, window or this, we use self. This method is asynchronous, I used await here for compactness and convenience, why not?

    Next, we get the same image and do the same thing that we did before. Draw on the canvas and return.

    From the simple. We used to take DataURL and convert everything to blob. But here the convertToBlob method is immediately available to us. Why haven't I used it before? Because the support was worse. But since we went all the way here and use OffscreenCanvas, what prevents us from using convertToBlob?



    We will return this blob basically a stream, from where we will send it to the server. Or, as in the demos, draw it.

    Here we create a Worker in the main thread, listen to some messages from it and draw or send to the server. There is nothing important here. Worker will accept our files.

    Let's get back to our demo.

    Watch the fourth demo

    All the same demo, all the same three cats and a hedgehog. I will turn on throttling again, slowing the processor six times. I’ll upload the same photo. As we see, at the time the picture was drawn, the animations did not stop, the hedgehog continued to spin, the interface remained, and we achieved what we wanted.

    But can this decision be improved?



    Here, by the way, is a profiler. Here we do not see the huge Microtasks for the five seconds that we saw before.

    Improvement is possible. Using Transferable objects. Here it is worth going back again. When we passed our DataURL or blob through the postMessage mechanism, we copied this data. This is probably not very effective. It would be cool to avoid it. Therefore, we have a mechanism that allows you to transfer data to Web Workers as if in a package.

    Why do I say “like”? When we transfer this data to Workers, we lose control over them in the main stream - we cannot interact with them in any way. There is a second limitation here. We cannot transfer all data types to Web Workers. This cannot be done with the string, we will do it differently.



    Let's look at the code. Firstly, we transmit data a little differently. Here is our postMessage. You see, there is such an array with loadEvent.target.result. Such an interface allows us to transfer our data as Transferable objects, losing control over them.

    By the way, anyone who writes in Rust will probably hear something familiar. And we will read our file not as a string, but as an ArrayBuffer. This is a stream of lidar binary data to which there is no direct access. Therefore, we will have to do something else with them.



    Back to our ImageWorkers. Here it became much more interesting. First, we take our buffer and do such a terrible thing as Uint8ClampedArray. This is a typed array. As the name implies, the data in it are the sign numbers, that is, numbers from zero to 255 that will represent the pixel of our image.

    The third argument we pass such a strange thing, as the width, multiplied by the height, multiplied by four. Why exactly four? Exactly, RGBA. There are three values ​​per color and one per alpha channel.

    Next, we will make ImageData from this array, a special data type that can be easily drawn on the canvas. Nothing interesting here. We just take an array and pass it to the constructor. Further, in the same way we draw our picture on the canvas, but using a different method, under ImageData. Further, everything is the same as it was before.

    Let's move on to the conclusions. Today I told you about one task that I did not so long ago. What did I notice in it?



    The smoothness of the interface is very important. When the user lags a little, a little freezes, the button is not pressed, this can lead to a severe deterioration in UX. Browsers work differently. We looked at a spherical example with Safari and Yandex.Browser. We see that if you checked your interface for smoothness in one browser, you should look at the others.

    You need to do something with blocking scripts if they continue for a long time. In my case, I put it on Web Workers. But there are probably other approaches, you can somehow divide them into smaller ones, here you have to think. And we have a whole set of modern or not very modern tools, for example Web Workers, that help us solve all these problems.

    What next? I urge you to check all your interfaces for smoothness. It is very important. And remember about weak devices. We sit with coffee, or smoothies with a laptop for 200 thousand and do not always look at how our interfaces work on popular phones.

    Nothing prevents you from using Web Workers now. The support is very good, the technology is not new, proven and quite solves some problems of interface freezes.

    A few related links:


    Many thanks.

    Also popular now: