July 3, 2019 at 12:04How we learned to draw texts on canvas
We are developing a platform for visual collaboration . We use Canvas to display content: everything is drawn on it, including texts. There is no ready-made solution for displaying texts on Canvas one to one as in html. For several years of working with text rendering, we studied various implementation options, filled a lot of bumps and, it seems, found a good solution. I’ll tell you in an article how we moved from Flash to Canvas and why we abandoned SVG foreignObject.
We created the product in 2015 on Flash. Inside Flash there is a text editor that can work well with texts, so we did not need to do anything extra to work with texts. But at that time Flash was already dying, so we moved from it to HTML / Canvas. And before us the task was to display the text on the Canvas as in the html editor, while not breaking the texts created in the Flash version when moving.
We wanted to make it so that the user could edit the text directly in our product, without noticing the transition between the editing and rendering modes. The solution we saw is this: when you click on an area with text, a text editor opens in which you can change the text; You can close the editor by moving the cursor away from the text area. In this case, the display of text on the Canvas should 1 in 1 correspond to the display of text in the editor.
As an editor, we used an open library, but ready-made libraries for rendering from html to Canvas did not suit us with the speed of work and insufficient functionality.
We examined several solutions:
How SVG foreignObject works: we have HTML from the editor → we put HTML in foreignObject → some magic → we get the image → we add the image to canvas
About magic. Despite the fact that most browsers support the foreignObject tag, each has its own characteristics for using the result with canvas. FireFox works with a Blob object, in Edge you need to do Base64 for the image and return data-url, and in IE11 the tag does not work at all.
After working with SVG, we got interesting bugs that we did not notice on Flash. Text with the same size and font in different browsers was displayed differently. For example, the last word in a line could be wrapped and run into the text below. It was important for us that users get the same kind of widgets, regardless of the browsers they work in. There was no problem with Flash on this, as he is the same everywhere.
We have solved this problem. Firstly, for all single-line texts, they began to always consider the width regardless of the browser and data from the server. For height, the difference remains, but in our case it does not bother users.
Secondly, experimentally we came to the conclusion that it is necessary to add some unusual css styles for the editor and svg in order to reduce the difference in display between browsers:
What in the end we got thanks to SVG:
Everything worked well, but once designers came to us and asked to add font support to create mockups.
We wondered if we could do this with foreignObject. It turned out that he has a feature that, when solving this problem, becomes a fatal flaw. It can display HTML inside itself, but cannot access external resources, so all the resources it works with must be converted to base64 and added inside svg.
This means that if you have four texts written by OpenSans, you need to download this font to the user four times. This option did not suit us.
We decided that we would write our Canvas Text with ... good performance, support for vector images, we will not forget about IE 11
Why is a vector image important to us? In our product, any object on the board can be zoomed, and with a vector image we can create it only once and reuse it regardless of the zoom. Canvas.fillText draws a bitmap: in this case, we need to redraw the image with each zoom, which, as we thought, greatly affects performance.
First of all, we created a simple prototype to test its performance.
The principle of operation of the prototype:
The prototype had several tasks: to check that Canvas redrawing with the scaling will take place without delay and that the time for turning html into an object will be no more than creating an svg image.
The prototype coped with the first task, the scaling almost did not affect the performance when drawing texts. There were problems with the second task: processing large amounts of text takes enough time and the first measurements of performance showed poor results. To draw text from 1K characters, the new approach took almost 2 times more time than svg.
We decided to use the most reliable way to optimize the code - “replace the test with the one we need” ;-). But seriously, we went to the analysts and asked how long the texts are most often created by our users. It turned out that the average text size is 14 characters. For such short texts, our prototype showed significantly better performance results, as the dependence of speed on the volume of text is linear, and wrapping in svg is almost always done at the same time, regardless of the length of the text. It suited us: we can lose in performance on long texts, but in most cases our speed will be better than svg.
After several iterations of working on the Canvas Text update, we got the following algorithm:
Stage 1. Break into logical blocks
Stage 2. We collect in one object with coordinates and styles
The advantage of this approach is that we can cover all the code from HTML to a text object with unit tests. Thanks to this, we can separately check the rendering and the parsing itself, which helped us to significantly speed up the development.
As a result, we made support for fonts and IE 11, covered everything with unit tests, and the rendering speed in most cases became higher than that of foreignObject. Checked in beta users and released. Success seems to be!
So far, guys with a right-handed writing system have not written technical support. It turned out that we forgot about the existence of such languages:
Fortunately, adding support for the right-sided writing system was not difficult, since the standard Canvas.fillText already supports it.
But while we were dealing with this, we came across even more interesting cases that fillText could no longer support. We came across bidirectional texts in which part of the text is written from right to left, then from left to right and again from right to left.
The only solution we knew was to go into the W3C specification for browsers and try to repeat this inside Canvas Text. It was difficult and painful, but we were able to add basic support. More on bidirectional: one and two .
If you have experience in solving such problems - share it in the comments.
Moving with Flash
We created the product in 2015 on Flash. Inside Flash there is a text editor that can work well with texts, so we did not need to do anything extra to work with texts. But at that time Flash was already dying, so we moved from it to HTML / Canvas. And before us the task was to display the text on the Canvas as in the html editor, while not breaking the texts created in the Flash version when moving.
We wanted to make it so that the user could edit the text directly in our product, without noticing the transition between the editing and rendering modes. The solution we saw is this: when you click on an area with text, a text editor opens in which you can change the text; You can close the editor by moving the cursor away from the text area. In this case, the display of text on the Canvas should 1 in 1 correspond to the display of text in the editor.
As an editor, we used an open library, but ready-made libraries for rendering from html to Canvas did not suit us with the speed of work and insufficient functionality.
We examined several solutions:
- Standard Canvas.fillText. Able to draw text as in html, can be styled, works in all browsers. But it doesn’t know how to draw links like in a html editor multi-line texts with different formatting. These difficulties can be solved, but require a lot of time;
- Draw DOM on top of Canvas. The option did not suit us, because in our product, each created object has its own z-index on canvas. And mixing it with the DOM z-index will not work.
- Convert html to svg. He is able to turn html into an image thanks to the foreignObject element. This allows you to bake html inside svg and work with it as an image. We have chosen this option.
Features SVG foreignObject
How SVG foreignObject works: we have HTML from the editor → we put HTML in foreignObject → some magic → we get the image → we add the image to canvas
About magic. Despite the fact that most browsers support the foreignObject tag, each has its own characteristics for using the result with canvas. FireFox works with a Blob object, in Edge you need to do Base64 for the image and return data-url, and in IE11 the tag does not work at all.
getImageUrl(svg: string, browser: string): string {
let dataUrl = ''
switch (browser) {
case browsers.FIREFOX:
let domUrl = window.URL || window.webkitURL || window
let blob = new Blob([svg], {type: 'image/svg+xml;charset=utf-8'})
dataUrl = domUrl.createObjectURL(blob)
break
case browsers.EDGE:
let encodedSvg = encodeURIComponent(svg)
dataUrl = 'data:image/svg+xml;base64,' + btoa(window.unescape(encodedSvg))
break
default:
dataUrl = 'data:image/svg+xml,' + encodeURIComponent(svg)
return dataUrl
}
After working with SVG, we got interesting bugs that we did not notice on Flash. Text with the same size and font in different browsers was displayed differently. For example, the last word in a line could be wrapped and run into the text below. It was important for us that users get the same kind of widgets, regardless of the browsers they work in. There was no problem with Flash on this, as he is the same everywhere.
We have solved this problem. Firstly, for all single-line texts, they began to always consider the width regardless of the browser and data from the server. For height, the difference remains, but in our case it does not bother users.
Secondly, experimentally we came to the conclusion that it is necessary to add some unusual css styles for the editor and svg in order to reduce the difference in display between browsers:
- font-kerning: auto; controls the kerning of the font. More details
- webkit-font-smoothing: antialiased; responsible for smoothing. More details .
What in the end we got thanks to SVG
- We can draw any html: text, tables, graphics
- The tag returns a vector image.
- The tag works in all modern browsers except IE11
Why we abandoned foreignObject
Everything worked well, but once designers came to us and asked to add font support to create mockups.
We wondered if we could do this with foreignObject. It turned out that he has a feature that, when solving this problem, becomes a fatal flaw. It can display HTML inside itself, but cannot access external resources, so all the resources it works with must be converted to base64 and added inside svg.
This means that if you have four texts written by OpenSans, you need to download this font to the user four times. This option did not suit us.
We decided that we would write our Canvas Text with ... good performance, support for vector images, we will not forget about IE 11
Why is a vector image important to us? In our product, any object on the board can be zoomed, and with a vector image we can create it only once and reuse it regardless of the zoom. Canvas.fillText draws a bitmap: in this case, we need to redraw the image with each zoom, which, as we thought, greatly affects performance.
Create a prototype
First of all, we created a simple prototype to test its performance.
The principle of operation of the prototype:
- We give to the function “text”;
- From it we get an object in which there is every word from the text, with coordinates and styles for rendering;
- Give the object to Canvas;
- Canvas draws text.
The prototype had several tasks: to check that Canvas redrawing with the scaling will take place without delay and that the time for turning html into an object will be no more than creating an svg image.
The prototype coped with the first task, the scaling almost did not affect the performance when drawing texts. There were problems with the second task: processing large amounts of text takes enough time and the first measurements of performance showed poor results. To draw text from 1K characters, the new approach took almost 2 times more time than svg.
We decided to use the most reliable way to optimize the code - “replace the test with the one we need” ;-). But seriously, we went to the analysts and asked how long the texts are most often created by our users. It turned out that the average text size is 14 characters. For such short texts, our prototype showed significantly better performance results, as the dependence of speed on the volume of text is linear, and wrapping in svg is almost always done at the same time, regardless of the length of the text. It suited us: we can lose in performance on long texts, but in most cases our speed will be better than svg.
After several iterations of working on the Canvas Text update, we got the following algorithm:
Stage 1. Break into logical blocks
- We break the text into blocks: paragraphs, lists;
- We break blocks into smaller blocks according to styles;
- We break the blocks into words.
Stage 2. We collect in one object with coordinates and styles
- Count the width and height of each word in px;
- We connect the divided words, since in point 2 some words were divided into several;
- From the words we collect the lines, if the word does not fit into the line, we cut until it fits;
- We collect paragraphs and lists;
- We calculate x, y for each word;
- We get a ready-made object for rendering.
The advantage of this approach is that we can cover all the code from HTML to a text object with unit tests. Thanks to this, we can separately check the rendering and the parsing itself, which helped us to significantly speed up the development.
As a result, we made support for fonts and IE 11, covered everything with unit tests, and the rendering speed in most cases became higher than that of foreignObject. Checked in beta users and released. Success seems to be!
Success lasted 30 minutes
So far, guys with a right-handed writing system have not written technical support. It turned out that we forgot about the existence of such languages:
Fortunately, adding support for the right-sided writing system was not difficult, since the standard Canvas.fillText already supports it.
But while we were dealing with this, we came across even more interesting cases that fillText could no longer support. We came across bidirectional texts in which part of the text is written from right to left, then from left to right and again from right to left.
The only solution we knew was to go into the W3C specification for browsers and try to repeat this inside Canvas Text. It was difficult and painful, but we were able to add basic support. More on bidirectional: one and two .
Brief conclusions we made for ourselves
- To display HTML in a picture, use SVG foreignObject;
- Always analyze your product for decision making;
- Make prototypes. They can show that complex decisions can only seem so at first glance;
- Write code immediately so that it can be covered with tests;
- In an international product, it is important not to forget that there are many different languages, including biderectional.
If you have experience in solving such problems - share it in the comments.