Create your own Angular 2

Hello, this article describes the working mechanism of Angular 2. Today we look under the hood of a popular framework.

This article is a free translation of the report by Tobias Bosch - The Angular 2 Compiler. You can find a link to the original report at the end of the article.

Overview

Tobias Bosch is a Google employee and member of the Angular development team who created most of the compiler components. Today, he talks about how Angular works from the inside out. And this is not something sky-high complicated.

I hope that you use some of the knowledge gained in your applications. Perhaps you, like our team, will create your own compiler or a small framework. I will talk about how the compiler works and how we achieved our goal.

What is a compiler?

This is all that happens between the application input (it can be your commands, templates, etc.) and your running application.

Everything that happens between them is the domain of the compiler. In appearance it may seem that this is pure magic. But this is not so.

What will we consider today?

First, we'll talk a little about performance: what is meant by fast work? We will talk about what types of input we have.

We will also talk about lexical analysis, what it is, and what the Angular 2 compiler does with analysis. We will talk about how Angular 2 takes the analyzed data and processes it. In general, the implementation of this process took place in three attempts, at first there was a simple implementation, then improved and even more improved. So did We. These are the stages thanks to which we were able to achieve accelerated work of our compiler.

We will also discuss different environments, the pros and cons of dynamic (Just In Time) and static (Ahead Of Time) compilation.

Performance: what does fast work mean?

Imagine a situation: I wrote an application and claim that it works quickly. What I mean? The first thing you might think is that the application loads very quickly. Did you see Google AMP pages? They load unrealistically fast. This can be attributed to the concept of "fast."

Perhaps I am using a large application. I switch from one heading to another. For example, from a brief overview to a maximized page, and such a transition occurs very quickly. It can also characterize speed.

Or, say, I have a large table and now I just change all its values. I create new values ​​without changing the structure. It is really fast. All these are different aspects of the same concept, different things that you need to pay attention to in the application.

Path Navigation

I would like to dwell in more detail at the stage of switching along the paths (switching a route).

An important point - when moving along paths, the framework destroys and recreates everything anew. He does not adhere to the structure: everything is destroyed and everything is recreated anew.

How to make this process fast?

To do something quickly, you need to measure this speed. To do this, you need a test. One of the performance tests we use is a deep-tree performance test. You may have heard of this. This is a component that is used twice. This is the recursion of the component until a certain depth is reached.



We have 512 components and buttons that can destroy or create these components. Next, we measure how much time it takes to destroy and create components. And so there is a transition along the paths. Transition from one view to another. The destruction of everything is the creation of everything.

What input do we have?

Components

We have components, and I think everyone knows them.



They have a template. You can make templates inline or put them in a separate file, it is worth remembering that they have a context.

Instances of components are in essence a context, data that is used to build a template. In our case, we have a user, the user has a name (in our case, this is my name).

Template

Next we have a template. This is simple HTML, you can insert something like an input form here, you can apply everything that HTML offers.



Here we have some new syntax: remember double braces, square and parentheses? This is the binding of Angular to properties or events. Today we will talk only about braces and what they mean. From the point of view of semantics, this means "take data from the component and put it in a specific place." When data changes, the text should be updated.

Directives



Directives contain a selector - this is a CSS selector. The bottom line is that when Angular goes through the markup, if it finds a directive that matches an element, then it executes it.

Suppose we have a selector for forms, and that’s how we say it - every time we create a form element, please create this directive.

Likewise with ngModel. When you create the ngModel attribute, you must create a directive as well.
These directives may have dependencies. This is our expression of addiction.



The dependency has a hierarchical structure so that ngModel requests ngForm.

And what does Angular do?

He scans the entire tree in search of the nearest ngForm, which is one level higher in the tree structure. It will not view siblings, only parent ones.

There are other input data, but we will not dwell on them in detail.

Everything that is done in Angular goes through the compiler.

Ok, we have input. The next thing we need is to understand them.
Is it just a bunch of gibberish, or does it make any sense?

Lexical analysis process

First stage

Let's say we have some kind of template, for example, HTML.

Presentation of the template



How can we present it in such a way that the compiler understands it?

This is what the analyzer does. He reads each character and then analyzes the meaning. That is, he creates a tree. Each element has only one object. Suppose there is some name - the name of the element, there are child elements. Let's just say a text node is a JSON object with text properties. We also have element attributes. Let's just say we code it all as nested lists. The first value is the key, the second is the attribute value. And so on, there is nothing complicated. Such a representation is called an abstract syntax tree (ASD, english - AST). You will often hear this concept. That is all HTML.

How can we portray an element with data?



We can display it as follows. This is a text node, that is, we have a JSON object with text characteristics. The text is empty because initially there is no text to display. The text depends on the input. Incoming data is presented in this expression.
Any expressions are also analyzed for what they mean.
You cannot declare a function inside expressions or use a for loop, but we have things like pipes with which you can work with expressions.

We can portray this expression, user.name, as the path to the property. And also we can catch where this expression came from exactly from your template.

Locating an expression
So why is this so important? Why is it important for us to know where exactly this expression came from ASD?

This is because we want to show you error messages during program execution. Let's say your user knows about these errors.

And then there are exceptions? For example, it is not possible to read a name from undefined. If this happens, then you need to go to the error debugger and check to set the breakpoint on the first error. Then you must understand exactly where the error occurred.

Angular compiler provides you more information



It shows from which exact place in the template the legs “grow” of this error. The goal is to show you that the error takes its roots, for example, from this particular interpolation in the second row, column 14 of your template. To do this, we need the row and column numbers in the ASD.

Further, what analyzers do we need to build this ASD?

There are many possibilities. For example, we can use a browser.

The browser is a great HTML parser, right? He does this every day. We had this approach when developing Angular 1, we started using the same approach when developing Angular 2.

Now we do not use the browser for this purpose for two reasons:

1. You cannot get row and column numbers from the browser. When parsing HTML, the browser simply does not use them.
2. we want Angular to work on the server as well.

Obviously there is no browser on the server. We could say: in the browser we use the browser, and on the server we use something else. So it was. But then we got into deadlocks, for example, with SVG, or with the placement of commas, so we needed to have the same semantics everywhere. Therefore, it’s easier to insert a JavaScript fragment, and a parser. This is exactly what we do.

So, we talked about HTML and expressions.

How do we present the directives we find?

We can display them through JSON objects that represent the elements by simply adding another property: directives. And we refer to the constructor functions of these directives.

In our input example with ngModel, we can depict this element as a JSON object. It has input name, attributes, ngModel and directives. We have a pointer to the constructors, and we also catch dependencies, because we need to indicate that if we create ngModel, we need ngForm, and we need to catch this information.

Given an SDA with HTML information, links, and directives, how do we bring this to life? How can this be done easiest?

We’ll deal with HTML first. What is the easiest way to create a DOM element? First, you can use innerHTML. Secondly, you can take an existing element and clone it. And thirdly, you can call document.createElement.

Let's vote. Who believes innerHTML is the fastest? And who believes that element.cloneNode will create the element the fastest? Or maybe the fastest way is element.createElement?

Obviously, everything changes over time. But for now:

• innerHTML is the slowest option. This is obvious because the browser needs to call its parser, take your string, go through each character and build a DOM element. Obviously, this is very slow.
  
  
  
  
• element.cloneNode is the fastest way, because the browser already has a projection built, and it just clones that projection. This is just adding another element to the memory. That is all that needs to be done.
  
  
• document.createElement is something between the two previous ways. This method is very close to element.cloneNode. Slightly slower, but very close.
  
  
  

You will say, “OK, let's use element.createElement to create a new DOM element.”

This is how Angular 1 worked, and we also started when developing Angular 2. And by tradition, it turns out that this is not a completely honest comparison, at least not in the case of Angular. In the case of using Angular, we need to create some elements, but, in addition to this, we must place these elements. In our case, we want to create new text nodes, but also we need to find the one that is responsible for user.name, since later we want to update it.

Therefore, if we compare, then we must compare both creation and placement. If you use innerHTML or cloneNode, you will have to go all the way through the construction of the DOM again. When using createElement or createTextNode you bypass these actions. You simply call the method and immediately get its execution. No new constructions and other things.

In this regard, when comparing createElement and createTextNode, they are both approximately the same in speed (depending on the number of bindings).

Secondly, much less data structures are required. You do not need to track all of these indices and so on, so such methods are simpler and almost identical in speed. Therefore, we use these methods, and other frameworks also switch to this approach.

So, we can already create DOM elements.

Now we need to create directives.

We need to inject dependencies from child to parent. Suppose we have a data structure called ngElement that includes a DOM element and directives for that element. There is also a parent element. This is a simple tree inside the DOM tree.



And how can we create DOM elements from the SDA?

We have a template, we have an element from which we built an ASD. What can we do with all this?



In our ngElement and constructor, call document.createElement, look at the attributes and assign them to the elements, and then add the element to our parent element. As you can see, no magic.

Further we pass to directives. How it works?

We look at the bindings, somehow get them (I'll talk about this a bit later) and just call new again for the constructor, give it the bindings and save the Map. Map will go from directive type (ngModel) to directive instances.

And all this directive search will work this way: we will have a method that receives a directive that first checks the element itself (if it has a directive). If not, then return to the parent and check there.

This is the easiest thing to do. We ourselves started in this direction. And it works.

Important detail: bindings. How to display bindings?

You just create a binding class that has target - Node. It will be a text node.
Target has a property, in our case it will be the value of the node, this is the place where the value will fit. And the expression.



The binding works this way: every time you evaluate an expression or when it just changes, you save it to target.

That is, you can have the following method: first you evaluate the expression, if it has changed, then update the target and other values ​​previously saved.



As for the exceptions that were discussed earlier, we call the try catch methods to track the evaluation path. When an exception is thrown, we re-generate it and create a model for it from the row and column numbers.

So we get the row and column numbers in which there are errors. This is all we tie into the view. This is the last data structure.



A view is an element of a template. That is, when we look at the error code, we will see many representations. These are just elements of the template. And we put them together in a performance. The view refers to the component, ng elements and bindings, as well as the dirty-check method, which looks at the bindings and checks them.

So we have finished the first stage. We have a new compiler. How fast are we? Almost at the same level as Angular 1. Already not bad. Using the simpler approach, we achieved the same speed. But Angular 1 is not slow.

Second stage

How do we speed up the process? What is the next step? What did we miss? Let's get it right.

We need something that is relevant to our data structures. When it comes to data structure, this is actually a very difficult question. If we compare with the last program we wrote, where try-catch appears, but if we discard it, we will see that many functions slow down the process, and that many points need to be optimized. If you consider the cause of the slow operation of your data structure program, then this is a very difficult question, because they are scattered throughout your program. This is just an assumption that the point is data structures.

We conducted experiments and tried to figure it out. We watched these directives: Map inside ngElements.



It turns out that for each element in the DOM tree we create a new map? One could say that there are no directives, we did not create them. But still, we always create a map, fill it with directives and read information from it. It can be uneconomical, it can overload memory, reading still takes some time too.

Alternative approach: we can say: “OK, we allow only 10 directives for one element. Next we create an inlinengElement class, 10 properties for directive elements and directive types, and in order to find the directive we create 10 conditional IF statements. ” It's faster? Maybe.

It does not consume a lot of memory resources, right?

For example, setting: you are setting a property, not a map. Reading may be a little slow due to 10 conditions. This is exactly the case for which the JavaScript VM has been optimized. JavaScript VM can create hidden classes (you can google about them at your leisure). This makes JavaScript VM faster. Switching to this data structure is what speeds things up. Later we look at the results of performance tests. Another thing that needs to be optimized for data structures is the reuse of existing instances.

You can ask a logical question - If some lines are destroyed and others are restored, then why not cache these lines in the cache and change the data as soon as the lines appear? So we did. We created the so-called view cache, which restores old instances of views.



Before moving to a new pool, you need to destroy the state. The state is contained in the directive. So we kill all the directives. Next, when the pool exits, you need to recreate these directives. The hydrate and dehydrate methods do this. We saved the DOM nodes, because everything comes from the model, all the status is in the model. Therefore, we saved it. And again, we conducted a performance test.

Test environment

For you to understand the results of these tests, it is worth noting that Baseline is a program with manual JavaScript code and hard coding. No frameworks were used in such a program, the program was written only for this deep-tree test. The program produces dirtyChecking. We took this program as a unit. Everything else is compared in the ratio. Angular 1 got a mark of 5.7.



Previously, we showed the same speed with optimized data structures and without a view cache. We were at 2.7. So, this is a good indicator. We doubled the speed due to quick access to properties. At first we thought that our work was ending there.

Disadvantages of the second stage.

We have created applications based on this. But then we saw the flaws:

1. ViewCache is a bad friend of memory. Imagine you are switching request processing routes. Your old queries remain in memory because they are cached, right? The question is when to remove requests from the cache? This is actually a very difficult question. One could create a few simple elements that would allow the user to choose whether to cache something or not. But that would be, at least, strange.
   
   
2. Another problem: DOM elements have a hidden state. For example, an element is placed in focus. Even if you don’t have a snap to focus, and the element can be both in focus and out of it, then deleting it or returning it can change the focus of this or other elements. We did not think about this. Bugs related to this have appeared. One could go this way: we could completely remove the elements in order to remove their state, and even restore them. But that would negate ViewCache performance if we had to recreate the DOM. After all, we had an indicator of 2.7. How would we achieve speed in such a situation?

Third stage

The view class.

Then the thought came to us: let's look again at our view class. What do we have there?
We have a component - these are already optimized ngelement, right? We have bindings. But the view class still contains these arrays. Can we create an InlineView that also uses only properties? No arrays. Is it possible? It turned out, yes.

What does it look like? Like that.



Template

So, we will have a template, as before, and for each element we will just create code. For this template, we will create code that displays our view class. In the constructor for each element, we call document.createElement, which will be stored in the Node0 property - for the first element, for the second, we again call document.createElement, which will be stored in Node1. Further, when we need to attach an element to our parent, we have properties, right? We just need to do everything in the correct order. We can use the property to refer to the previous state. This is exactly what we will do with the DOM.



Directives

With directives we do the same. We have properties for each instance. And again, we just need to make sure that the order of the actions is correct: that dependencies come first, and then those components that use these dependencies. That we use ngForm first and then ngModel.

Bindings

Next, the bindings. We just create code that does dirty-check. We take our expressions, convert them back to JavaScript. In this case, it will be this.component user.name. This means that we pull user.name from the component, compare it with the previous value, which is also a property. If the value has changed, we update the text node. In the end, we reduce everything to a representation with a data structure. It has only properties. There are no arrays, Map, everywhere quick access through properties is used.

This greatly speeds up the process. Soon I will show you the numbers so that you can see this.

The question is: how do we do this?

Let's say someone needs to create a string that evaluates this new class. How it's done? We just apply what we did in implementation 101 in our current implementation.

The bottom line is this: If earlier we created DOM nodes, now we are creating code to create DOM nodes. If we used to compare elements, now we are creating code to compare elements. Finally, if we used to refer to directive instances or DOM nodes, now we store properties where directive instances and DOM nodes are stored. In this case, the code looks like this.



Before, we had our ngelement, now we have compileElement. In fact, these classes now exist in the compiler. There is compileElement, compileView and so on.

The display will be like this: before we had a DOM element, now we only have a property in which the DOM element is stored. We used to call document.createElement, now we are creating a line with this new string interpolation, which is great for creating code in which we say that this.document + its property name is equal to document.createElement with the name of the ADD.

And finally, if we called appendChild before, now we are creating code to attach the child to the parent. The same thing happens with the search for directive dependencies. Everything happens according to the same algorithm, only now for these purposes we are creating code.



If we now look at the indicators, we will see that now we have significantly increased speed. If earlier our indicator was 2.7, now it is 1.5. It is almost twice as fast. ViewCache, still, remains a little faster. But we ruled out the option of using it, and you already know the reasons for our decision. We did a great job, and could have finished already. But no.

Dynamic (Just in Time) compilation

So, at the beginning we talked about dynamic (Just in Time) compilation. Dynamic - this means that we compile in the browser.



Recall this works something like this: you have some incoming data that is on your server. The browser loads and picks them up, analyzes everything, creates the original view class. Now we need to evaluate this source code in order to get another class. After that, we can create this class, and then we get a working application. There are certain problems with this part:

1. The first problem is that we need to analyze and create code inside the browser. It takes some time, depending on how many components we have. We have to go through all the characters. Of course, this happens fast enough. And yet, this also takes time. Therefore, your page will not load in the fastest way.
   
   
2. Вторая проблема – это то, что анализатор должен находиться в браузере, поэтому нужно загрузить весь Angular компилятор в браузер. Поэтому, чем больше свойств мы добавляем и чем больше мы создаем кода, тем больше становиться размер.
   
   
3. Следующая проблема – это использование eval. При использовании eval вы можете сделать ошибки, из-за которых кто-то может взломать ваш сайт. Поэтому вам не стоит использовать eval, даже если вы думаете, что с вашим кодом все в порядке.
   (примечание, — детальнее о том почему стоит избегать использование eval читайте тут)
   
   
4. Еще одной проблемой является то, что сейчас браузер добавляет политику защиты содержимого в качестве отличительной особенности (возможно, вы знаете об этом). С таким раскладом, сервер может дать указание браузеру, что не нужно вообще оценивать ту или иную страницу. Для браузера это может означать, что на этой странице не нужно запускать Angular вообще. Это не очень хорошо.
   
   
5. Следующая проблема, с которой мы столкнулись, это минификация кода. Если вы хотите минифицировать код (а я надеюсь, что вы так и делаете, чтобы предоставлять клиенту как можно меньше кода), существуют кое-какие уловки, которые вы можете применить:
   
   
   1. Во-первых, вы можете переименовать переменные, обычно это безопасно, потому что вы можете определить, где используется та или другая переменная.
   2. Во-вторых, вы можете переименовать свойства объектов. Это сложнее, поскольку вы должны знать, где же именно используется тот или иной объект. Компилятор Closure поддерживает такую функцию (она называется «улучшенная минификация»), мы его использовали в Google и использовали часто.

So, if we use improved minification while the compiler is running, this is what happens.

There are our components, our markup, they are loaded into the browser, the browser parses the template and creates the view class. So far, everything is going fine. The browser uses user.name, the component also contains user.name, just this user.name is minified using advanced minification technology. Thus, the component is called, say, C1, and my user suddenly turns out to be just U, and the name is N. The problem is that the minifier is not aware of my template. Indeed, the user.name is still in the template.

So, the template is executed, it still creates user.name, which simply does not exist in the component. There are certain solutions to this problem. You can tell the component that this property does not need to be minified. But that is not what we need. We need to be able to minify this property, but this will not work with online compilation and evaluation.

Static (Ahead of Time) compilation

For this reason, our next step was the appearance of static (Ahead of Time) compilation. She works as follows.


Again, we have input that is parsed on the server, and the view class is also created on the server.

Then the browser just picks them up, loads it like regular script text (like you load your regular JavaScript code), and then just creates the elements you need.

This compilation method is great, because: Analysis takes place on the server (and therefore happens quickly), the compiler does not need to be transferred to the browser (and this is also excellent). Also, we no longer use the rating, because this is a script. Therefore, such a compilation is suitable for any browser.

Also, static compilation is great for improved minification, because now we create a view class on the server, and if we run the minifier, we can also minify our view classes. Now that the minifier has done its job, we get the renamed properties in the classes.

Great, now we can use the improved minification. Therefore, our speed indicators have fallen even lower.

Disadvantages of Static Compilation

So now we have static compilation. Now everything is fine, right? But as always, there are disadvantages. The first problem is that using static compilation we need to create different code.

To evaluate in a browser, you need to create code according to the ES5 standard. You are creating code that uses local variables, or that receives the arguments passed to the function. You are not using require.js or anything like that.

Since the code is generated on the server, it was necessary to generate TypeScript code for two reasons:

• First, we wanted to check the types of your expressions (whether they exist in the components).
• Во-вторых, мы хотели эффективно использовать систему модулей TypeScript. Мы хотели создать импорт, чтобы уже вы могли выбирать, какие системы модулей вы хотите использовать. Создание импорта — это гораздо удобнее со стороны возможностей генерации кода, чем думать о require.js, system.js, Google Closure, системе модулей и так далее.
• В конце концов, мы также хотели применить код стандарта ES6 2016.

How did we manage to provide support for ES6 2016?

In fact, if you are familiar with compilers, then there is a common pattern. Instead of creating strings, a data structure similar to the output is created, but this data structure, this ADA, can be serialized into different output data.

ASD contains such features as variable declaration, method calls and so on, plus types. Then for ES5 we just serialize it all without types, and for TypeScript we serialize with types.

It looks something like this: our generated SDA output, inside the declared variables (we specify declare var name EL). This will create the var el code in TypeScript and create the type. In ES5, the type will be omitted.

Next we can call the method. First we read the document variable (since it is a global variable). And then for her we can call the createElement method with these arguments.

We placed literal with the value “div”, because if you parse strings you must escape them correctly. The value may contain quotation marks, so when reading the code you need to escape them, skip them. Therefore, in this way we can do it. The good news is that now our generated code looks the same both on the server and in the browser. No different pieces of code.

The second problem we encountered while developing a static compiler is the allocation of metadata. Take a look at this code.



What is the problem of this code? Let's say we have some kind of directive that is dependent on cookies, and if you have cookies, the directive does something else. It works. You can compile. Super.

But this does not work with static compilation. If you think about it, why? If all this is lowered to the level of ES5, you will get exactly that code.

What does the decorator ultimately do? It adds a property to your constructor for metadata. In the end, he simply adds SomeDir with notes.

The problem is that if you run this on the server, it will not work. After all, there is no document on the server. What to do?

You can suggest building a browser environment on the server, declaring a document variable, a window variable, and so on. In some cases, this will work, but in most cases it will not. The second way (we are now well versed in ASD, right?) Is to process the ASD and remove metadata from it without evaluating the code. In ASD this can be represented somehow.



Thus, our ADD SomeDir class may have a decorators property that refers to the element that you are calling (this is the expression where the directive plus arguments are defined). The following happens. We pull the metadata into JSON files, then our static compiler takes these files and creates a template from them. Obviously, there are limitations. We do not evaluate JavaScript code directly, so you cannot do everything that you are used to doing in the browser.

So, our static compiler limits us, but you can put a note. But if you use the above method, it will work in all cases.

Ok, let's dwell on performance tests again.



This is a simple program we wrote. And its loading time was reduced very significantly. This is because we no longer do the analysis, and also because the Angular 2 compiler no longer loads. It is almost three times faster. The size has also been reduced from 144 kilobytes already minified in gzip format to 49. For comparison, the React library with improved minification now weighs 46kb.

Now for some illustrative examples.

Let's say we have an Angular component, we have ngForm and ngModel. ngModel has ngForm as a dependency.



Add NGC here - these are _node modules that span the TypeScript compiler, because when you pull out and support metadata, these modules depend on TypeScript. And if we look at the generated code, we will see that it looks very familiar. We have user.name. If its value changes, we update the directive input. We simply compare with the previous value and set it.

There is a good side to this. Suppose in my template I change user.name to wrongName. Our template refers to user.name, not user. wrongName. Now, when we look at the generated code, TypeScript will throw an error.



Because these representations have types, based on the type of component. And now when you compile them, you will find errors in your code, simply because we used TypeScript. We did not need anything else.

We did well, but we strive to become even better.

Our goal is to achieve a size of 10 kb gzip. The size of some of our prototypes is 25 kb and I have some ideas on how to make the compiler even smaller. Further, we want to become even faster than our Baseline starting point. We are aware that this is not the limit, and we can be even faster than Baseline.

You will not notice any changes. You will make changes, generate different code. The only thing - you will see the difference only in speed.

In conclusion

We talked about performance, about various aspects of performance.
We discussed input.
We learned what analysis is, ASD (AST). I talked about how we can present a template and how to implement it.
We learned that document.createElement is great for the Angular framework.
We learned that quick access by properties and hidden classes are great tools for optimizing your code.
We learned that code generation (if done correctly, that is, not only with the help of evaluation, but also with offline support, as well as the derivation of ASD) is a great and powerful thing. This will help you generate hidden classes.
We talked about static (Ahead Of Time) and dynamic (Just In Time) compilation, as well as things that can be optimized.
And finally, we looked at a good example. There is a link to my presentation on the slide.

Thank you so much for your attention.