May 18, 2017 at 14:01How and why to create a NginX-module - theory, practice, profit
Vasily Soshnikov ( Mail.Ru )
Today I will tell you about how to create nginx modules and, most importantly, try to answer why this should be done. This is not always necessary, but there is a certain range of tasks that can be solved on the nginx side.
Here is our brief outline. First, I’ll get you up to speed on the nginx architecture. Second, I’ll try to immediately answer some frequently asked questions, because very often people working at Mail.ru come up to me and ask the same questions, and I decided to simply put them in a small FAQ. Right away so that there is a better understanding of what will happen next.
The most interesting part is the anatomy. I’ll talk about how to create nginx modules, what types are there, a little bit about pitfalls, but without all the subtleties, because, you know, nginx is a very complex technology, there are 1000 nuances, 1000 subtleties. And I’ll try to answer the questions: “Why create them ?,“ Why do we create them in Mail.ru? ” and “Why did I create them, working not in Mail.ru?”.
Also last night I decided to write examples, literally templates of nginx-modules, which you can just take and use. In the end I will give a reference. Because my comrades told me that without this, my presentation is incomplete.
Let's start with the “Introduction” part and frequently asked questions.
The first is nginx architecture. I always associate it with a nested doll. In fact, the way it is. There is a certain core in which the entire core API is located, from which you can dazzle everything you want. In this case, http was blinded, upstreams and also the script engine were blinded from http. And the mail engine that works with mail is also made on the basis of core. Modules that are already being written for http are made on http core. Those. you understand, this is such a nesting doll, there is a base - this is core, there are layers below, below. And moreover, the lower the layer, the usually more than any modules. You yourself understand that there are a lot of modules for http, and all of them are based on http core, and the capabilities of nginx.
And one of the most frequently asked questions is about the nginx memory model. Nginx uses pool. This must always be remembered. And you need to take the right pool. A lot of people make a frequent mistake and try to nail their buffers to the connection, and connection in keep-alive mode can live very long. You understand that such buffers will accumulate as long as the connection is alive, and nginx can simply leak. Therefore, you must always choose the right pool. If we just want to save some buffer for the life of the request, it’s more logical to nail it to the request, and not to the connection.
You also need to remember that these pools are periodically cleaned and they have their own life-times. They, in principle, are all quite logical, connection lives within the framework of the connection, request - within the framework of the request, the config is always alive and periodically dominated. There are also many other contexts, but about them a little further.
Also try to always use nginx locators inside nginx. An API reference is at the top. Why use them? a) they are faster; b) they are easier to debug if you use Valgrind for nginx. True, nginx buffers have not debased in life. I didn’t have to, because pool, if everything is done correctly and everything is calculated, works fine.
But there is such a situation, for example, I recently had it, when I had to use standard mallock, alloc, reallock. Why? Because in nginx there is no reallock. This is a big problem. If you have some kind of library that uses reallock, then you simply cannot replace these allocation functions in the library. Here, a good example - yajil. In principle, I figured out how to do this, if someone is familiar with pascal strings, you can simply allocate 4 bytes more and nail another size into this piece of memory. And to use the usual nginx'ovskiy allocation for realization.
And this is an example, this is nginx_conf_s. This is its internal structure. Here you can see just this link to the pool. In this case, if you select something as part of the config, you will live as long as the config is alive. And in nginx, the config always lives until you clean it with pens. But there are different configs, more about that a bit further. So we must remember, we must always look at the structure that you want to nail, and approximately understand where it lives. And after some time you will have all this on the machine, but first you need to follow this very clearly.
And now very briefly about what is in nginx, and how its structure in the repository is arranged. At the top of the API, and below - what is there. First, there are a lot of data structures in nginx, and try to use them precisely because: a) they already know about nginx pools; b) this is the most native way to use it with nginx data types, because in nginx almost all data types are either typedef system types or their own structures.
Also in nginx core you can find OS with API, naturally wrapped in a facade. This may come in handy if you are no longer writing your own http-module on nginx, but, say, some kind of your own TCP-module, this is also possible. Because http is made on the basis of core and it can be expanded, if desired, as you wish. There you can also find a state-machine that is responsible for polishing the file descriptor, and many other useful functions like logging, working with scripts and configs.
And now the most interesting and the most painful, or rather, the most unknown. This is how nginx modules are arranged, what types are. Since I am limited in time, I decided to talk about only three main ones. And then about the most probably interesting - it's about upstream. In short, because it really is a topic for a separate report.
Therefore, today we will consider only 3 main modules - these are Handlers, Filters and Proxies. Proxies in a nutshell. In fact, there are many other types of modules there, as I said, because http, in fact, is also a core-based module. But we will not go into such nuances, because it is a very long time and, again, this is a topic for a separate report.
Everything in nginx is subject to one pattern. This is the Chain of responsibility pattern. In other words, your request from the user passes through the http module, the http module starts some sequence - chain of modules. And if they all worked well, then the user will get a good answer, if not, then an error will result. The closest analogy is this line of code:
I thought it would be a perfect analogy, I hope I'm right. Does everyone understand this line? This is essentially a bash. We grab something, get the first line by the delimeter; make a sort and count words. The principle of operation of the nginx module is very similar to this. Those. if the pipe breaks, your whole chain will fly off, and you will get an error in the bash. Also there are signals, etc. Nginx has the same thing, so all nginx modules should be treated that way.
Also, if you dig a little deeper, you can find that in addition to being a chain, this chain is also divided into phases. There are a lot of phases in nginx, each phase is called at a certain point in time under certain conditions, all handlers inside the phase are passed, after which there is already a switch with the result of that phase to another phase. Basically, this figure illustrates this. We have a certain set of modules in the phase, go through, get the result, go to the next phase.
And here are the phases that are now available in nginx_http:
They are self-speaking, I think there are no questions here. Those. server_rewrite, config search, rewrite normal, content_phase.
And now let's get down to how modules are developed. It is to those things that must be done for each module.
First, any module in nginx starts with a config. Nginx has a specific way of naming such structures for your module and for any other modules, and even for internal nginx. It is to use such a naming convention, i.e. ngx_http shows that we are doing something for http now. Next is the name of your module - it can be anything. And accordingly, one of the config types - main, server, loc - and _conf.
By the way, we posted publicly available videos of the last five years of the conference of developers of high-loaded systems HighLoad ++ . Watch, study, share and subscribe to the YouTube channel .
Does everyone remember nginx_config? This one is coming from there. There is main - this is the most global scope, there is a server, a beta server, and there is a location. Location if - also consider location, only specific. In fact, these configs and these structures can merge at different stages, i.e. nothing prevents you from making this structure so that it is present in main, and in the server, and in location and in location if, nothing. All that is needed is to write one function and set a specific bitmask. More about this later.
Directives. There are directives in the nginx config. And, accordingly, when you are developing your module, you should understand that you have to give some kind of pen to administrators and users. In nginx there is a special structure for this - it is ngx_command_s. I post all the links to the API here, so you can just see later, there’s nothing complicated. And here, in fact, all that is present is a name. The name is how your directive will be presented in nginx config. This is a type, and it is that mask what this directive is. Those. it is present in location, in main conf, server conf. Callback exists to parse the value of this variable. There is a large set of ready-made callbacks. For example, if we use ngx_string, and we just need to save the line, then we should not implement this callback, and take it ready from nginx. And then a few system fields, I’ll talk about them on the next slide.
Suppose we create a module called MODULE_NAME. Now, we have a certain variable, a regular integer, and we want to transfer this integer to config. All we need to do is declare such a structure in this array. If we had many different variables, we would have many elements of this array. And now, as I said, this mask, here in this case, this bitmap says that one argument is accepted. Here is the fact that this directive may be present in the location of the config, the server config, and in the main config. If I still wanted to have a location if, I would have to add another piece of this mask.
Accordingly, this is a nginx function, it exists for parsing regular ints. This is specifically to save time so that you do not have to do this function each time. offsetof just talks about where this field is located in our structure, this is the good old C-shny hack. I will not focus on it, because, in fact, in this context it must be taken as given. Those. he does not carry additional semantic load.
And such an array ends with the usual ngx_null_command - a marker indicating that the array has run out. Because in nginx, almost everything you declare inside your module is static, and it is externalized somewhere there, because in nginx there are now two ways to develop modules - either compile the module with nginx or load the module, but more on that later. In this case, we must remember that for now everything will be static, and that you need a final marker, because it goes through this array to the end, to this marker.
Now, we created some directives, it all works fine for us, but now we need to install inside nginx, create the config and its merge. To do this, there is such a structure as ngx_http_module_t. It is also quite simple, there are several stages of installation, this is preconfiguration, this is before any postconfiguration occurs, as well as several other functions, more about them in the next slide.
For example, we want that what we announced earlier, our directive merges correctly on the location. All that needs to be done is to hammer two of our callbacks into that structure, which we ourselves declared inside our module, and using ngx_palloc to deploy our config. And make it merge. In nginx, if it is a primitive type that nginx understands, there is already a large set of functions to merge any variables. If this is your own type, say, you have some very tricky line that needs to be parched, say xml, and you want to write it in a separate structure, then you will need a function and a merge, and describe the variable’s set yourself and pass it to a callback. And at the stage of the merge config in this function, you can do some additional checks and say that such and such an error has occurred.
Everything, the config is flickering, we got the directives. Now the most interesting part is how do we tell nginx that our module has appeared? There is ngx_module_t for this - this is a special type. And, again, it is declared static inside our object, i.e. ordinary variable. You describe, convey the context, these commands that we have formed, and additional callbacks are also possible for more flexible management. Those. let's say if we want some specific action on an int master, on exit from our master, etc. Here, in principle, I think everything is clear too, there are no questions. Just extra control knobs. For example, why is this needed? Let's think, if our module used shared memory, we would certainly want to unlock it when exiting and killing a stream. For this, these pens exist - for additional actions.
Based on this point, I think everyone understood, or already guess how to create nginx-modules. All that needs to be done is, in fact, to describe several static variables and declare several callbacks and that’s all. And our module will pick up. But this is not all, now the most important nuance is how to add a handler to phases and how to make filters. Just to have our functions called for some events inside nginx.
And we’ll talk about this now. About handlers and filters. These are two different entities.
I'll get back to the phases just in case. Phases cause a certain point in time, cause a certain chain, end chain and go to the next phase, and so on, until all phases are completed, or the phase is not interrupted. And just handlers work with phases. Those. What is a handler? You can bind any function to any of these phases. And now I will show you how to do it.
As you remember, when declaring a module, we could pass additional pens. Let's say here on postconfiguration, i.e. when the directives have already been worked out for us, we got the variables, filled them, we must do it, if we want to bind some phase, add a function during initialization and, accordingly, describe in this function the following - what we get, as you see, the content of the phases from nginx core config, because it's all in http core lives. And just add your function there with a regular array_push. And that's it, we have this function called on the content.
In other words, what is phase content? When someone returns an answer from us, the content of the phases twitches, and we can do anything there, say, we can count the number of words in the response from the server or additionally compress it with some algorithm of our own. And every time someone returns an answer from us, whether it is a file, or something else, this function will be called. In it, we can already work with the http-request, pull out the data that we need, add something there, or, conversely, delete it. And so we can do with each phase. The main thing to remember is that all phases are called at different times. In other words, if you need to work with content, then you do not need to add your handler to the log phase. Because the log phase occurs when you need to record something, roughly speaking. There you can, on the contrary, collect some additional statistics, if you are interested in this phase. And in this phase, you need to process the content.
By the way, I remembered a good example. The last time I did the content phase, I had to generate a picture on the fly for the counter at top.mail.ru. Here is a good example. Those. I just didn’t do anything, I just generated this new picture depending on the query parameters, depending on the counter id, etc.
Filters This is a slightly different entity. What are filters? Remember our chain? Filters, in fact, is a simple, single-linked list of filters.
And all you need to do to add your handler to a specific filter is just to take the nginx static variable called top_header_filter, add your handler there and that's it. In this case, this is our header_filter, and your handler will be called every time at some point in time at the stage of processing the headers.
Good practical example. We need to set our cookies for the counter and check. All you need to do is add a new header filter and check the headers that arrive and somehow convert to a normal cookie. This is a good example of why this is used.
Another good example of why this is used. Everyone knows a directive like add_header, the standard nginx directive. She uses these headers.
Those. she can do anything with http headers. And the whole installation always looks like this. In fact, there is still a body, I forgot about him. He has the same ideology, the same list, but it works at the stage of content processing.
And the most interesting. Our structure is requvest. If you ever develop for nginx, this framework will be your best friend. You will often go to this code, perhaps you will bind something to them in order to quickly jump on it. She is huge, she needs to know. There are a lot of nuances. Therefore, if you want to develop something, get acquainted with this structure, there is nothing complicated there, you just need to read and understand it. It stores the connection in itself, the headers that came from the user, which will leave the user, it stores all the handler handlers, its own pool, which works on this request. She also stores the body. Moreover, when you need to subtract the entire body from the guarantee, or send it guaranteed ... because nginx pays all, in fact, it works all the time as a pipe, it may not accumulate to the end. If you need to send something exactly guaranteed, you need to call special functions and use a specific request. More precisely, the specific request structure that came to your handler.
And here we come to the most interesting and most difficult moment of nginx - this is chain buffer. In nginx, any work with any buffer is a chain buffer. This is essentially a singly linked list of buffers. It is very tricky, it can be in a file, it can be shared memory, it can be read only memory, it can be temporary memory. I thought that I won’t go into the chain buffer, because there are so many nuances. The main thing that you need to know about him is that your message can come, but in general the body, not in one piece, it can come in buffers, in small, small pieces. This is especially important when you are developing upstream. Because upstream you can get in 4K or byte in general, if you need to parse it all, you will need to do some tricky streaming parser that works with chains. I will give a reference in the examples in read me later, I threw off about two of these upstream ones, who would be interested in how to implement a stream parser inside nginx. Because there is still such an important rule - nginx cannot be blocked. You have one worker. Even if ThreadPool is there, it is rare for anyone to use it, if you have blocked this worker for a long time, you have lost a lot of rps. This, too, must always be remembered.
And back again, after this digression, to Chain buffer. Basically, the whole structure looks like here:
Ie we have a position, last, if this is a file - file_pos, tags and a bunch of flags. And, in principle, all these comments are now taken from nginx. They must be remembered. Imagine, in your filter or your handler, a piece of memory has come to anything that is marked that it cannot be modified, then you do not have the right to modify such content. You must copy it to yourself, change it as you want, and give it further. Therefore, these flags must be constantly checked.
Or another good example. Many of the modules I wrote do not work with chain buffer if there is a check in_file there. Because I wanted, there was such a requirement for this to work quickly. If, God forbid, nginx starts pulling from a file, from its cache or from somewhere else, then it all dies. Therefore, in this case, it is very logical to check the chain buffer for the fact that it is not in_file, if you do not need it, and write to the administrator in the log: "Friend, unscrew the buffer size, please." This is also a good structure with which you should make friends.
And here is an example, this is me from my upstream module.
I wrote 6 modules for nginx. Only one opensource, and I showed the code from it. What I talked about. Those. here, however, the upstream chain buffer is used, not the buffer request, but the logic is the same. What is simple to check is the need for flags and these flags need to somehow definitely respond. This is an example of code that now works for many. Here, in particular, tp_transcode is a streaming parser, which I did painfully for a long time.
So we got to the Proxies. About proxies, as I said, or upstream, or load-balancers, I do not know how to characterize them correctly, because they are all similar to the so-called implementation. Therefore, I’ll only tell you the idea of why this exists. But without details, because there are a lot of nuances.
Basically, what is upstream? This is a proxy pass. What does proxy pass do? In fact, all that he does is in the chain mode gives data to the backends, can balance them according to different rules. Nothing prevents you from fastening any protocol there. Suppose we have some kind of daemon that runs on a protobuff, nothing prevents nginx from balancing it with protocol conversion. Those. the input is http, https with some kind of json or even a raw protobuff. And you can convert it to a clean protobuf and give it to the application server or to any other protocol. This is the whole idea of upstream.
And, in fact, upstream is a ready-made API inside nginx, which allows you to: a) easily configure such key balancing, say, by some other criteria; b) make backups, i.e. in case all upstream fails, it will go to a different url. And a lot of different pens for upstream'ov. How many maximum fails can be, what timeouts, etc. And everything is available out of the box.
I didn’t manage to finish the example of how to do upstream, but I’ll finish it and put it on the link that I will give. I hope I clearly explained what upstream is and what can be done with it.
Also, in addition, if you do not need conversions to another protocol, you can use them for custom balancing. True, in my opinion, now this is not required, because there is OpenResty, and there is an excellent directivebalancer_by_lua there, i.e. balancing can also be written on lua with any custom logic, I don’t know, at least go to the radish, though it will all die under load, but this can be done. Dies, dies. Checked. I tried, my CPU shot up to the ceiling, possibly due to lua strings, I suspect because there is balancing on url there when I tried.
And the most painful question, tons of copies were broken on it, thousands of admins cried, thousands of developers swore at admins. This is deploaming.
At first I forgot to say about the assembly. Here's a new build method. In nginx, it has recently changed. I didn’t show the old one, and I didn’t write a word about him specifically. I just took from my repository what a typical configuration of this conf, nginx looks like to add a module. In fact, this is bash, or shell. It just describes a few variables on the shell and that’s it. You can even link C ++ runtime here instead of this static library if you want. There are no problems. Moreover, to be honest, I linked in nginx ++ runtime. There is no problem. The main thing is that the implementation in nginx does not penetrate from the pros, because nginx will blow it into cabbage soup. Because C is not very friendly with plus actions. Here we simply describe what and where lies.
And then we simply fill in with these variables, as I wrote above, nginx variables, and call the auto module. The car module is located inside the nginx repository, where you can see it. This is a new build method and you need to use it, because the old build method will not be available soon, as soon as everyone moves to the fresh nginx.
Delivery. Here it is a sore subject, and I have been waiting for it for a long time and hard.
For probably three years, all the time we have been just building our nginx with a bunch of modules and just deploying it. The main thing is separate from the system nginx paths. This is very important, because some sleepy admin wants to install nginx with the status, writes apt-get install nginx and completely puts your server. Therefore, it is very important.
Plus we have several projects where there are several completely different nginx modules. It happened because they are different projects inside mail.ru. And I don’t know why this is not to be combined, there are no contradictions, it just happened. And we got to the point that we simply split up by the name of the project in the way where we put it all. This is pain and suffering, but now it is.
The second way, which appeared relatively recently, it is true, so far non-working - in nginx you can load SO'shki. Why is he non-working? Because nginx checks the fingerprint of this binary binary. And for it to accept this module, you have a large number of nginx assembly flags and your module must match.
Now I have come up with a way to solve this. Here we have, let's say, the Debian system, and there we can pull out the information with which flags nginx was built. And nothing prevents its module from being assembled with the same flags, and then packed. I haven’t tried to put it on the rails yet, but now I’m thinking in that direction, because loading modules is easier. But now it’s more painful, because there is a problem with fingerprint. It seems like the guys are trying to solve it, according to their wiki.
And the second option, it is less popular, at least inside mail.ru, maybe in reality it is popular. This is Docker. People take the usual nginx, the usual paths and just make a docker image from it, put it on the server and run it there. I don’t know how widespread it is, but people use it with us.
And now, probably, the most provocative thing - why create them?
The very answer to the forehead is because new technologies always appear, new Wishlist. If you want to add your part to the nginx contribution, albeit from the side, you can add some cool feature to nginx or introduce new technology inside nginx. Let's say http 3.0 appears, it will be a new module.
Here, this is my argument, which often ended with many disputes "Why do I need a nginx-module?". This is what many use - nginx ReverseProxy - to balance in some pluses that all they do is tell nginx or the client how to work. Those. in such a situation, it is more logical when there are no states, nothing, instead of ReverseProxy, to write a nginx-module, especially in highly loaded projects. Of course, if your application stores some states, this is already harder, but, again, here you need to look at the task. And so we moved on to solving business problems.
Here is a typical example, these were my 3 penultimate modules for top mail.ru. Before that, we had a regular http server written there by hand, no one knows when it was written, it was almost impossible to maintain it, it was constantly breaking down, there were problems, etc. And there was a strong-willed decision - just take and rewrite all the logic on nginx. Make some modules. The module that distributes cookies, the module that generates the JS file that gives the counter, and two additional image modules special for this project. 4 modules even worked out.
What is he doing? In fact, it receives this data from the user, forms a line and logs to the disk asynchronously. After that, these logs are raised by the counter and calculated. When I made these modules on nginx, we got better on the CPU, as it turned out. Finally, SSL started up correctly, because before that it worked incorrectly. It’s hard to do kosher SSL inside your application. It hurts. Especially check it out.
So, a good example, statistics analytics. Fits perfectly. There are no states, there are cookies, all of your user states are usually stored. There is some kind of atomic increment, if you have such logic, it is quite simple to do.
Advertising systems. Generally a perfect case. Here either OpenResty, or write directly on the nginx-module, depends on your needs. Because this is a classic of the genre, I won’t even add it there. There are almost no states there at all, you need some kind of map or shared memory between nginx and your daemon, which will have all the connections. In general, an ideal case. But I'm not sure about OpenResty, to be honest.
And protocol conversion. We admit, people, we all have http, we all have browsers, etc. And when we write heaps of the binary protocol, it all results in us that somewhere there is http, then why do we need extra layers? You can just write upstream, which will convert http to the X protocol. Everything is quite simple and in fact it is not so difficult. This is actually my work in the evenings for 2 weeks. I think it will take about the same for you. Moreover, it was still a difficult protocol with strange handshakes.
And here is the linkfor the examples that I promised, there are 3 examples, a few links to a couple of upstream'ov. There is a very primitive code, literally in 100 lines of code, there everything is what I talked about, plus it does additional logic. For example, Word Body Body Count when filtering, replacing content, etc. Those. it can be copied and even indulged. There is a special make file, instructions on how to assemble it are described there. I will try to expand this and write a comment in the code.
Contacts
→ github
Этот доклад — расшифровка одного из лучших выступлений на профессиональной конференции разработчиков высоконагруженных систем Highload++, а конкретно — секции «Бэкенд».
На наш взгляд — отличным дополнением этого материала будет доклад "Проксирование HTTP-запросов web-акселератором", который смогут услышать через три недели посетители HighLoad++ Junior и прочитать читатели Хабра через несколько месяцев.
- How does HTTP proxying without cache work;
What are persistent connections and how do they differ from HTTP keep alive;
How, when and how many connections an HTTP accelerator with upstream can establish;
What happens to requests that are waiting for the queue to be sent to connect to the upstream, but the upstream is out of the box and drops connections every 100 requests;
What is HTTP pipelining, and how is it used by modern HTTP accelerators;
What are non-idempotent requests, and why you need to worry about them.
