Lapis: a site on Lua in Nginx configs

Tl; dr Lapis (Lua) = RoR (Ruby) = Django (Python)

Introduction

Lua is a powerful and fast scripting language that integrates very easily in C. Developed in PUC-Rio (Brazil).


LuaJIT is the fastest implementation of Lua (the JIT compiler), a real work of art. According to some estimates , it has a six-fold advantage over the standard Lua interpreter and in many tests it beats V8. Developer Mike Pall (Germany).

And LuaJIT can bind C functions and structures on the Lua side (without writing C bindings):

local ffi = require("ffi")
ffi.cdef[[int printf(const char *fmt, ...);]]
ffi.C.printf("Hello %s!\n", "wiki")

Nginx is one of the most efficient web servers developed by Igor Sysoev. Many large sites use Nginx. Starting with version 0.8, it became possible to directly embed the Lua language. Lua is executed in the Nginx process itself, and is not put into a separate process, as is the case with other languages. Lua code in the Nginx context runs in non-blocking mode, including database queries and external HTTP requests generated by a web application (for example, a request to the API of another site).


OpenResty is a Nginx assembly with many third-party modules, including for non-blocking access to popular databases. Recent versions use LuaJIT to execute Lua. Developer Yichun Zhang (USA, place of work: CloudFlare, main developer of lua-nginx-module).


SailorMoonScript is a scripting language that translates to Lua. Adds syntactic sugar, simplifies the spelling of some things, such as list expressions; implements classes with inheritance. You could say that MoonScript for Lua is CoffeeScript for JavaScript. Developer Leaf Corcoran (USA).


Lapis is a web framework for writing web applications in Lua and MoonScript, which lives inside OpenResty. Developer Leaf Corcoran (USA).

What advantage does Lua have in Nginx?

Tl; dr All the features of a high-level language and efficient use of resources under heavy loads.

For an answer, let us return to the distant past when all sites were served by the Apache web server.

The delays are introduced by the red nodes and edges of the graph. Yellow shaded components located on the same machine.

Apache allocated a separate thread of the operating system, which read the request, processed and sent the result to the user. (Modern Apache can be taught not to do this.) It turns out how many active requests, so many OS threads, and they are expensive. Most of the flow lifetime with such a scheme is spent not on request processing, but on data transmission over the network, limited by the Internet speed of the user.


How to deal with it? We must instruct the operating system to monitor the data transfer so that our web server can work only when the network has completed the next task. This approach is called non-blocking input-output and is implemented in most modern operating systems. The Nginx web server uses this feature, due to which it can serve tens of thousands of simultaneous requests using only one OS thread.


Thus, we optimized the data transfer between the browser and the web server, but there is another bottleneck where OS threads are idle: working with the database and external resources (for example, the HTTP-API of another site). It is important to understand that this is not so much about the inevitable delays of the database itself or the external API, but that our web application is idlely idle until it receives an answer from them.

The usual solution : in the web application itself, spawn the streams that we so successfully got rid of in the web server. Effective solution: Make the web application and database communicate in a non-blocking way. The web application sends the request to the database and immediately proceeds to the next request from the user. The database considers it returns the result, and the web application, when released, returns to processing the request from the user who generated this query to the database. This approach is used, for example, in node.js: the

database and external APIs are still painted red, as they can introduce a delay. The advantage of the approach is that the web application does not just wait for them, but processes other requests at this time.

Wonderful! Now let's see how external HTTP requests are programmed in node.js:

var request = require("request");
request.get("http://www.whatever.com/my.csv", function (error, response, body) {
    if (!error && response.statusCode == 200) {
        console.log("Got body: " + body);
    }
});

Suppose we want to download a file by URL and do something with it. The result has to be processed in lambda functions. Inconvenient? Is this an inevitable asynchronous fee? Fortunately, this is not so; let's see a similar code in Lapis:

local http = require("lapis.nginx.http")
local body, status_code, headers = http.simple("http://www.whatever.com/my.csv")
if status_code == 200 then
    print(body)
end

The code for Lapis is convenient to write, as if it is synchronous, but behind the scenes it runs completely asynchronously. This is possible due to the active use of coroutines ( coroutines , green threads , and in Lua terminology simply threads ). All code that processes the request from the user is executed in a separate coroutine, and coroutines can stop and continue in certain places. In our example, this place was inside the http.simple function call .

Why are coroutines more efficient than OS threads? Have we dragged all the overhead into the app? In fact, the key difference between coroutines and OS threads is the freedom of the programmer, in which place the coroutine falls asleep and wakes up. (In the case of OS threads, the decision is made by the OS.) Started a query to the database - euthanized the coroutine that generated the query. An answer came from the database - we wake up the coroutine and continue its execution. We carry out many tasks at the same time and all in one OS thread!

Note. A similar mechanism is about to appear in node.js.

Note. I advise you to read a wonderful articleabout coroutines in the context of C ++. At the end of the article, an asynchronous code was written, written as synchronous, and all thanks to coroutines. It is a pity that in C ++ coroutines are more likely to be a hack than a generally accepted technique.

In addition, Lapis runs directly in Nginx, which eliminates the overhead of transferring information between Nginx and the web application. Of course, node.js can be used as the main web server, without Nginx, but then the opportunity to use different features of Nginx disappears.



On the other hand, not everyone decides to run Lua code directly in the main Nginx. In this case, we start a separate Nginx with Lua on behalf of an individual user with truncated rights, and basically we prescribe Nginx proxies.



Lapis Efficiency Confirmed in 10 Gigabit Benchmark. Lapis takes the leading places at the level of languages ​​C ++ and Java.

Lapis

On April 1, 2014, the April Fools' article “LUA in nginx: inline php style noodles” was published on Habré . This article looked at comic code that implements PHP-like templates in Lua. In the comments to the same article, Lapis was mentioned . I did not find other mentions of Lapis on Habré, so I decided to write it myself.

Writing Hello World is boring. Instead, let's write a simple web proxy on Lapis.

Install OpenResty

Install perl 5.6.1+, libreadline, libpcre and libssl and make sure the ldconfig command is available (its parent folder may not be in PATH).

$ wget http://openresty.org/download/ngx_openresty-1.7.4.1.tar.gz
$ tar xzvf ngx_openresty-1.7.4.1.tar.gz
$ cd ngx_openresty-1.7.4.1/
$ ./configure
$ make
# make install

Install Lapis

First you need to install LuaRocks (available in the main distributions).

# luarocks install lapis

Create a web application

Create the backbone of the site:

$ lapis new --lua
wrote	nginx.conf
wrote	mime.types
wrote	app.lua

If we did not pass the --lua option, then a core in MoonScript would be created.

Now we implement the logic of our application in app.lua: the form for entering the URL is displayed on the main page of the site. The form is sent to / geturl, where the page is loaded at the specified URL and the content is transferred to the user's browser.

local lapis = require("lapis")
local app = lapis.Application()
local http = require("lapis.nginx.http")
app:get("/", function(self)
  return [[
    

    ]]
end)
app:post("/geturl", function(self)
  local url = self.req.params_post.url
  local body, status_code, headers = http.simple(url)
  return body
end)
return app

The main page simply displays the HTML code with the form. Double square brackets are another notation for strings in Lua. The / geturl page receives a POST request from the form, takes the URL entered by the user into the form from it, downloads the contents of this URL using the http.simple function (the OS stream is not blocked, see above) and shows the result to the user.

For http.simple to work, you need to change nginx.conf:

    location / {
      set $_url "";
      default_type text/html;
      content_by_lua '
        require("lapis").serve("app")
      ';
    }
    location /proxy {
      internal;
      rewrite_by_lua '
        local req = ngx.req
        for k,v in pairs(req.get_headers()) do
          if k ~= "content-length" then
            req.clear_header(k)
          end
        end
        if ngx.ctx.headers then
          for k,v in pairs(ngx.ctx.headers) do
            req.set_header(k, v)
          end
        end
      ';
      resolver 8.8.8.8;
      proxy_http_version 1.1;
      proxy_pass $_url;
    }

This code creates in Nginx a location / proxy through which Lua makes external requests. In the main location you need to add set $ _url ""; Read more about this in the documentation .

Run our web proxy:

$ lapis server

Click on the “Get” button. The ip4.me site shows the IP address of the server running Lapis.



If there is no path in the URL, then / proxy is used as the path. Apparently, this is a bug of Lapis, on which a report has already been compiled .

Conclusion

Lapis, Lua and Nginx have a lot of interesting things, for example, asynchronous work with Postgres databases , wrapper classes for database objects, HTML generation , etlua powerful template language , caching variables between different Nginx working processes, protection against CSRF , two tools for testing and interactive Lua-console directly in the browser. If the article finds a reader, I will continue the story about Lapis in other articles.

Without a doubt, Lapis has long outgrown the level of an April Fools' joke and is rapidly gaining ground in the web development community. I wish you a pleasant study of these promising technologies!

References

• Lapis documentation (the menu is hidden on the left, and on top there is a switch between Lua and MoonScript)
• a selection of good links on Lua (projects + tutorials)
• HttpLuaModule documentation
• MoonScript language
• opinions on using coroutines on the Web: pessimistic , optimistic
• article source codes available on github