Nginx on steroids - expanding functionality with LUA

To ensure the operation of all our external products, we use the popular nginx. It is fast and reliable. There are almost no problems with it. Our products are also constantly evolving, new services appear, new functionality is added, the old one expands. The audience and the load is only growing. Now we want to talk about how we accelerated development, notably increased productivity and made it easier to add this new functionality to our services, while maintaining the availability and fault tolerance of the affected applications. It will be about the concept of “nginx as web application”.
Namely, about third-party modules (mainly LUA) that allow you to do completely magical things quickly and reliably.

Problems and solution

The basic idea is quite simple. Take the following factors:
- the complexity of the application logic,
- the number of application components,
- the size of the audience.
From a certain point, it becomes quite difficult to keep the application responsive and fast, sometimes even functional. The product becomes multi-component, geographically distributed. And more and more people are using it. At the same time, there are business requirements for responsiveness and fault tolerance, which must first be observed.
There are several ways to solve this problem. You can break everything and remake it with other technologies. Of course, this option works, but we did not really like it and we decided to redo it gradually. Openresty assembly was taken as a basis(nginx + LUA). Why LUA. Without the help of cgi, fastcgi and other cgi, powerful, beautiful and fast functionality can be scripted directly in the nginx configuration file. Everything works asynchronously. And not only with customers, but also with backends. At the same time, without interfering in the event loop of the web server, without callbacks, fully using the existing nginx functionality.

Currently, the following backends are available:
- Redis
- Memcache
- MySQL
- PostgreSQL.
In addition, you can also connect modules for use, for example, RabbitMQ and ZeroMQ .
It works pretty fast. Anyway, faster than php-fpm))

The logical question is, why not rewrite everything at all in C? Writing on LUA is much easier and faster. And we are immediately spared the problems associated with asynchrony and nginx event loop.

Examples. Ideas

We, as usual, will not provide the full code, only the main parts. All of these things were done in php before.

1. This part was invented and made by our colleague AotD . There is a repository of pictures. They need to be shown to users, and it is desirable to perform some operations at the same time, for example, resize. We store pictures in ceph, it is an analog of Amazon S3. ImageMagick is used for image processing. There is a cache directory on the resizer, processed pictures are added there.
We parse the user's request, we determine the picture, the resolution he needs and go to ceph, then we process and show it on the fly.
serve_image.lua

require "config"
local function return_not_found(msg)
    ngx.status = ngx.HTTP_NOT_FOUND
    if msg then
        ngx.header["X-Message"] = msg
    end
    ngx.exit(0)
end
local name, size, ext = ngx.var.name, ngx.var.size, ngx.var.ext
if not size or size == '' then
    return_not_found()
end
if not image_scales[size] then
    return_not_found('Unexpected image scale')
end
local cache_dir =  static_storage_path .. '/' .. ngx.var.first .. '/' .. ngx.var.second .. '/'
local original_fname = cache_dir .. name .. ext
local dest_fname = cache_dir .. name .. size .. ext
-- make sure the file exists
local file = io.open(original_fname)
if not file then
    -- download file contents from ceph
    ngx.req.read_body()
    local data = ngx.location.capture("/ceph_loader", {vars = { name = name .. ext }})
    if data.status == ngx.HTTP_OK and data.body:len()>0 then
        os.execute( "mkdir -p " .. cache_dir )
        local original = io.open(original_fname, "w")
        original:write(data.body)
        original:close()
    else
        return_not_found('Original returned ' .. data.status)
    end
end
local magick = require("imagick")                                                                                                                                                                                                 
magick.thumb(original_fname, image_scales[size], dest_fname)                                                                                                                                                                     
ngx.exec("@after_resize")

We connect imagic.lua binding . Must be accessible LuaJIT .

nginx_partial_resizer.conf.template

# Old images
location ~ ^/(small|small_wide|medium|big|mobile|scaled|original|iphone_(preview|retina_preview|big|retina_big|small|retina_small))_ {
    rewrite /([^/]+)$ /__CEPH_BUCKET__/$1 break;
    proxy_pass __UPSTREAM__;
}
# Try get image from ceph, then from local cache, then from scaled by lua original
# If image test.png is original, when user wants test_30x30.png:
# 1) Try get it from ceph, if not exists
# 2) Try get it from /cache/t/es/test_30x30.ong, if not exists
# 3) Resize original test.png and put it in /cache/t/es/test_30x30.ong
location ~ ^/(?(?.)(?..)[^_]+)((?_[^.]+)|)(?\.[a-zA-Z]*)$ {
    proxy_intercept_errors on;
    rewrite /([^/]+)$ /__CEPH_BUCKET__/$1 break;
    proxy_pass __UPSTREAM__;
    error_page 404 403 = @local;
}
# Helper failover location for upper command cause you can't write
# try_files __UPSTREAM__ /cache/$uri @resizer =404;
location @local {
    try_files /cache/$first/$second/$name$size$ext @resize;
}
# If scaled file not found in local cache resize it with lua magic!
location @resize {
#    lua_code_cache off;
    content_by_lua_file "__APP_DIR__/lua/serve_image.lua";
}
# serve scaled file, invoked in @resizer serve_image.lua
location @after_resize {
    try_files /cache/$first/$second/$name$size$ext =404;
}
# used in @resizer serve_image.lua to download original image
# $name contains original image file name
location =/ceph_loader {
    internal;
    rewrite ^(.+)$ /__CEPH_BUCKET__/$name break;
    proxy_set_header Cache-Control no-cache;
    proxy_set_header If-Modified-Since "";
    proxy_set_header If-None-Match "";
    proxy_pass __UPSTREAM__;
}
location =/favicon.ico {
    return 404;
}
location =/robots.txt {}

2. Firewall for the API. Request validation, customer identification, rps control and a barrier for those we do not need.
Firewall.lua

module(..., package.seeall);
local function ban(type, element)
    CStorage.banPermanent:set(type .. '__' .. element, 1);
    ngx.location.capture('/postgres_ban', { ['vars'] = { ['type'] = type, ['value'] = element} });
end
local function checkBanned(apiKey)
    -- init search criteria
    local searchCriteria = {};
    searchCriteria['key'] = apiKey;
    if ngx.var.remote_addr then
        searchCriteria['ip'] = ngx.var.remote_addr;
    end;
    -- search in ban lists
    for type, item in pairs(searchCriteria) do
        local storageKey = type .. '__' .. item;
        if CStorage.banPermanent:get(storageKey) then
            ngx.exit(444);
        elseif CStorage.banTmp:get(storageKey) then
            -- calculate rps and check is our client still bad boy 8-)
            local rps = CStorage.RPS:incr(storageKey, 1);
            if not(rps) then
                CStorage.RPS:set(storageKey, 1, 1);
                rps=1;
            end;
            if rps then
                if rps > config.app_params['ban_params']['rps_for_ip_to_permanent_ban'] then
                    CStorage.RPS:delete(storageKey);
                    ban(type, item);
                    ngx.exit(444);
                elseif config.app_params['ban_params']['rps_for_ip_to_tmp_ban'] > 0 and rps == config.app_params['ban_params']['rps_for_ip_to_tmp_ban'] then
                    local attemptsCount = CStorage.banTmp:incr(storageKey, 1) - 1;
                    if attemptsCount > config.app_params['ban_params']['tmp_ban']['max_attempt_to_exceed_rps'] then
                        -- permanent ban
                        CStorage.banTmp:delete(storageKey);
                        ban(type, item);
                    end;
                end;
            end;
            ngx.exit(444);
        end;
    end;
end;
local function checkTemporaryBlocked(apiKey)
    local blockedData = CStorage.tmpBlockedDemoKeys:get(apiKey);
    if blockedData then
        --storage.tmpBlockedDemoKeys:incr(apiKey, 1); -- think about it.
        return CApiException.throw('tmpDemoBlocked');
    end;
end;
local function checkRPS(apiKey)
    local rps = nil;
    -- check rps for IP and ban it if it's needed
    if ngx.var.remote_addr then
        local ip = 'ip__' .. tostring(ngx.var.remote_addr);
        rps = CStorage.RPS:incr(ip, 1);
        if not(rps) then
            CStorage.RPS:set(ip, 1, 1);
            rps = 1;
        end;
        if rps > config.app_params['ban_params']['rps_for_ip_to_permanent_ban'] then
            ban('ip', tostring(ngx.var.remote_addr));
            ngx.exit(444);
        elseif config.app_params['ban_params']['rps_for_ip_to_tmp_ban'] > 0 and rps > config.app_params['ban_params']['rps_for_ip_to_tmp_ban'] then
            CStorage.banTmp:set(ip, 1, config.app_params['ban_params']['tmp_ban']['time']);
            ngx.exit(444);
        end;
    end;
    local apiKey_key_storage = 'key_' .. apiKey['key'];
    -- check rps for key
    rps = CStorage.RPS:incr(apiKey_key_storage, 1);
    if not(rps) then
        CStorage.RPS:set(apiKey_key_storage, 1, 1);
        rps = 1;
    end;
    if apiKey['max_rps'] and rps > tonumber(apiKey['max_rps']) then
        if apiKey['mode'] == 'demo' then
            CApiKey.blockTemporary(apiKey['key']);
            return CApiException.throw('tmpDemoBlocked');
        else
            CApiKey.block(apiKey['key']);
            return CApiException.throw('blocked');
        end;
    end;
    -- similar check requests per period (RPP) for key
    if apiKey['max_request_count_per_period'] and apiKey['period_length'] then
        local rpp = CStorage.RPP:incr(apiKey_key_storage, 1);
        if not(rpp) then
            CStorage.RPP:set(apiKey_key_storage, 1, tonumber(apiKey['period_length']));
            rpp = 1;
        end;
        if rpp > tonumber(apiKey['max_request_count_per_period']) then
            if apiKey['mode'] == 'demo' then
                CApiKey.blockTemporary(apiKey['key']);
                return CApiException.throw('tmpDemoBlocked');
            else
                CApiKey.block(apiKey['key']);
                return CApiException.throw('blocked');
            end;
        end;
    end;
end;
function run()
    local apiKey = ngx.ctx.REQUEST['key'];
    if not(apiKey) then
        return CApiException.throw('unauthorized');
    end;
    apiKey = tostring(apiKey)
    -- check permanent and temporary banned
    checkBanned(apiKey);
    -- check api key
    apiKey = CApiKey.getData(apiKey);
    if not(apiKey) then
        return CApiException.throw('forbidden');
    end;
    apiKey = JSON:decode(apiKey);
    if not(apiKey['is_active']) then
        return CApiException.throw('blocked');
    end;
    apiKey['key'] = tostring(apiKey['key']);
    -- check is key in tmp blocked list
    if apiKey['mode'] == 'demo' then
        checkTemporaryBlocked(apiKey['key']);
    end;
    -- check requests count per second and per period
    checkRPS(apiKey);
    -- set apiKey's json to global parameter; in index.lua we send it through nginx to php application
    ngx.ctx.GLOBAL['api_key'] = JSON:encode(apiKey);
end;

Validator.lua

module(..., package.seeall);
local function checkApiVersion()
    local apiVersion = '';
    if not (ngx.ctx.REQUEST['version']) then
        local nginx_request = tostring(ngx.var.uri);
        local version = nginx_request:sub(2,4);
        if tonumber(version:sub(1,1)) and tonumber(version:sub(3,3)) then
            apiVersion = version;
        else
            return CApiException.throw('versionIsRequired');
        end;
    else
        apiVersion = ngx.ctx.REQUEST['version'];
    end;
    local isSupported = false;
    for i, version in pairs(config.app_params['supported_api_version']) do
        if apiVersion == version then
            isSupported = true;
        end;
    end;
    if not (isSupported) then
        CApiException.throw('unsupportedVersion');
    end;
    ngx.ctx.GLOBAL['api_version'] = apiVersion;
end;
local function checkKey()
    if not (ngx.ctx.REQUEST['key']) then
        CApiException.throw('unauthorized');
    end;
end;
function run()
    checkApiVersion();
    checkKey();
end;

Apikey.lua

module ( ..., package.seeall )
function init()
    if not(ngx.ctx.GLOBAL['CApiKey']) then
        ngx.ctx.GLOBAL['CApiKey'] = {};
    end
end;
function flush()
    CStorage.apiKey:flush_all();
    CStorage.apiKey:flush_expired();
end;
function load()
    local dbError = nil;
    local dbData = ngx.location.capture('/postgres_get_keys');
    dbData = dbData.body;
    dbData, dbError = rdsParser.parse(dbData);
    if dbData ~= nil then
        local rows = dbData.resultset
        if rows then
            for i, row in ipairs(rows) do
                local cacheKeyData = {};
                for col, val in pairs(row) do
                    if val ~= rdsParser.null then
                        cacheKeyData[col] = val;
                    else
                        cacheKeyData[col] = nil;
                    end
                end
                CStorage.apiKey:set(tostring(cacheKeyData['key']),JSON:encode(cacheKeyData));
            end;
        end;
    end;
end;
function checkNotEmpty()
    if not(ngx.ctx.GLOBAL['CApiKey']['loaded']) then
        local cnt = CHelper.tablelength(CStorage.apiKey:get_keys(1));
        if cnt == 0 then
            load();
        end;
        ngx.ctx.GLOBAL['CApiKey']['loaded'] = 1;
    end;
end;
function getData(key)
    checkNotEmpty();
    return CStorage.apiKey:get(key);
end;
function getStatus(key)
        key = getData(key);
        local result = '';
        if key ~= nil then
            key = JSON:decode(key);
            if key['is_active'] ~= nil and  key['is_active'] == true then
                result = 'allowed';
            else
                result = 'blocked';
            end;
        else
            result = 'forbidden';
        end;
        return result;
end;
function blockTemporary(apiKey)
    apiKey = tostring(apiKey);
    local isset = getData(apiKey);
    if isset then
        CStorage.tmpBlockedDemoKeys:set(apiKey, 1, config.app_params['ban_params']['time_demo_apikey_block_tmp']);
    end;
end;
function block(apiKey)
    apiKey = tostring(apiKey);
    local keyData = getData(apiKey);
    if keyData then
        ngx.location.capture('/redis_get', { ['vars'] = { ['key'] = apiKey } });
        keyData['is_active'] = false;
        CStorage.apiKey:set(apiKey,JSON:encode(cacheKeyData));
    end;
end;

Storages.lua

module ( ..., package.seeall )
apiKey = ngx.shared.apiKey;
RPS = ngx.shared.RPS;
RPP = ngx.shared.RPP;
banPermanent = ngx.shared.banPermanent;
banTmp = ngx.shared.banTmp;
tmpBlockedDemoKeys = ngx.shared.tmpBlockedDemoKeys;

3. Additional services, such as inter-component communication via the AMQP protocol. An example is here .

4. As I already wrote . Application self-diagnosis module with the ability to “smartly” control the routes of the request through the backends. Still in development.

5. Adapters for APIs. In some cases, it is necessary to tweak, supplement or expand existing methods. In order not to rewrite everything, LUA will help . For example, json <-> xml conversion on the fly.

6 ... there are many more ideas.

There will be no benchmarks as such. The products are too complex and rps after bench is highly dependent on many factors. However, for our products, we achieved a 20-fold increase in performance for the affected functionality, and in some cases everything became faster up to ~ 200 times.

Pros and cons

Tangible pluses. Everything that used to be 5 megabytes of code in php turns into a 100kb file in lua.
- development speed,
- application speed,
- reliability,
- asynchronous work with clients and backends, without breaking event loop nginx,
- LUA sugar feel good! Coroutines, shared dictionary for all forks nginx, sub-quests, a bunch of binders.

Imperceptible cons.
- you need to do everything carefully and remember about asynchrony and event loop nginx.
- The frontend is so fast that the backend may not like it. There is a direct connection between them, without layers. For example, I’m sure that 10,000 requests per second LUA on the frontend will work easily. But, if at the same time it wants to go to the base, then problems may arise.
- It’s rather difficult to debug if something goes wrong.

By the way, while this article is being written, right at that moment our programmer talks about all this in detail on highload.

We will be happy to answer questions in the comments.

Finally, here you can find a small selection of information on the topic.