Get the IP addresses of HTTPS clients from HAProxy (frontend) to Nginx (backend) in HTTP and TCP balancing modes

Quite often, load balancing between multiple web servers is required. In this case, as a rule, it is necessary that web applications receive real IP addresses of clients, and not IP balancer.

In the case of balancing and terminating HTTP (S) traffic on HAProxy (Layer 7 [ 1 ]), this task is easily solved by adding the “X-Real-IP” header and processing it on Nginx using the ngx_http_realip_module [ 2 ] module . When balancing TCP traffic from HTTPS clients and transferring it to web servers directly without modification or termination (Layer 4 [ 3 ]), adding this header is impossible, therefore, you need to take advantage of the features provided by Proxy Protocol [ 4 , 5 , 6 ].

Let's consider both options (balancing L7 and L4) using excerpts from the configuration files haproxy 1.5.9 and nginx 1.6.2 as an example.

Balancing at the application level (Layer 7): terminating HTTPS traffic on HAProxy and transmitting via HTTPS to Nginx

In this example, HTTPS traffic from the client is terminated on HAProxy, modified and transmitted to Nginx via HTTPS as well.

haproxy.cfg

global
        maxconn 4096
        chroot /usr/share/haproxy
        uid 99
        gid 99
        daemon
        tune.ssl.default-dh-param  2048
defaults
        log     global
        option  redispatch
        option  tcp-smart-accept
        option  tcp-smart-connect
        retries 3
        maxconn 2000
        timeout connect 5000
        timeout check   3000
        timeout client  50000
        timeout server  50000
frontend http_frontend *:80
        mode http
        redirect scheme https code 301 if !{ ssl_fc }
frontend https_frontend_ssl_terminate
        mode http
        bind *:443 ssl crt /etc/haproxy/ssl/public.example.com.pem
        option forwardfor header X-Real-IP
        default_backend web_server_http
backend web_server_http
        mode http
        balance roundrobin
        # Отправляем трафик на backend по HTTPS
        server s1_https 192.168.1.10:443 ssl verify none
        server s2_https 192.168.1.20:443 ssl verify none

nginx.conf

server {
    server_name localhost;
    listen 443 ssl default_server;
    ssl_certificate      /etc/nginx/ssl/internal.example.com.pem;
    ssl_certificate_key  /etc/nginx/ssl/internal.example.com.key;
    # Адрес HAProxy
    set_real_ip_from 192.168.1.254;
    real_ip_header X-Real-IP;
    root /usr/share/nginx/html;
    index index.html index.htm;
    error_page 500 502 503 504 /50x.html;
    location = /50x.html {
        root /usr/share/nginx/html;
    }
    location ~ /\.ht {
        deny all;
    }
}

Transport-Level Balancing (Layer 4): Transfer TCP Traffic from HAProxy to Nginx

In this example, the client’s HTTPS traffic is not modified (HAProxy interferes with the transport layer) and its termination occurs directly on Nginx.

haproxy.cfg

global
        maxconn 4096
        chroot /usr/share/haproxy
        uid 99
        gid 99
        daemon
defaults
        log     global
        option  redispatch
        option  tcp-smart-accept
        option  tcp-smart-connect
        retries 3
        maxconn 2000
        timeout connect 5000
        timeout check   3000
        timeout client  50000
        timeout server  50000
frontend http_frontend *:80
        mode http
        redirect scheme https code 301 if !{ ssl_fc }
frontend https_frontend_ssl_pass
        mode tcp
        bind *:443
        default_backend web_server_tcp
backend web_server_tcp
        mode tcp
        balance roundrobin
        # ВНИМАНИЕ! Работа с send-proxy возможна только,
        # когда принимающая сторона понимает, что это такое.
        # Для Nginx необходимо включить в директиву listen
        # опцию proxy_protocol.
        server s1_tcp 192.168.1.10:443 send-proxy
        server s2_tcp 192.168.1.20:443 send-proxy

nginx.conf

server {
    server_name localhost;
    # ВНИМАНИЕ! Работа с директивой proxy_protocol возможна только в связке с haproxy.
    # Для прямого доступа данную директиву необходимо отключить.
    listen 443 ssl default_server proxy_protocol;
    ssl_certificate      /etc/nginx/ssl/public.example.com.pem;
    ssl_certificate_key  /etc/nginx/ssl/public.example.com.key;
    # Адрес HAProxy
    set_real_ip_from 192.168.1.254;
    real_ip_header proxy_protocol;
    root /usr/share/nginx/html;
    index index.html index.htm;
    error_page 500 502 503 504 /50x.html;
    location = /50x.html {
        root /usr/share/nginx/html;
    }
    location ~ /\.ht {
        deny all;
    }
}

Conclusion

Using the settings described above, we were able to transfer the real IP addresses of clients when working via HTTPS to the Nginx web server located behind HAProxy. A similar approach can also be used when working with third-party load balancers, such as CloudFlare [ 7 , 8 ] and AWS ELB [ 9 , 10 ].

Literature

1. OSI Network Model Application Layer Protocols - en.wikipedia.org
2. Ngx_http_realip_module module - nginx.org
3. OSI Network Transport Layer - wikipedia.org
4. The PROXY protocol - haproxy.org
5. HAProxy Configuration Manual: send-proxy - cbonte.github.io
6. Ngx_http_core_module module: listen directive - nginx.org
7. Getting Real IP Addresses Using CloudFlare, Nginx, and Varnish - danielmiessler.com
8. Getting Real IP Addresses Using Nginx and CloudFlare - babaei.net
9. Using Proxy Protocol With Nginx - chrislea.com
10. AWS Elastic Load Balancing - aws.amazon.com