HTTP Latency Diagnostics: from Nginx to the Kernel

# Diagnosing HTTP Request Latency: From Load Balancer to Linux Kernel

Rising response times or sudden 5xx errors in production demand instant and precise problem localization. Instead of chaotically checking services, it's more effective to use a systematic approach that divides the entire request processing pipeline into network and application segments. We'll break down a method for quickly finding bottlenecks using standard Nginx logs, Server-Timing headers, and low-level TCP_INFO metrics.

Starting Point: Analyzing Load Balancer Logs

A typical web app architecture includes several layers: user, CDN, load balancer (SLB), application, connection pool, and database. Each can introduce latency, but the load balancer should always be your starting point for diagnostics—it's at the center of the pipeline and lets you instantly determine the search direction.

For Nginx, the key variables are $request_time and $upstream_response_time. The first records the full request processing time from receiving the first byte from the client to sending the last byte of the response. The second measures only the time waiting for a response from the upstream server. If these metrics aren't being collected yet, expand your logging format:

log_format timing '$remote_addr - $request_uri '
                  'status=$status '
                  'rt=$request_time '
                  'uct=$upstream_connect_time '
                  'urt=$upstream_response_time';

access_log /var/log/nginx/access.log timing;

Data interpretation is based on a simple comparison. If $upstream_response_time is close to $request_time, the bottleneck is to the right of the load balancer: in the app code, connection pool, or database. If $upstream_response_time is normal but $request_time is anomalously high, the issue is to the left: in the network infrastructure, CDN routing, or on the client side. Note that injecting $request_time into response headers via add_header isn't recommended for accurate measurements, as Nginx forms headers before completing response body transmission. Reliable data is only available in the access log after connection closure.

Pinpointing Application-Side Issues

When the load balancer points to upstream delays, you need to drill down into the app's internal operations. The W3C Server-Timing standard lets you send precise timings for individual request processing stages to the client or monitoring system. The header looks like: Server-Timing: app;dur=120, db;dur=95, pool-wait;dur=18. This provides transparency without deploying heavy APM agents.

Integrating the header into your app stack requires minimal changes. Implementation examples for popular languages:

Go (net/http):

start := time.Now()
rows, err := db.QueryContext(ctx, query)
dbDur := time.Since(start)

w.Header().Set("Server-Timing",
    fmt.Sprintf("db;dur=%.2f", float64(dbDur.Microseconds())/1000))

Python (Django middleware):

class ServerTimingMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        start = time.monotonic()
        response = self.get_response(request)
        dur = (time.monotonic() - start) * 1000
        response["Server-Timing"] = f"app;dur={dur:.2f}"
        return response

Node.js (Express):

app.use((req, res, next) => {
  const start = process.hrtime.bigint();
  res.on('finish', () => {
    const ms = Number(process.hrtime.bigint() - start) / 1e6;
    // already sent, but for logging:
    console.log(`Server-Timing: app;dur=${ms.toFixed(2)}`);
  });
  // for header — time up to response:
  const origEnd = res.end;
  res.end = function(...args) {
    const ms = Number(process.hrtime.bigint() - start) / 1e6;
    res.setHeader('Server-Timing', `app;dur=${ms.toFixed(2)}`);
    origEnd.apply(this, args);
  };
  next();
});

To forward Nginx's own metrics via Server-Timing, use the map directive to avoid sending empty values for static files or errors:

Nginx (upstream timing as Server-Timing):

      # In http {} block of nginx.conf:
      map $upstream_header_time $server_timing_upstream {
      "-"     "";
       default "ngx-upstream;dur=$upstream_header_time;desc=\"NgxUpstream\"";
      }

      # In location {} block of your server:
      add_header Server-Timing-Ngx "ngx-total;dur=$request_time;desc=\"NgxTotal\"" always;
      add_header Server-Timing-Ngx $server_timing_upstream always;

Note the use of the Server-Timing-Ngx prefix and $upstream_header_time variable. Browsers expect values in milliseconds, while Nginx returns seconds. Also, $upstream_header_time captures the moment headers are received from upstream, accurately reflecting response generation time—unlike $upstream_response_time, which can be zeroed out under certain buffering conditions.

Analyzing the data lets you quickly classify the incident:

• Dominant db;dur points to suboptimal SQL queries, missing indexes, or table locks.
• High pool-wait;dur signals connection limit exhaustion in PgBouncer or similar pooler.
• Large app;dur with small db;dur indicates CPU-bound operations, blocking I/O, or runtime memory leaks.
• 502/504 errors on the load balancer with normal timings usually mean app process crashes or orchestrator timeouts.

Network Segment: HTTP Phases and Kernel Metrics

If upstream responds quickly but overall latency grows, the issue is in the network path. Every HTTP request goes through six sequential phases: DNS, Connect (TCP handshake), TLS handshake, Send, Wait (TTFB), and Receive. Anomalies in specific phases narrow the search immediately. Long DNS points to resolver issues or complex CNAME chains. High Connect and TLS values indicate network latency to the CDN edge node, lack of TLS session resumption, or outdated protocols. If Connect and TLS are minimal but Wait is anomalously high, the delay is at the WAF, CDN workers, or origin fetch level.

For deep network diagnostics, standard HTTP breakdowns aren't enough. You need Linux kernel metrics via getsockopt(TCP_INFO). The kernel tracks every TCP connection and provides precise data: RTT, retransmit count, congestion window (cwnd) size, and RTO. These are critical because HTTP timings can't separate network latency from server processing time when the response fits in one TCP segment. Subtracting kernel RTT from the Wait phase gives the true request processing time. Retransmits and squeezed cwnd directly point to packet loss and congestion control algorithms adding hundreds of milliseconds per request. Combining app headers with low-level network metrics creates full observability without relying on third-party vendors.

Key Takeaways

• Comparing $request_time and $upstream_response_time in Nginx instantly separates network and app responsibilities.
• Server-Timing headers provide granular visibility into internal delays (DB, pools, runtime) without heavy APM systems.
• Analyzing HTTP request phases precisely identifies DNS, TLS handshake, and CDN routing issues.
• TCP_INFO kernel metrics give an objective view of network losses and delays undetectable at the app level.
• A systematic diagnostics approach cuts MTTR and eliminates chaotic service restarts during incidents.

— Editorial Team