eBPF NetFlow traffic monitoring bare metal K8s

eBPF and NetFlow for Traffic Monitoring: From Bare Metal to Kubernetes

A system built on eBPF/XDP and NetFlow delivers granular network telemetry across all layers—from physical servers to Kubernetes clusters. It overcomes limitations of traditional sFlow/NetFlow, such as high hardware overhead, sampling, and lack of custom fields. Traffic is processed through a unified router-on-a-stick interface, enabling XDP to capture incoming packets without kernel modifications.

eBPF runs programs in the OS kernel securely and with minimal latency, making it ideal for real-time monitoring. Statistics are collected per 5-tuple (IP addresses, ports, protocol), bytes, and packets, then exported in NetFlow format for collectors.

Choosing BPF Maps for High Performance

BPF maps store statistics as key-value tables: the key is the 5-tuple, the value is counters. Testing revealed issues with BPF_MAP_TYPE_HASH:

• Lock contention under multi-threaded access from multiple cores.
• Race conditions leading to inaccurate counters.

Atomic operations (__sync_fetch_and_add) eliminate errors but reduce performance due to synchronization overhead.

Switching to BPF_MAP_TYPE_PERCPU_HASH creates a separate map per CPU core, eliminating locks. Aggregation happens in user space. Benchmarks on 100 cores under varying loads confirmed PERCPU maps outperform others.

When full (600K 5-tuples in a 60K-entry map), BPF_MAP_TYPE_LRU_PERCPU_HASH evicts old entries—but at a cost. Standard PERCPU_HASH remains preferred for fast, one-time reads.

// Determine which map to write to
#define SHARD_COUNT 25 // number of shards
map_hash_idx = ctx->rx_queue_index % SHARD_COUNT + shard_offset;

Map Sharding and Multi-Threaded Export

A single map holding 10M records was bottlenecked by a single-threaded exporter. Sharding into 25 maps (based on rx_queue_index via RSS) enabled parallel reading, speeding up processing by 3x.

Benefits of sharding:

• Scalability: adding maps without restarts.
• Parallelism: sensors read multiple maps simultaneously.
• Cache efficiency: small maps fit in L1/L2 cache.

The Go-based sensor (using Cilium libraries) leverages BPF_MAP_LOOKUP_AND_DELETE_BATCH to read and clear maps every minute—minimizing data loss during overflow.

Sampling and Dynamic Load Adaptation

To balance forwarding performance with telemetry needs, sampling is implemented. A secondary stats_map (BPF_MAP_TYPE_HASH, max_entries=128) tracks packets per shard:

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 128);  // Max number of shards
    __type(key, __u32);        // Shard ID
    __type(value, __u64);      // Packet/byte counters
    __uint(pinning, LIBBPF_PIN_BY_NAME);
} stats_map SEC(".maps");

eBPF timers periodically calculate throughput (packets/sec) and trigger sampling when thresholds are exceeded per shard. Timers run background tasks outside packet context—aggregating data and resetting counters.

This maintains stability during traffic spikes (like DDoS) while preserving session-level telemetry.

Adapting to Virtualization and Kubernetes

The bare metal architecture scales seamlessly to virtualized environments and Kubernetes. In K8s clusters, eBPF/XDP runs as a DaemonSet, capturing pod traffic via CNI. Sharding adapts to node affinity; PERCPU_HASH aligns with vCPUs.

NetFlow export aggregates metadata (namespace, pod IP), enhancing observability of microservices.

Key takeaways:

• PERCPU_HASH eliminates lock contention, boosting throughput by 2–3x vs HASH.
• Sharding accelerates export by 3x through parallelism.
• eBPF timers + sampling balance load without sacrificing forwarding.
• LRU variant suits dynamic scenarios with overflow.
• Go sensor with Cilium libraries excels at high-throughput parsing.

— Editorial Team