Face swap on Rust: 60 FPS without Python | Technical Breakdown

Real-Time Face Swap on Rust: 60 FPS Without Python and Lock-Free Architecture

Existing solutions for real-time face swapping are often tied to Python and its ecosystem, leading to excessive dependencies and performance issues. This article breaks down the implementation of an app in pure Rust that achieves 60 frames per second without using Python. Key components: ONNX Runtime for inference, lock-free threads, and full memory optimization.

Frame Processing Pipeline

Each frame goes through four neural network models in strict sequence. RetinaFace detects faces and extracts five keypoints. ArcFace generates a 512-dimensional embedding of the source face. InSwapper takes the target face region and source embedding, returning the swapped face. GFPGAN optionally improves the result quality. All models run through ONNX Runtime without custom CUDA kernels—only tensors on input and output.

Architectural feature: no intermediate allocations. Pixel buffers are allocated once at app startup. RGBA to RGB conversion, tensor filling, and affine transformations are performed without additional memory allocations. The only allocation per frame is creating an Arc for the final snapshot, which is inevitable when using ArcSwap.

Processing loops are optimized for instruction-level parallelism. For example, color space conversion processes 4 pixels per iteration:

for i in 0..chunks {
    let si = i * 16;
    let di = i * 12;
    rgb[di] = rgba[si];
    rgb[di + 1] = rgba[si + 1];
    rgb[di + 2] = rgba[si + 2];
    rgb[di + 3] = rgba[si + 4];
    rgb[di + 4] = rgba[si + 5];
    rgb[di + 5] = rgba[si + 6];
    // ... also 2 pikselya
}

Lock-Free Multithreading Architecture

The system uses three threads with zero locks on the hot path. The capture thread receives frames via nokhwa and publishes them through ArcSwap. The pipeline thread processes frames and passes the result through a second ArcSwap. The UI thread renders the current buffer via egui. Critical optimization: aligning structures to cache lines (64 bytes) to prevent false sharing:

const CL: usize = 64;
#[repr(C, align(64))]
struct ProducerCell {
    frame: ArcSwap<FrameSnapshot>,
    generation: AtomicU64,
    _pad: [u8; CL - 16],
}
const _: () = {
    assert!(std::mem::size_of::<ProducerCell>() == CL);
};

Compile-time asserts guarantee the size compliance. Atomic generation counters replace mutexes, eliminating synchronization overhead. Avoiding async and channels simplifies the logic and reduces overhead.

Macros for Minimizing Code Duplication

Four neural networks require identical operations: RGBA to BGR planar format conversion and normalization. Instead of copying code, macros are used. The fill_tensor_planar! macro is parameterized by the normalizer:

// For ArcFace: (v - 127.5) / 127.5
fill_tensor_planar!(
    self.tensor,
    self.aligned,
    PLANE,
    FS,
    FS,
    FS * 3,
    bgr_norm_arcface
);

// For InSwapper: v / 255.0
fill_tensor_planar!(
    self.tensor,
    self.aligned,
    PLANE,
    SS,
    SS,
    SS * 3,
    bgr_inv255
);

Similarly, build_session! eliminates the triple-nested ort_err! call during ONNX Runtime initialization:

macro_rules! build_session {
    ($path:expr, $threads:expr) => {{
        let builder = ort_err!(Session::builder())?;
        let mut builder = ort_err!(builder.with_intra_threads($threads))?;
        ort_err!(builder.commit_from_file($path))?
    }};
}

Geometric Transformations and UI

Face alignment is implemented using a custom similarity transform. The five keypoints from RetinaFace are matched to the ArcFace template by solving a 4x4 system using the Gauss method with partial pivoting. The inverse transform is applied for bilinear warping and paste-back with alpha blending:

let edge_dist = ax.min(ay).min(fs - ax).min(fs - ay) - margin;
let alpha = (edge_dist * inv_blend).clamp(0.0, 1.0);

This ensures smooth transitions at the edges without artifacts. The UI is built on egui in immediate mode, ideal for live video. A custom frameless theme with macOS-style traffic lights and a transparent background allows dragging the window without retained state. Rendering each frame without delays maintains stable 60 FPS.

Key Takeaways

• Rust's ownership model eliminated data races and use-after-free even in a complex multithreaded architecture.
• ONNX Runtime via ort proved production-ready: stable model loading and inference without CUDA dependencies.
• egui for real-time—immediate mode rendering without retained state minimizes latency with video streams.
• Zero allocations on the hot path achieved by pre-allocating buffers and optimizing loops.
• Lock-free via ArcSwap with cache alignment eliminated inter-thread synchronization overhead.

The solution proves Rust is effective for high-performance computer vision tasks. A single binary without Python dependencies runs out of the box—just download the release from GitHub. For further performance scaling, switching to multi-session ONNX Runtime looks promising, enabling parallel model execution instead of sequential processing.

— Editorial Team