VOID: object removal from video using diffusion

# Netflix's VOID: Latent Diffusion for Removing Objects from Videos

Netflix has released the open-source framework VOID (Video Object Inpainting and Decomposition), which tackles the task of removing objects from videos using latent diffusion. The model decomposes the scene into background layers and dynamic masks, ensuring reconstruction that accounts for depth and perspective. This eliminates typical artifacts of classical methods, such as flickering and floating textures.

VOID Architecture and Mechanisms

VOID focuses on temporal consistency—a key challenge for diffusion models in video. Specialized attention mechanisms analyze inter-frame connections, minimizing artifacts in dynamic scenes. In benchmarks, the model outperforms ProPainter: stable mask filling even with fast camera movement or object occlusion.

The architecture is built as a modular pipeline on PyTorch with integration into Diffusers. It supports masks of any complexity—from thin lines to large objects. Step-by-step inference preserves texture detail without losing context.

Resource Requirements and Optimizations

Base run requires an NVIDIA A100 with 40 GB VRAM. For accessibility, optimizations have been implemented:

• model_cpu_offload: offloads inactive model parts to RAM.
• sequential_cpu_offload: sequential processing for 24 GB VRAM (RTX 3090/4090).

# Example run with offload
pipe = StableDiffusionInpaintPipeline.from_pretrained(
    "netflix/void",
    torch_dtype=torch.float16
)
pipe.enable_model_cpu_offload()  # or enable_sequential_cpu_offload()

Model weights and code are available on GitHub, simplifying integration into VFX pipelines and fine-tuning.

Applications in Post-Production

VOID automates rotoscoping and cleanup—the most labor-intensive stages. Decomposition allows not just painting over, but coherently rebuilding the scene. For developers, it's a foundation for custom tools in filmmaking.

In tests on complex videos with motion, VOID demonstrates:

• 40% reduction in flickering compared to ProPainter.
• Preservation of perspective when removing foreground objects.
• Scalability to long sequences without quality degradation.

Key Points

• Layer decomposition ensures temporal stability in dynamic scenes.
• Integration with Diffusers simplifies use in existing pipelines.
• Optimizations enable running on consumer GPUs (24 GB VRAM).
• Outperforms SOTA in benchmarks on fast-motion videos.
• Open-source code on GitHub for fine-tuning and extensions.

The framework changes the approach to video inpainting, making it accessible to mid/senior VFX developers.

— Editorial Team