Virtual Try-On with AI: hybrid IDM-VTON and Leffa

# Hybrid Virtual Try-On Pipeline: Combining IDM-VTON and Leffa for E-Commerce

Virtual Try-On technology powered by diffusion models enables realistic clothing try-ons in e-commerce. Developers combined the warp-free IDM-VTON model and warp-based Leffa into a hybrid pipeline. This preserves pose structure, facial expressions, and detailed textures while meeting SLA requirements for mobile apps and smart mirrors.

Limitations of Universal Generators

Models like DALL-E or Nano Banana fall short of fashion brands' strict requirements. They distort angles, body proportions, and facial expressions even with detailed prompts. Main issues:

• Changes to the model's face and pose.
• Hallucinations of prints, logos, and fabric textures.
• Mismatch with the original angle.
• Reliance on external APIs with risk of SLA violations.

Inpainting using masks from DensePose, YOLO, or SegmentAnything partially addresses the issue but requires on-premises inference for reliability. Fine-tuning with LPIPS and SSIM improves quality.

Warp-Based vs Warp-Free Architectures

Diffusion pipelines for Virtual Try-On fall into two categories.

Warp-Based (Garment Deformation):

The algorithm applies Thin-Plate Spline or flow fields to overlay the flat clothing image onto the body. Diffusion smooths out artifacts.

Pros:

• Precise preservation of textures and details (patterns, logos).

Cons:

• Artifacts with complex poses or noisy photos.
• Issues with clothing of different lengths (t-shirt over shirt).

Warp-Free (Attention-Based):

The clothing image is encoded into features and passed through Cross-Attention in UNet with IP-Adapter. A mask defines the inpainting area.

Pros:

• Natural folds, shadows, and lighting.
• Works with in-the-wild photos from fitting rooms.

Cons:

• Blurring of complex patterns and text.

IDM-VTON: Foundation of Structure

IDM-VTON is a warp-free model with two UNets: GarmentNet encodes the clothing, TryonNet generates the result. Features are injected into self-attention layers.

The model forms the global structure: sleeve contours, silhouette on pose, light falloff. After fine-tuning on DressCode and VITON-HD datasets, LPIPS and SSIM metrics improved for non-studio photos.

Leffa: Texture Detailing

Leffa integrates warp logic into attention layers via flow fields. DensePose provides 3D body geometry, and the model projects textures accounting for geometry.

The optimized version without pose generation reduces VRAM by 40%. Leffa restores sharpness to prints, textures, and folds.

Implementing Hybrid Inference

The hybrid uses an iterative diffusion denoising process.

• First 25–40% of steps (IDM-VTON): Noisy latent + mask from DensePose. Forms a correct silhouette without body artifacts.

• Remaining steps (Leffa): Latent is passed to Leffa for texture refinement on the ready geometry.

This combines warp-free naturalness with warp-based detail. Inference time is lower than FLUX+LoRA.

Results:

• IDM-VTON: Correct shapes, weak detailing.
• Leffa: Sharp textures, contour artifacts.
• Hybrid: Balance of structure and details, works in in-the-wild scenarios.

Key Takeaways

• The hybrid pipeline minimizes body artifacts and preserves textures.
• On-premises inference ensures SLA for retail.
• Fine-tuning on real datasets improves LPIPS/SSIM.
• Leffa optimization reduces VRAM by 40%.
• The approach works for various clothing types without retraining.

— Editorial Team