Realtime video generation: FLOPS calculation and optimizations

Real-Time Video Generation in 2026: FLOPS Calculation, Optimizations, and Hardware Limits for DiT Architectures

How many computational resources are needed to generate video in real time—not as a demo, but as an industrially applicable inference? The answer requires not abstract estimates, but first-principles analysis: from the specific Wan2.1-14B architecture to the physical limitations of Tensor Cores and mobile NPUs. This article provides a rigorous calculation of the computational load at each stage, calibration against real-world measurements on H100 GPUs, and a quantitative assessment of the impact of every optimization: FlashAttention, step distillation, sparse attention, quantization, and latent space compression.

Architectural Breakdown of Wan2.1: Where FLOPS Are Concentrated

Wan2.1-14B is a three-stage system: a text encoder (UMT5, 5.3B), a DiT decoder (14B), and a VAE decoder (127M). Crucially, true real-time implies autoregressive frame-by-frame generation with a fixed context window. We use the standard task: 5 seconds of video at 16 FPS = 81 frames → 720p (1280×720), which results in 21 temporal × 3600 spatial latent tokens = 75,600 tokens per denoising step.

DiT consists of 40 layers. Each layer includes:

• Self-attention (QKV projections + score matrix)
• Cross-attention (text → video)
• Feedforward network (FFN)

With d=5120, d_ff=13,824, and S=75,600:

• Self-attention: ≈133 TFLOP (82% of the total workload)
• Cross-attention: ≈8.4 TFLOP per layer
• FFN: ≈21.4 TFLOP per layer

Total per layer: ~163 TFLOP. For 40 layers and 50 steps: 6.5 PFLOP per 5 seconds of video. Real-world measurements confirm this calculation: 242 seconds on a single H100 → 48 seconds per second of video, which aligns with the theoretical 6.5 PFLOP at an MFU of ~30%.

Optimizations: From Kernel Acceleration to Representation Compression

Increasing throughput requires the combined application of algorithmic and hardware improvements. Below is the quantitative contribution of each approach to reducing latency for 720p:

• FlashAttention + torch.compile: eliminates materialization of the 75K×75K attention matrix, reducing memory demands. Increases MFU to 50–60%. Speedup: ×1.7.
• Step distillation (from 50 to 4 steps): trains the model to achieve similar quality in fewer denoising steps. Conservative factor: ×10.
• CFG distillation: replaces two forward passes (with and without prompt) with one. Effect: ×2.
• Sparse attention (VSA / Block Sparse): skips insignificant token pairs. Realistic gain: ×4.
• Quantization to FP8: activates specialized Tensor Cores. On H100: ×2 in compute throughput.

The cumulative effect of an aggressive configuration (4 steps, full sparse, no CFG, FP8): ×1.7 × 10 × 2 × 4 × 2 = ×272. The baseline latency of 48 seconds drops to 0.18 seconds—real-time is achieved even on a single H100.

Why VAE Is the Key Lever of the Future

Attention is the dominant bottleneck: its computational complexity grows quadratically with the number of tokens (S²). Therefore, compressing the latent space yields not linear, but exponential acceleration. Wan2.1 uses 8×8 spatial compression → 75,600 tokens. Wan2.2 switched to 16×16 → four times fewer tokens (18,900), and attention-FLOPs are 16 times lower. Given that attention accounts for 82% of all computations, the overall gain is ~13.6×.

This is equivalent to two generations of GPUs (following the trend of ~2× every 18 months). LTX2.3 confirms this trend: 32× spatial and 8× temporal compression allow achieving 1080p real-time today—not by increasing FLOPS, but by more efficient data representation.

Mobile Devices: Why Real-Time Is a 2030s Issue

Performance comparisons reveal a three-order-of-magnitude gap:

• H100 (FP8): 1,979 TFLOP/s
• Snapdragon 8 Elite (FP8): 45 TOPS = 0.045 TFLOP/s

Even with ideal optimization (×272) and switching to a 1.3B model, mobile NPUs remain 2–3 orders of magnitude slower. Historical TOPS growth (~2× every 2 years) suggests the following timeline:

• 480p@1FPS: achievable by 2027
• 480p@12FPS: ~2031–2033
• 720p real-time: no earlier than 2033–2035

However, every 10× improvement in model efficiency (e.g., through hardware-aware architectures or a new VAE) shifts the timeline by about 3 years. Breakthroughs in this direction are the only way to achieve mobile real-time without compromising quality.

What Matters

• Self-attention is the main bottleneck: its quadratic dependence on the number of tokens makes latent space compression critically important, not just useful.
• Optimizations work multiplicatively: FlashAttention, step distillation, sparse attention, and quantization don’t compete—they combine, delivering a total speedup of over 200×.
• Real-time has already been achieved on servers: 720p in under 1 second on a single H100 with aggressive optimization—is not a prediction, but a measured result.
• Mobile NPUs will never catch up to servers in raw compute: their path lies in specialization (e.g., dedicated video inference units), not in scaling TOPS.
• VAE is the key engineering lever: improving latent space compression delivers more acceleration than two generations of GPUs—it’s a fundamental shift in the paradigm of model design.

— Editorial Team