FLUX.2-dev GGUF Q4_K_M: 29 GB VRAM on Apple Silicon

Why FLUX.2-dev GGUF Q4_K_M Uses 29GB VRAM on Apple Silicon

The FLUX.2-dev model in the quantized GGUF Q4_K_M format should theoretically use around 19GB of memory, but on Apple Silicon with an M3 Pro chip, it peaks at 28.92GB. Analysis reveals the extra gigabytes stem from MPS overhead, weight dequantization, and data transfers between CPU and GPU in unified memory.

Loading Architecture and Sequential Offload

FLUX.2-dev uses the Mistral-Small-3.2-24B text encoder, which requires 45GB VRAM in full precision. For Apple Silicon, a 4-bit version via MLX is used, consuming 16GB. The process follows a sequential offload pattern:

• Load the MLX encoder (~13GB).
• Encode the prompt into embeddings on the CPU.
• Offload the encoder and clear the cache.
• Load the DiT model (GGUF Q4_K_M) from disk (~19GB).
• Initialize the VAE and scheduler on the CPU.
• Transfer the entire pipeline to MPS.

The theoretical peak is the maximum of individual component sizes, but reality differs due to unified memory behavior.

Test Environment

• macOS: 26.3.1 (Tahoe)
• Chip: Apple M3 Pro, 36GB unified memory
• Python: 3.14.3
• PyTorch: 2.11.0
• diffusers: 0.38.0.dev
• mlx-lm: 0.31.1

Profiling was done using phys_footprint (similar to Activity Monitor) and PyTorch’s MPS metrics. psutil.rss does not account for MPS allocations.

Detailed Memory Usage Breakdown

Step-by-Step Measurements

| Stage | phys_footprint | MPS driver | MPS current |

|-------|----------------|------------|-------------|

| Baseline | 0.27 GB | 0 | 0 |

| MLX encoder loaded | 13.22 GB | 0 | 0 |

| GGUF on CPU | 19.23 GB | 0 | 0 |

| pipe.to(mps) | 23.79 GB | 20.06 GB | 18.75 GB |

| Mid-inference | 26.50 GB | 25.62 GB | 18.78 GB |

The peak of 28.92GB occurs during pipe.to("mps") due to simultaneous presence of CPU and MPS copies of data.

Anatomy of the 28GB Peak

• DiT Q4_K_M (uint8): 18.59 GB
• VAE (bf16): 0.16 GB
• MPS pool overhead: +1.31 GB
• IOKit/Metal graphCache: ~3.73 GB (shaders, page tables, limbo buffers)

GGUF weights remain in uint8 on MPS; dequantization to bf16 happens layer-by-layer during forward pass. Temporary bf16 tensors (up to 6.92GB) accumulate in cache.

Optimization Strategies to Reduce Peak Memory

• Incremental MPS Transfer: Move parameters one by one with gc.collect() every 30 parameters.

def _move_to_device_incremental(pipe, device):
    n = 0
    for attr in list(vars(pipe)):
        comp = getattr(pipe, attr, None)
        if not isinstance(comp, torch.nn.Module):
            continue
        for param in comp.parameters():
            param.data = param.data.to(device)
            n += 1
            if n % 30 == 0:
                gc.collect()
        for buf in comp.buffers():
            buf.data = buf.data.to(device)
        gc.collect()
    torch.mps.synchronize()
    torch.mps.empty_cache()

• empty_cache() Between Steps: Use a callback during inference steps with torch.mps.empty_cache() to flush dequantization cache.

• Post-Generation Flush: After inference, run gc.collect() followed by torch.mps.empty_cache().

Results of Optimizations

| Metric | Before | After | Delta |

|--------|--------|-------|-------|

| Peak phys_footprint | 28.92 GB | 27.36 GB | -1.56 GB |

| Steady state after inference | 26.25 GB | 21.76 GB | -4.49 GB |

| Generation time (1 step) | 2:04 | 2:11 | +7s |

Peak reduction comes from incremental garbage collection during loading; steady-state improvement results from cache flushing. Non-optimizable MPS/IOKit overhead remains at ~3.5GB.

Key Takeaways:

• Minimum steady-state: ~24GB, peak inference ~27GB.
• Primary cost: 18.59GB for uint8 DiT weights + 3.5GB MPS overhead.
• Dequantization to bf16 adds 3–5GB of temporary buffers.
• CPU→MPS transfer doubles the peak due to unified memory.
• Optimizations yield -1.5GB peak and -4.5GB steady-state without quality loss.

The real minimum for FLUX.2-dev GGUF on M3 Pro is 24+ GB of unified memory. For smaller configurations, more aggressive quantization or CPU offloading is required.

— Editorial Team