SD 3.5 Medium on M1: optimization to 11.6 GB

Memory Optimization for Stable Diffusion 3.5 Medium on Apple Silicon M1

The Stable Diffusion 3.5 Medium model with 2.5B parameters can consume up to 21 GB of unified memory on Apple Silicon due to the T5-XXL encoder (10.75 GB in fp16). Optimized loading reduces peak usage to just 11.6 GB, enabling smooth operation on M1 Macs with 16 GB of RAM. This guide breaks down the issue and introduces a step-by-step loading strategy.

Root Causes of Memory Spikes:

• Weight duplication: model loaded on CPU, then copied to MPS.
• enable_model_cpu_offload() is ineffective—CPU and MPS compete for unified memory.
• Accelerate hooks with device_map hold strong references, preventing garbage collection.

Standard code:

pipe = StableDiffusion3Pipeline.from_pretrained(
    "stabilityai/stable-diffusion-3.5-medium",
    torch_dtype=torch.float16, variant="fp16")
pipe.to("mps")

Peak memory: 21.4 GB regardless of num_inference_steps.

Solution: Sequential Loading

Split the process into phases: prompt encoding (T5), unload, then generation (DiT + VAE).

1. Load with device_map="balanced"

Directly load into MPS without CPU staging:

pipe = StableDiffusion3Pipeline.from_pretrained(
    path, torch_dtype=torch.float16, device_map="balanced")

T5 peak: ~11 GB.

2. Proper Hook Removal

from accelerate.hooks import remove_hook_from_submodules

for attr in list(vars(pipe)):
    comp = getattr(pipe, attr, None)
    if isinstance(comp, torch.nn.Module):
        remove_hook_from_submodules(comp)
        setattr(pipe, attr, None)

del pipe
gc.collect()
torch.mps.synchronize()
torch.mps.empty_cache()

Without this, T5 remains in memory (10.75 GB).

3. Full Sequential Workflow

Phase 1: Encoder

enc_pipe = StableDiffusion3Pipeline.from_pretrained(path,
    transformer=None, vae=None,
    text_encoder=None, text_encoder_2=None,
    tokenizer=None, tokenizer_2=None,
    variant="fp16", torch_dtype=torch.float16,
    device_map="balanced")

prompt_embeds, neg_embeds, pooled, neg_pooled = enc_pipe.encode_prompt(
    prompt="a warrior with a sword",
    prompt_2="a warrior with a sword",
    prompt_3="a warrior with a sword",
    device="mps", num_images_per_prompt=1)

Unload after this step.

Phase 2: Generator

gen_pipe = StableDiffusion3Pipeline.from_pretrained(path,
    text_encoder=None, text_encoder_2=None, text_encoder_3=None,
    tokenizer=None, tokenizer_2=None, tokenizer_3=None,
    variant="fp16", torch_dtype=torch.float16)
gen_pipe.to("mps")

image = gen_pipe(
    prompt_embeds=prompt_embeds,
    negative_prompt_embeds=neg_embeds,
    pooled_prompt_embeds=pooled,
    negative_pooled_prompt_embeds=neg_pooled,
    num_inference_steps=40).images[0]

.to("mps") reuses the T5 cache, cutting DiT footprint from 11.5 GB down to 6 GB.

Memory Usage Comparison

| Stage | Before Optimization | After Optimization |

|-------------------|---------------------|--------------------|

| T5 Load | 15.3 GB | 11.5 GB |

| T5 After Unload | 10.75 GB | 0.01 GB |

| DiT + Diffusion | 8.9 GB | 8.9 GB |

| Peak | 21.4 GB | 11.6 GB |

Performance on Hardware

• M1 16 GB: 4:54 min (T5: 1:06, DiT: 3:34), 11–13 GB peak.
• M3 Pro 36 GB: 1:47 min (T5: 0:14, DiT: 1:27), 9–11 GB peak.

Resolution: 512x512, 40 inference steps.

Key Takeaways:

• T5-XXL is the primary memory consumer (~71% of total).
• device_map="balanced" combined with hook removal is essential for unloading.
• Sequential loading cuts peak memory by 45%.
• Works reliably on M1 16 GB Macs without swapping.
• .to("mps") on DiT enables efficient cache reuse.

— Editorial Team