ACE-Step 1.5: music neural network outperforms Suno locally

ACE-Step 1.5 XL: Open-Source Music Generation Model Outshining Suno on SongEval

ACE-Step 1.5 XL is an open-source solution under the MIT license for synthesizing full music tracks. This 4B-parameter DiT decoder model runs locally on GPUs with as little as 4 GB VRAM in the base version. It scores 8.09 on the SongEval benchmark, beating Suno v5. Developers from ACE Studio and StepFun released a research paper detailing its architecture.

The two-stage pipeline combines a language model for planning and a DiT for audio synthesis, delivering highly coherent tracks.

Architecture: Splitting LM and DiT

The model uses a two-stage process to boost generation quality.

Stage 1 — Language Model (0.6B–4B parameters):

Takes a text prompt and applies chain-of-thought reasoning for step-by-step planning. It generates the track structure: sequence of sections (verse, chorus, bridge), style, instruments, tempo, arrangement, and lyrics. Output is an intermediate text representation, not audio.

Stage 2 — Diffusion Transformer (DiT, 2B–4B parameters):

Converts the plan into audio using DCAE for latent compression. This cuts VRAM needs and speeds up inference.

Task separation lets the LM handle high-level logic while the DiT focuses on detailed synthesis. The turbo variant uses just 4–8 diffusion steps instead of 50–100, thanks to distillation.

Performance and System Requirements

Key metrics from the research paper:

• SongEval: 8.09 (beats Suno v5)
• Lyric Alignment: 8.35 — precise vocal-to-lyrics sync
• Inference Speed: ~2s on A100, ~10s on RTX 3090 for a full track
• VRAM: 4 GB (base), 12 GB (XL with offload)
• Track Length: 10s to 10 minutes

The model supports cross-platform runs: NVIDIA, AMD ROCm, Apple Silicon. Local execution keeps your data private.

Extra Features

ACE-Step goes beyond basic text-to-music:

• Cover Generation: Re-arrange audio into a new style
• Audio Repainting: Locally regenerate track sections
• Vocal-to-BGM: Create backing tracks from vocals
• LoRA Fine-Tuning: Adapt to user tracks to capture their style

Repainting is great for iterative tweaks: fix dodgy bars without regenerating everything. That said, the authors note artifacts at seams.

Model Limitations

The developers are upfront about real issues:

• Inconsistency: Results vary by seed and length — reruns can flop
• Vocals: Rough synthesis lacking nuance, good for backgrounds but not lead vocals
• Genres: Struggles with niche styles, like Chinese rap
• Parameter Control: No precise BPM, key, or chord tweaks — just descriptive prompts

These are common hurdles for diffusion models at this stage.

Local Setup and Running

For a quick test, use the Gradio interface after cloning the repo.

git clone https://github.com/ace-step/ACE-Step-1.5
cd ACE-Step-1.5

# Windows
start_gradio_ui.bat

# Linux
chmod +x start_gradio_ui.sh && ./start_gradio_ui.sh

# macOS (Apple Silicon)
chmod +x start_gradio_ui_macos.sh && ./start_gradio_ui_macos.sh

The script auto-downloads weights. Base version runs on RTX 3060 (12 GB); XL needs 20+ GB. ComfyUI integration is available via ready-made nodes.

Key Takeaways

• First open-source model topping Suno v5 on SongEval (8.09)
• Two-stage LM+DiT architecture with DCAE for efficient latent synthesis
• Turbo mode: 4–8 diffusion steps, seconds-long inference on consumer GPUs
• Features: repainting, LoRA, cover generation for customization
• Honest limitations: inconsistency, rough vocals, limited control

— Editorial Team