Benchmark of LLM Servers for Qwen 3.5 on Mac — 2026 Results

# Benchmark of 8 Local LLM Servers for Qwen 3.5 on Mac: Surprising Results in 2026

A detailed benchmark of eight local LLM servers for running Qwen 3.5 35B MoE was conducted on a MacBook Pro with M2 Max. With a single user, the speed difference was only 2%, but with two parallel requests, one framework delivered a 2.17x speedup while another dropped to 0.85x. The analysis uncovered critical nuances in continuous batching, phantom speed metrics, and memory management issues.

Why Run LLMs Locally: Three Technical Reasons

For developers and engineers, running models locally addresses three key challenges:

• Data privacy. Process contracts, specs, and client chats without sending them to cloud APIs. A local model anonymizes personal data (names, credentials) in real time.
• Autonomous coding agents. Tools like OpenCode and Aider need an OpenAI-compatible endpoint. Setting base_url: http://mac.local:8000/v1 lets you run agents without telemetry or rate limits, saving on cloud tokens.
• No network latency. A 35B model in 4-bit quantization on M2 Max generates responses faster than cloud APIs due to zero round-trip time. On 64GB unified memory, the model takes ~20GB, leaving room for KV cache.

These factors make a local server on Mac a viable alternative for professional use, especially with sensitive data.

Why MLX Beats llama.cpp on Apple Silicon

Choosing MLX over llama.cpp stems from Apple Silicon's architecture:

• Unified memory. On M-series, CPU and GPU share a single memory pool. The 35B model lives in 20GB without copying between RAM and VRAM (unlike NVIDIA).
• Performance. MLX is consistently 10-30% faster than llama.cpp thanks to native Metal compilation. M2 Max delivers ~400 GB/s memory bandwidth, enabling 50-80 tokens/sec in real time.
• Native continuous batching. MLX ecosystem supports parallel request handling out of the box, while llama.cpp needs extra setup.

Important: Metal doesn't support GPU context sharing between processes. All tests used a single server to avoid resource conflicts.

Frameworks Compared: Selection and Criteria

The benchmark included six servers from the MLX ecosystem (two dropped due to incompatibility with Qwen 3.5 35B MoE):

• mlx-openai-server (Python 3.11) — continuous batching support, multi-model via YAML, structured output.
• mlx-omni-server (Python 3.11+) — dual API (OpenAI + Anthropic), TTS/STT.
• Rapid-MLX (Python 3.10+) — easy deployment, 1900+ tests, Cursor/Aider integration.
• vllm-mlx (Python 3.10+) — paged KV cache, multimodality, but with a critical SSE parser bug.
• omlx (Python) — tiered KV cache (RAM + SSD), web dashboard, but hard 32768 token limit.
• mlx-vlm (Python 3.10+) — 40+ architecture support, but pathological slowdown on long prompts.

Excluded:

• higgs (Rust) — doesn't support qwen3_5_moe.
• mlx-serve (Zig) — requires macOS 15 (Sequoia).

Key selection criterion: native MLX format compatibility with Qwen 3.5 35B MoE, no conversion needed.

Benchmark Methodology: How Performance Was Measured

Tests ran on a 16" MacBook Pro M2 Max (64GB unified memory) with mlx-community/Qwen3.5-35B-A3B-4bit. The Python benchmark harness included:

• Load scenarios:

* run_single — sequential processing of 8 prompts (100 to 53000 tokens).

* run_double_batch — two parallel requests via asyncio barrier:

gate = asyncio.Event()
task_a = create_task(chat_stream(..., start_gate=gate))
task_b = create_task(chat_stream(..., start_gate=gate))
await asyncio.sleep(0.05)
gate.set()
res_a, res_b = await gather(task_a, task_b)

• Metrics:

* wall_tps_p50 — median tokens/sec by wall-clock (primary metric).

* ttft_p50 — time to first token (TTFT).

* speedup — batching efficiency (wall_tps ratio for two vs. one request).

Median was used instead of mean to avoid outliers from prompt length variance. Each test repeated 5 times.

Results: Single-User vs. Parallel Processing

Single-user performance was similar across servers (within 2%). Gaps emerged with two parallel requests:

• mlx-openai-server hit 2.17× speedup — the only framework with effective continuous batching.
• vllm-mlx showed phantom 14000 tokens/sec due to an SSE parser error skewing gen_tps.
• Rapid-MLX lacks streaming generation, so TTFT equals total response time.
• mlx-omni-server degraded to 0.85× speedup without --workers 2.
• omlx returned HTTP 400 on prompts >32768 tokens.
• mlx-vlm had cyclic speed drops (790 → 3.5 tok/s) from GC issues.

Notable: quadratic attention complexity in some implementations made 52k-token prefills take 31 minutes instead of 2-3.

Key Takeaways for Developers

• Continuous batching is crucial for multi-user scenarios. Only mlx-openai-server delivered real speedup on parallel requests. Others either didn't process them in parallel or needed manual worker tweaks.
• Validate metrics on real data. vllm-mlx's phantom 14000 tok/s came from SSE stream handling bugs, making gen_tps unreliable.
• Unified memory is an edge, but not a cure-all. Despite shared pools, tiered KV cache (like omlx) matters for very long contexts.
• Test long prompts. mlx-vlm's pathological slowdown on >30k-token prompts can render it unusable for large-context tasks.
• Python version affects stability. Rapid-MLX with 1900+ tests was most robust but lost on batching speed.

— Editorial Team