# Running Qwen 3.5 with 397 Billion Parameters on iPhone 17 Pro Using Flash-MoE and External SSD
The iPhone 17 Pro successfully ran the MoE model Qwen3.5-397B-A17B with 397 billion parameters. This was achieved using streaming weight loading from an external SSD via Flash-MoE. Inference speed reached 0.6 tokens/s, demonstrating the fundamental feasibility of running massive models on mobile devices.
MoE Architecture and Memory Requirements
Qwen3.5-397B-A17B uses Mixture of Experts (MoE), where only a subset of parameters is activated per token. Total volume—397 billion parameters—but with 4-bit quantization, it requires about 200 GB for storing weights. The iPhone 17 Pro has 12 GB RAM, which isn't enough without special techniques.
Flash-MoE solves the problem with streaming weight loading directly to the GPU. This bypasses RAM limitations by offloading storage to an external SSD. Without such optimization, the model won't launch even with aggressive compression.
Test Results on Different Platforms
| Device | RAM | Inference Speed (tokens/s) |
|-----------------|------|----------------------------|
| iPhone 17 Pro | 12 GB | 0.6 |
| MacBook Pro M3 Max | 48 GB | 4.3 |
On the iPhone, generating one token takes 1.7 seconds (roughly 2–3 seconds per word). Tests confirmed functionality, but the speed is unacceptable for production. Developers used code generated by Claude and gentle 4-bit quantization.
Optimization Opportunities
The project isn't aimed at practical use but serves as a proof-of-concept. Further optimization could include:
- Improved quantization (e.g., 2-bit or AWQ).
- Parallel weight loading.
- Caching frequently used MoE experts.
- Integration with Metal Performance Shaders for iOS.
- Adaptation for the A18 Pro NPU in iPhone 17 Pro.
Such steps could potentially speed up inference 5–10x without quality loss.
Key Takeaways
- Flash-MoE enables running 397B models on devices with 12 GB RAM via streaming from SSD.
- MoE architecture reduces compute load by activating ~17B parameters per step.
- Baseline speed 0.6 t/s on iPhone; 4.3 t/s on M3 Max—benchmark for optimizations.
- The test highlights the shift toward edge inference for massive LLMs on consumer devices.
- Test code based on Claude; 4-bit quantization without major artifacts.
This technology paves the way for running frontier models locally without the cloud, minimizing latency and ensuring data privacy.
— Editorial Team
No comments yet.