# Distilling Gemini for On-Device AI on Apple Devices
Apple is integrating Google’s Gemini capabilities not just through cloud queries, but by distilling the model to run on iPhone. Access to Gemini’s internal computations enables the creation of compact versions that mimic the original’s chain of thought with 1.2 trillion parameters. This reduces resource requirements without losing core logic.
Model Distillation Mechanism
Distillation involves training a compact 'student model' on data from the 'teacher'—the full Gemini. Apple gets not just ready answers, but the full chain of thought, which is crucial for preserving quality. This student model reproduces Gemini’s complex reasoning locally, using minimal computations.
The process includes:
- Access to intermediate computations: Not API responses, but internal tensors and activations.
- Imitating chain of thought: The student model learns to repeat the sequence of thinking steps.
- Parameter compression: From 1.2 trillion down to sizes suitable for iPhone’s NPU.
Despite its efficiency, the method isn’t perfect: Gemini’s specializations (e.g., multimodality) don’t always align with Siri’s tasks. Fine-tuning is needed to adapt to the voice assistant and data privacy.
Apple’s Strategy: On-Device Inference
Apple focuses on local AI processing, minimizing cloud calls. Distilled Gemini models fit into Apple Intelligence, giving Siri autonomy. This aligns with their privacy policy—data never leaves the device.
Testing includes:
- Standalone Siri app for iOS 27.
- Enhanced Ask Siri feature.
- Integration into the WWDC presentation on June 8.
The Apple Foundation Models team is also developing proprietary solutions in parallel. Goals are ambiguous: could be general-purpose models or niche ones optimized for Apple Silicon hardware.
Technical Challenges and Prospects
Distillation requires balancing model size and performance. The iPhone Neural Engine (NPU) with 35 TOPS can handle models up to a few billion parameters, but Gemini’s chain of thought adds overhead. Optimization includes quantization and pruning.
Potential:
- Improving Siri in areas where Gemini excels (coding, analysis).
- Hybrid mode: local model + cloud fallback.
- Scaling to other devices (iPad, Mac).
Developing in-house models could reduce reliance on Google, but distillation speeds up rollout.
Key Points
- Apple uses Gemini’s internal computations for distillation, not just the API.
- Compact models copy chain of thought, running locally on iPhone.
- Siri will get on-device AI in iOS 27, presentation at WWDC on June 8.
- Fine-tuning is needed due to model specialization mismatches.
- Strategy focuses on privacy without cloud queries.
— Editorial Team
No comments yet.