Quantization Gemma 4 31B under 57GB Kaggle

Optimizing Gemma 4 31B for Kaggle Limits: Quantization and Memory Management

The Gemma 4 31B model in float16 format takes up 62 GB, exceeding Kaggle's 57.6 GB disk limit. The task requires loading the full model, quantizing it to 4-bit representation (18 GB), and saving the result without exceeding available space. The solution is implemented by sequentially utilizing the VRAM of two T4 GPUs with 15 GB each, using device_map="auto" from the bitsandbytes library with NF4 type.

Quantization runs in parallel with loading: weights are compressed immediately after being transferred to video memory, minimizing peak disk usage.

Freeing Up Space: Deleting Cache During Operation

After fully loading the model into VRAM, shutil.rmtree() is applied to delete the Hugging Face cache folder. In Linux, this is possible for open files—the space becomes available for new operations, although du will still show it as occupied.

Process:

• The model is loaded and quantized in memory.
• The original 62 GB is deleted.
• The freed space is used to offload the 18 GB quantized weights.

# Example of streaming upload
model.push_to_hub("repo_id", safe_serialization=True)

This approach bypasses the mathematical barrier of 62 + 18 = 80 GB.

Implementation Details on T4 GPUs

The bitsandbytes library distributes model layers between two T4s. NF4 quantization reduces size by 4 times without significant loss in quality for code understanding and generation tasks. Full sequence:

• Initialization with device_map="auto".
• Parallel quantization of shards.
• Deleting cache after transfer to VRAM.
• Direct publication to repository via push_to_hub.

Result—a model size of 18.3 GB, compatible with consumer GPUs.

Advantages of the Method for MLOps

This technique is applicable to large models in constrained environments:

• Minimal requirements: Only free Kaggle or similar platforms.
• Automation: Script runs without manual intervention.
• Quality retention: NF4 maintains performance at float16 level for inference.
• Scalability: Easily adaptable to other limits (e.g., Colab).

The method does not require A100 clusters and works on standard hardware.

Key Takeaways

• NF4 quantization via bitsandbytes allows compressing 62 GB to 18 GB on the fly.
• Deleting Hugging Face cache with open files frees up disk space during operation.
• device_map="auto" distributes load across multiple T4 GPUs.
• Direct upload with push_to_hub eliminates intermediate saving.
• The approach is suitable for Gemma 4 31B and similar models with over 30B parameters.

— Editorial Team