Fine-tuning Gemma 4 on Cloud Run with GPU

Fine-Tuning Gemma 4 31B on Cloud Run Jobs with GPU for Image Classification

The Gemma 4 31B model is fine-tuned on the Oxford-IIIT Pet dataset using Cloud Run Jobs and NVIDIA RTX PRO 6000. Animal breed classification accuracy improved from 89% to 94% with QLoRA quantization and LoRA at rank 64. This guide covers architecture changes, data prep, and deployment on Google Cloud.

Gemma 4 Architecture Changes

Gemma 4 uses the Apache 2.0 license and a Mixture-of-Experts (MoE) architecture with 31B and 26B parameters. The context window is expanded to 256K tokens. It supports multimodal inputs: images, video, audio. Built-in function calling and structured JSON generation are included.

Previous Gemma scripts won't work due to changes in model loading and chat templates. For multimodality, use AutoModelForMultimodalLM.

Memory and GPU Requirements

The NVIDIA RTX PRO 6000 offers 96 GB VRAM. Gemma 4 31B in bfloat16 takes 62 GB. QLoRA with 4-bit quantization via bitsandbytes drops it to 18-20 GB, freeing up resources for longer contexts.

Code Adaptations for Gemma 4

Input Data Format

Images go before text. The instruction merges with the user prompt. The {"type": "image"} placeholder marks the spot for image tokens in the processor.

Label Masking

Dynamic image token counts require scanning the full input_ids array from the end to find the response boundary. Separate text tokenization causes length mismatches.

LoRA Configuration

The architecture uses Gemma4ClippableLinear. Set LoRA with target_modules="all-linear" to bypass wrappers and avoid training errors.

Fine-Tuning Results

On 700 images, intermediate accuracy matched the base model. The full Oxford-IIIT Pet dataset boosted it to 94% with:

• Rank 64 / Alpha 64
• Learning rate 5e-5

Base accuracy: 89%.

Google Cloud Environment Setup

Enable Cloud Run and Cloud Build APIs in a billed project. Grab your Hugging Face token.

• Clone the repo:

git clone https://github.com/GoogleCloudPlatform/devrel-demos
cd devrel-demos/ai-ml/finetune_gemma/

• Test on CPU with the 2B model:

python3 finetune_and_evaluate.py \
  --model-id google/gemma-4-e2b-it \
  --device cpu \
  --train-size 20 \
  --num-epochs 1

Upload weights to a GCS bucket in your target region.

Building and Running the Image

Build the Docker image:

gcloud builds submit --tag $REGION-docker.pkg.dev/$PROJECT_ID/$AR_REPO/$IMAGE_NAME:latest .

Create the Job with GPU:

gcloud beta run jobs create gemma4-finetuning-job \
  --region $REGION \
  --image $REGION-docker.pkg.dev/$PROJECT_ID/$AR_REPO/$IMAGE_NAME:latest \
  --gpu 1 \
  --gpu-type nvidia-rtx-pro-6000 \
  --cpu 30.0 \
  --memory 120Gi \
  --add-volume name=model-volume,type=cloud-storage,bucket=$BUCKET_NAME \
  --add-volume-mount volume=model-volume,mount-path=/mnt/gcs \
  --args="--model-id","/mnt/gcs/google/gemma-4-31b-it/","--output-dir","/mnt/gcs/gemma4-finetuned"

Run the job:

gcloud beta run jobs execute gemma4-finetuning-job --region $REGION --async

Key Takeaways

• QLoRA cuts VRAM to 20 GB for the 31B model on RTX PRO 6000.
• AutoModelForMultimodalLM is essential for images before text.
• Masking from end of input_ids handles dynamic tokens.
• target_modules="all-linear" stabilizes LoRA on Gemma4ClippableLinear.
• Accuracy jumped 5% to 94% on Oxford-IIIT Pet.

— Editorial Team