vLLM Production Stack: basic launch in K8s

vLLM Production Stack: Deployment and Basic Setup for Production

The vLLM Production Stack simplifies deploying high-performance LLM inference on Kubernetes. It consists of a group of vLLM instances paired with a router to handle request routing. Supports OpenAI-compatible APIs: chat/completions, embeddings, tokenization, audio transcription, and translation. Test environment: 2 servers with 8× NVIDIA H200 (141 GB), Astra Linux SE 8, Kubernetes 1.30, vLLM 0.17.0, Production Stack 0.1.10.

Configuration examples are shown as YAML snippets. Full manifests include standard Kubernetes fields, omitted here for brevity. Parameter conflicts are noted separately.

Supported Models and Loading

vLLM supports most current LLMs. Use nightly builds for new models. Models are downloaded from Hugging Face by default. In offline environments, mount via PVC:

extraVolumeMounts:
  - mountPath: /models/Qwen3-8B
    name: model
    readOnly: true
extraVolumes:
  - name: model
    persistentVolumeClaim:
      claimName: pvc-qwen3-8b

Basic Qwen3-8B Deployment

Deploy Qwen3-8B with local mounting and OpenAI API support:

servingEngineSpec:
  enableEngine: true
  modelSpec:
    - name: qwen3-8b
      repository: vllm-openai
      tag: v0.17.0
      env:
        - name: HF_HUB_OFFLINE
          value: "1"
      modelURL: /models/Qwen3-8B
      extraVolumeMounts:
        - mountPath: /models/Qwen3-8B
          name: model
          readOnly: true
      extraVolumes:
        - name: model
          persistentVolumeClaim:
            claimName: pvc-qwen3-8b
      replicaCount: 1
      requestCPU: 8
      requestGPU: 1
      requestMemory: 16Gi
      vllmConfig:
        extraArgs:
          - --served-model-name
          - Qwen/Qwen3-8B
        gpuMemoryUtilization: 0.98

Setting HF_HUB_OFFLINE=1 disables all Hugging Face Hub calls. Port forwarding: kubectl port-forward svc/... 9191:9191. Test request:

curl -N http://localhost:9191/v1/chat/completions \
  -H "Content-Type: application/json" \
  --data-raw '{
    "model": "Qwen/Qwen3-8B",
    "messages": [
      { "role": "user", "content": "Write: Hello!" }
    ]
  }'

Models in thinking mode return a <think>...</think> block.

Tool Calling and Thinking Modes

Enable server-side options for tool calling:

extraArgs:
  - --enable-auto-tool-choice
  - --tool-call-parser
  - qwen3_coder

Without these, agent workflows fail. Enable/disable thinking mode via --default-chat-template-kwargs:

extraArgs:
  - --default-chat-template-kwargs
  - '{"enable_thinking": false}'

Client-side chat_template_kwargs override server settings. For DeepSeek: {"thinking": false}.

Multimodal Models and CPU Backend

Multimodal models require a custom image:

FROM vllm/vllm-openai:v0.17.0
RUN uv pip install --system "vllm[audio]==0.17.0" && \
    uv pip install --system "vllm[video]==0.17.0"

Increase shmSize: "32Gi" for mm_processor_cache. Set tensorParallelSize: 1. For Qwen3-Omni-30B: use --limit-mm-per-prompt.

CPU-only inference for embedding models (Qwen3-Embedding-0.6B):

env:
  - name: VLLM_CPU_KVCACHE_SPACE
    value: "48"
vllmConfig:
  extraArgs:
    - --served-model-name
    - Qwen3-Embedding-0.6B

VLLM_CPU_KVCACHE_SPACE must be set with memory buffer.

Key Runtime Parameters

| Parameter | Purpose | When Critical |

|----------|---------|---------------|

| enablePrefixCaching | Cache shared prefix KV-cache | Repeated prompt beginnings |

| enableChunkedPrefill | Split long prefill stages | Long inputs, mixed workloads |

| maxModelLen | Max context length | Memory savings, long contexts |

| dtype | Weight data type | Precision control |

| tensorParallelSize | GPU partitioning | Large models |

| maxNumSeqs | Max sequences per batch | Parallelism, latency |

| gpuMemoryUtilization | GPU memory usage | OOM optimization |

gpuMemoryUtilization=0.98 reserves space for KV-cache. maxModelLen limits context to save memory. enablePrefixCaching speeds up repeated prefixes.

Key Takeaways

• vLLM Production Stack is a ready-to-use Kubernetes layer with built-in router and observability.
• For offline use, mount models via PVC and set HF_HUB_OFFLINE=1.
• Tool calling requires --enable-auto-tool-choice and model-specific parser.
• Multimodal models: custom image + shmSize 32Gi + tensorParallelSize.
• CPU mode: VLLM_CPU_KVCACHE_SPACE with buffer.

— Editorial Team