Automating Alert Analysis in DevOps with AI and MCP Protocol
DevOps teams often spend hours digging through alerts from Slack, Grafana, and Alertmanager, especially in complex setups with Kubernetes, VictoriaMetrics, Loki, and multiple clouds. Manual triage means keeping track of details from 10+ environments, databases, and monitoring stacks in your head. The fix? A proof-of-concept (PoC) agent built on n8n, integrated with AI via MCP servers (Model Control Protocol), delivering full alert context, ready-to-run commands, and actionable recommendations.
The agent gets read-only access to your infrastructure: vmetrics for PromQL queries, Loki for logs, k8s-mcp for Kubernetes resources, Yandex Cloud Toolkit, ArgoCD-mcp, and Alertmanager-mcp. This lets it analyze metrics, logs, pod states, and silences without risking any changes.
Building the PoC on n8n with MCP Integrations
We picked n8n as a low-code workflow platform for the prototype. We spun up MCP servers: alertmanager-mcp, vmetrics-mcp, Yandex Toolkit, and k8s-mcp. AI generates the workflow, tailored to your environments (Yandex Cloud, AWS, DigitalOcean).
Early tests with Gemini hit snags: wrong silences and excessive MCP calls. Switching to Claude with step-by-step thinking mode fixed hallucinations and loops. The final prompt specifies the environment, date, tools, and a strict action sequence.
Key prompt elements:
- Current environment with cloud_id, folder_id, cluster label.
- Tools: vmetrics (instant/range queries from -30m to now), Loki (-30m/-1h), k8s_mcp (context from table), Yandex Cloud Toolkit (Compute/VPC/IAM/YDB/S3 only).
- Alertmanager: get_silences, post_silence (JSON with matchers, ISO dates), delete_silence.
Standard Alert Investigation Protocol
The agent follows a standard playbook:
- vmetrics — metrics (restarts, memory, CPU, free_servers).
- Data table — kubeconfig_context for K8s.
- k8s_mcp — pods_log (previous=true) or pods_get.
- argocd_mcp — get_application by labels.
- Loki — only if no root cause in pod logs.
Call limits:
- No more than 2 retries per tool.
- Two empty responses? Stop.
- 422 error (too many points)? Switch to instant query.
- Stop conditions: OOMKilled/Killed/Error in logs, restarts >0 with reason, 5+ calls.
Alert labels (labelsRaw) are used exactly as provided: project/cluster only if present, no assumptions.
Handling Commands and Silences
@bot commands in English or Russian:
- "silence 2h" / "mute for 4 hours" — post_silence (default 2h, matcher by alertname).
- "unsilence" / "unmute" — delete_silences by alertname.
- "reanalyze" — full analysis.
Silence workflow:
- get_silences() — check existing ones.
- If found — report ID/endsAt.
- If not — post_silence (startsAt=now, endsAt=+duration, ISO Z).
Slack mrkdwn response format:
- Severity: 🔴/🟡/🟢 (by label, never downgrade).
- _Diagnosis:_ root cause.
- _What I found:_ numbers (restarts | exitCode | limit | log).
- _Next steps:_ numbered action list.
Key Takeaways
- MCP protocol gives AI safe read-only access to infrastructure without custom APIs.
- Claude's thinking mode prevents loops and hallucinations with tools.
- Strict stop criteria minimize calls, zeroing in on root causes (OOM, CrashLoopBackOff).
- n8n integration lets AI generate workflows, speeding up development.
- Supports multiple clouds and K8s clusters via context mapping.
— Editorial Team
No comments yet.