Automating Incident Diagnosis: Building an AI Assistant with n8n and LLMs
AI agents can handle the initial analysis of system alerts, slashing response times for your team. We'll walk through a practical implementation of an assistant using the n8n platform, LLMs, and the Model Context Protocol (MCP) for seamless integration with your infrastructure tools.
Architecture of the AI Assistant for Alert Handling
The system revolves around a central n8n workflow triggered by a webhook from Alertmanager. Key components include:
- Webhook trigger to receive events from monitoring systems
- Preprocessing node (Code node) to format alert data
- AI Agent node connected to an LLM (Codex or Opus recommended)
- Set of MCP servers for interacting with external systems
- Vector store (Qdrant) for quick access to documentation
- Chat integration (Slack) to deliver results
The standout benefit? The agent kicks off diagnostics the moment an alert fires—no waiting for an engineer.
Setting Up Data Sources with Model Context Protocol
MCP servers give the agent secure access to your infrastructure without custom-coded integrations. Common ones include:
- mcp-grafana — for querying metrics and logs
- kubernetes-mcp — to check pod status and cluster events
- digitalocean-mcp — for monitoring cloud resources
- gitlab-mcp — to review related CI/CD releases
Run each MCP in remote HTTP streaming mode. The agent's system prompt should include clear instructions on using each tool, plus restrictions (like no modifications).
Configuring the Knowledge Vector Store
Qdrant acts as a cache for frequently accessed info, saving LLM tokens and speeding up diagnostics. Setup steps:
- Create a collection via the Qdrant web UI (port 6333)
- Set up authentication with a Service API token
- Connect the store as a tool to the AI agent
- Load documentation via a dedicated n8n workflow
Typically, store:
- Alert descriptions and trigger conditions
- Service architecture docs
- Historical incident reports
- Configuration references
Diagnostic Process and Report Format
The analysis follows an experienced engineer's playbook:
- Trigger metric analysis — pinpoint the anomaly type (spike, ramp-up, sustained critical)
- Failure differentiation — app bug or infrastructure issue?
- Resource checks — CPU/memory load, network metrics, system limits
- Log review — hunt for errors in affected components
- Change analysis — scan recent releases and maintenance
- Direct verification — probe components via orchestrator or shell
Reports are human-friendly:
- Concise summary of findings
- List of likely root causes
- Fix recommendations
- Links to dashboards and logs
Results post to the alert's Slack thread, keeping context for team discussion.
Building the n8n Workflow in Practice
The workflow flows sequentially:
- Alert intake — Webhook node from Alertmanager
- Preprocessing — Code node to pull key fields: Alertname, Description, Labels, namespace, instance, env, region, Grafana Dashboard and Runbook links
- Diagnostics — AI Agent node with a system prompt detailing infrastructure and analysis rules
- Context lookup — queries to MCP servers and vector store
- Result delivery — Slack API integration to post the report in-thread
Average alert processing? About 30 seconds—enough time to check metrics, logs, Kubernetes cluster state, and generate a conclusion.
Key Takeaways
• AI agents don't replace engineers—they handle routine initial triage, cutting time from alert to insight
• MCP servers enable safe access to infrastructure without handing LLMs direct credentials
• Vector stores save tokens and boost speed by caching common info
• n8n offers a visual builder for complex workflows without heavy coding
• Shines during off-hours, when on-call engineers might not respond instantly
Testing and Optimizing the Assistant
When deploying:
- Start with non-critical alerts to validate logic
- Track LLM token usage to manage costs
- Set timeouts for MCP requests
- Refresh vector store knowledge regularly
- Review recommendation accuracy to refine prompts
Real-world wins: spotting container memory issues, network latency, deployment failures, and Kubernetes resource contention.
— Editorial Team
No comments yet.