Monitoring of Monitoring: Resilience Strategies | IT Practice

# Monitoring Your Monitoring: Strategies to Avoid Recursive Blindness

Systems monitoring creates an illusion of safety: when the controlling tool itself fails, dashboards continue to display outdated data, masking real outages. This isn't a hypothetical issue—real cases with Prometheus show how easy it is to miss critical incidents due to non-functional monitoring. The solution requires a fault-tolerant architecture, but without unnecessary complexity.

Why the Monitoring System Becomes a Blind Spot

When Prometheus or a similar tool stops functioning, the consequences are catastrophic. In one case, the monitoring server's disk filled up due to unlimited Prometheus data storage. In the second, kubelet killed the Prometheus pod due to an OOM error, and Pod Disruption Budget wasn't configured. In both scenarios, dashboards remained “green,” as they displayed data from the last successful scrapes. Alerts didn't trigger, since Alertmanager depended on the non-functional Prometheus.

The key paradox: monitoring can't check itself. If the primary control system fails, there's no mechanism to generate alerts. This isn't an abstract problem—every team implementing advanced observability faces the need to solve this recursive issue. Existing approaches fall into two categories: theoretically ideal (but economically unjustified) and practically applicable (with compromises).

Architectural Strategies for Fault-Tolerant Control

Independent failure domains—a fundamental principle. A failure domain groups components that fail together (VPS, Kubernetes cluster, cloud region). The most common mistake is colocating services and monitoring on the same server. Solution:

• Separate physical server for monitoring (reduces risk but doesn't eliminate it—the shared data center remains a single point of failure)
• Distribution across different clouds (Yandex Cloud + AWS), reducing the probability of simultaneous failure to 0.0001% per month
• Serverless architecture for critical checks (check chain in Cloud Functions, independent of the main infrastructure)

Meta-heartbeat implements a simple watchdog mechanism:

• The monitoring scheduler updates a timestamp in independent storage (Redis, file) every minute
• A separate lambda function checks the timestamp freshness every 5 minutes
• Upon detecting staleness—an alert is sent via an alternative channel (SMS, separate Telegram bot)

Critically important: the watchdog and notification channel must not depend on the same components as the main monitoring. Otherwise, you're just adding another point of failure.

Dead Man’s Switch and Cross-Monitoring

Dead Man’s Switch—a variant of meta-heartbeat with external checking. A service like Healthchecks.io or PagerDuty expects regular pings from your system. Upon their absence, it triggers an alert via a backup channel:

[Monitoring system] --ping--> [External Dead Man's Switch]
                                  |
                            (missed ping)
                                  |
                        [Alert via SMS/PagerDuty]

This approach creates a dependency on a third party, but the probability of your monitoring and the external service failing simultaneously is negligible. For mission-critical systems, cross-monitoring is used: two independent systems check each other. However, maintaining two full solutions (e.g., Prometheus + Datadog) is economically unfeasible for most projects. A more realistic hybrid: main monitoring + primitive watchdog (cron script on a separate server).

Practical Limit: The "Good Enough" Principle

Failure probabilities show that two levels of independence are the optimal point for 99.9% of systems:

• Your VPS goes down: ~0.1–1% per month
• Cloud unavailable: ~0.01% per month
• Two clouds fail simultaneously: ~0.0001% per month
• Three clouds: astrophysical event level

A rational configuration includes:

• Main monitoring in a separate failure domain (SaaS or serverless)
• Dead Man’s Switch in an external service
• Backup notification channels (Telegram + SMS)

The third level of protection doesn't justify the costs, given the already minimal risk reduction. If the main monitoring, dead man’s switch, and Telegram fail simultaneously—you're facing issues where infrastructure control is no longer a priority (global blackout, nuclear war).

Critical Aspects Often Overlooked

Notification channels as a failure domain. Even with functional monitoring, an alert might not reach you due to a Slack or Telegram outage. Solution—duplicate channels: critical alerts sent via SMS and email simultaneously. The probability of two channels failing is an order of magnitude lower than one.

Stateless vs stateful monitoring. Prometheus stores metric history, so data for the downtime period is lost upon failure. Stateless checks ("does the URL respond?") don't have this issue: they resume immediately after recovery. Combine stateful systems (Prometheus) with stateless control (external pings)—they fail differently and cover different scenarios.

Testing the monitoring. Many teams set up the system but never test it in action. Regularly conduct game days: deliberately simulate outages in the staging environment and ensure alerts trigger. It's cheaper than discovering non-functional monitoring during a real incident.

Key Takeaways

• Separate monitoring and monitored services into independent failure domains (different clouds, serverless)
• Implement dead man’s switch with a separate notification channel (SMS/PagerDuty)
• Duplicate notification channels for critical alerts
• Test the control system like you test code and backups
• For 99% of projects, two levels of protection suffice—the third is unjustifiably expensive

— Editorial Team