Alert Threshold Recommender

Alert fatigue occurs when teams receive so many notifications that they start ignoring them. Studies show that over 30% of monitoring alerts are never investigated, and teams with high alert volumes have slower incident response times. The root cause is usually static thresholds that do not account for normal variation in system behavior.

Sigma-based thresholds solve this by deriving alert boundaries from your actual baseline data. A 2-sigma warning threshold means the value is outside 95.4% of normal observations -- statistically significant but not rare. A 3-sigma critical threshold triggers only for observations beyond 99.7% of normal -- genuinely anomalous. This approach adapts naturally to your system's behavior rather than relying on guesswork.

Multi-window alerting adds another layer of noise reduction. Instead of alerting on a single 5-minute window, combine a short window (5m) for acute issues with a longer window (1h) for sustained degradation. Google SRE recommends this approach: alert only when both windows are in violation, dramatically reducing false positives while still catching real incidents.