How We Built an Alert System That Actually Stops Alert Fatigue
Alert fatigue is the silent killer of on-call reliability. Here's how we redesigned alerting from first principles to eliminate false positives.
Alert fatigue is one of the biggest problems in modern observability. When your team receives hundreds of alerts per day, they stop paying attention to any of them. The result? Critical issues slip through while engineers burn out from constant noise.
The Problem With Traditional Alerting
Most monitoring tools treat alerting as a simple threshold problem: if metric X exceeds value Y, fire an alert. This approach has three fundamental flaws:
- Static thresholds can't adapt to natural traffic patterns
- Every alert feels equally urgent, even when most are noise
- Alert rules multiply over time until nobody knows what each one does
We talked to 200+ SRE teams and the pattern was consistent: teams either had too many alerts and ignored them all, or too few and missed real incidents.
How Our Alert Pipeline Works
Instead of raw threshold comparison, every metric passes through a multi-stage pipeline before it becomes an alert:
Our Approach: ML-Powered Baselines
Instead of static thresholds, UptimePulse learns what "normal" looks like for each metric on your system. Here's the core idea:
baseline = rolling_average(metric, window=7d) * sensitivity_factor
alert = if current_value > baseline AND confidence > threshold
The sensitivity factor is configurable per service, and the confidence score ensures we only alert when the anomaly is statistically significant.
Key Design Decisions
Time-aware baselines. A web API behaves differently on Monday morning vs. Sunday night. Our baselines account for hourly and day-of-week patterns.
Multi-signal correlation. A single metric spike is often noise. But when latency spikes correlate with error rate increases AND memory pressure, that's a real incident. We score alerts based on how many signals agree.
Automatic resolution. Alerts clear themselves when the anomaly resolves. No manual acknowledgment required for transient issues.
Results in Beta
During our 3-month beta with 40 teams:
- 94% reduction in alert volume (from ~500/week to ~30/week per team)
- Zero missed critical incidents across all beta teams
- Mean time to acknowledge dropped from 12 minutes to 2 minutes
The teams that switched to UptimePulse reported something we didn't expect: they started trusting their alerts again. When every alert matters, engineers actually want to look at them.
What's Next
We're expanding our anomaly detection to cover infrastructure metrics (CPU, memory, disk I/O) and adding team-level learning so the system understands the difference between "this is fine for this service" and "this is genuinely anomalous."
The goal isn't to replace human judgment — it's to make sure humans only have to exercise judgment when it actually matters.