Alerting on symptoms not causes.

Alert on What the User Feels, Not on What the Server Feels

I spent three years in academia watching people build incredibly elegant, mathematically perfect monitoring systems that were utterly useless in a production environment. I remember sitting in a server room during a deployment failure, watching a dashboard scream with a thousand different “root cause” alerts—CPU spikes, memory leaks, disk I/O latency—while the actual users were just staring at a spinning loading icon. We were drowning in data, yet we were blind to the reality of the failure. The industry has a bad habit of treating observability like a scavenger hunt for bugs, but if you want to actually sleep through the night, you need to stop chasing the ghost in the machine and start alerting on symptoms not causes.

I’m not here to give you a lecture on theoretical observability or sell you a specific vendor’s dashboard. Instead, I want to walk you through the actual mechanics of why a “root cause” alert is often just noise disguised as intelligence. I’m going to show you how to distinguish between the underlying technical glitch and the actual user-facing failure, so you can build a system that tells you when you’re actually on fire, rather than just telling you the temperature of the stove.

Table of Contents

Observability vs Monitoring Moving Beyond Surface Level Data

Observability vs Monitoring Moving Beyond Surface Level Data

In my experience, the industry often uses “monitoring” and “observability” as if they were interchangeable synonyms, but that mistake is exactly where most on-call rotations go to die. Monitoring is essentially a list of predefined questions: Is the CPU above 80%? Is the disk full? It is a reactive check against known thresholds. If you only monitor these specific metrics, you are essentially building a dashboard of “known unknowns.” You might see a spike, but you won’t understand the why behind it.

Observability, however, is about the ability to ask questions you didn’t know you needed to ask when the system was healthy. It’s the difference between seeing a “check engine” light and having a telemetry stream that shows exactly how fuel pressure dropped relative to throttle position. When we talk about observability vs monitoring, we are really talking about the depth of the context available during a crisis. Instead of just tracking raw numbers, we focus on service level indicators (SLIs) that actually reflect the user’s experience. If the latency is climbing but the user-facing success rate is steady, your monitoring might scream, but your observability tells you the system is actually behaving within its design parameters.

Using Golden Signals of Monitoring to Map Real User Impact

Using Golden Signals of Monitoring to Map Real User Impact

When I was still in academia, we used to obsess over the mathematical purity of a model. In production, that kind of obsession is a liability. If you try to build an alerting suite based on every possible internal state of a distributed system, you aren’t monitoring; you’re just generating noise. This is where the golden signals of monitoring—latency, traffic, errors, and saturation—become practical anchors. Instead of alerting because a specific CPU core spiked to 90% (which might be perfectly normal during a garbage collection cycle), you should alert because the latency experienced by the user has crossed a meaningful threshold.

The goal here is to define your service level indicators (SLIs) based on what actually matters to the end-user experience. If your database replication lag increases, it’s a technical curiosity, but if your checkout service starts returning 500 errors, that is a catastrophe. By mapping your alerts to these signals, you move away from chasing ghosts and toward a system where an alarm actually signifies a broken promise to your users. It turns the chaos of a distributed system into a manageable set of observable outcomes.

Five ways to stop drowning in your own telemetry

  • Stop treating every spike in CPU as a crisis. High CPU is often just a system working exactly as designed, like a car engine revving higher to climb a hill; it only becomes an incident when the car actually stops moving. If your users aren’t feeling a slowdown, your pager shouldn’t be screaming.
  • Define your “burn rate” for error budgets rather than alerting on single, isolated failures. In a distributed system, a single 500 error is noise—it’s a transient hiccup that will likely resolve itself. You want to know when the rate of errors is consuming your reliability budget at a speed that threatens your SLO, not when a single packet gets lost in the ether.
  • Build your alerts around the user’s journey, not the component’s health. A database might be reporting high disk I/O (a cause), but if the checkout service is still completing transactions within 200ms (the symptom), you don’t have an outage yet. You have a capacity planning task for next Tuesday, not a 3:00 AM wake-up call.
  • Use “symptom-based” grouping to prevent alert fatigue. When a core dependency fails, it usually triggers a cascade of secondary failures across your entire stack. If you haven’t configured your alerting to recognize that these are all symptoms of one underlying issue, you’ll end up staring at fifty different notifications when you really only need to look at one.
  • Always include the “so what?” in your alert metadata. An alert that says “Memory usage at 90%” is useless because it lacks context. A useful alert says “Memory usage at 90% AND latency is increasing,” which connects the resource pressure to the actual user impact. If the symptom isn’t manifesting in the service’s performance, the alert is just trivia.

The Mechanics of Meaningful Alerting

Stop building alerts for every CPU spike or memory leak you see in your dashboard. Most of these are just noise—transient states that the system is designed to handle. If you alert on the cause, you’ll spend your life chasing ghosts; instead, alert when the user’s experience actually degrades, like when latency climbs or error rates spike.

Distinguish between “what is happening” and “why it is happening.” Monitoring tells you the disk is full (the cause), but observability tells you the user can’t complete a checkout (the symptom). Your on-call rotation should be triggered by the latter, while the former becomes a task for your post-mortem investigation.

Focus your telemetry on the Golden Signals—latency, traffic, errors, and saturation. These aren’t just arbitrary metrics; they are the direct proxies for system health. If these signals are stable, the underlying “causes” are likely irrelevant to the immediate survival of your service.

The Shift from Reactive to Intentional

At the end of the day, we have to stop treating our monitoring stacks like a collection of random tripwires. We’ve spent too much time building brittle alerts that trigger every time a single CPU spike occurs or a database connection flickers, only to realize the user experience hasn’t changed at all. By shifting our focus toward the Golden Signals and the actual symptoms that manifest as user pain, we stop chasing ghosts in the machine. We aren’t looking for every single way a system can break—that is a mathematical impossibility in a distributed environment—but rather for the specific ways a system fails to fulfill its primary purpose. If the symptoms aren’t visible to the person on the other side of the screen, your alert is likely just noise that will eventually lead to fatigue.

Building a resilient system isn’t about achieving a state of perfect, error-free operation; it is about building the mechanisms of awareness necessary to handle the inevitable chaos. When you stop obsessing over the “why” in the middle of an incident and start focusing on the “what” is actually broken, you reclaim your agency as an engineer. You move from being a firefighter who is perpetually exhausted to a researcher who can methodically diagnose a system. Don’t let the complexity of your infrastructure intimidate you into over-alerting. Trust the symptoms, respect the signal, and build systems that tell you when they are actually hurting, rather than just when they are busy.

About Dr. Ingrid Falk-Weller

I write for the person who wants to understand the mechanism, not memorise the conclusion. If a claim has a caveat, the caveat goes in the paragraph, not a footnote.