A crash loop looks like an outage from a distance: a service is unreachable, restarting doesn't help, and the failure repeats on a cycle. The difference matters for how you respond. An outage needs investigation. A crash loop needs a circuit breaker.
We found this gap while chasing a different problem — fleet-wide slowdowns with no clear cause. Crash loops were a reasonable suspect; restart bursts produce the kind of erratic load that looks like the thing you're chasing. They weren't. But several long-dead projects turned out to be crash-looping anyway, 160+ failed starts a week each, invisible until we went looking.
The signal to watch
systemd (and most process supervisors) already limit a single burst of restarts, then reset. That's not detection — it's just a rate limit with no memory. The actual signal is the repeat: watch the same “restarted too quickly” event fire again after a cooldown. One burst is normal recovery. Three bursts in a trailing window is a pattern.
That distinction is the whole trick. A longer timeout or a bigger burst limit doesn't fix anything — it just delays the same unbounded loop.
The case that's actually hard: you can't just stop calling it
If you host other people's services, some of them will be background workers wired to a webhook, not something a human visits. That's the case a simple “sleep when idle” policy doesn't cover.
A dead web app just waits; nobody visits, nothing happens. A dead bot still gets called — by a chat platform, a payment webhook, whatever integration fired it originally — on an external schedule you don't control and can't distinguish from a legitimate one. You can't refuse the call to avoid waking a dead process, because it might not be dead. So the wake has to happen, it crashes, and the next call rearms it.
Sleeping to save resources assumes waking succeeds. When it doesn't, the same mechanic that saves resources on a healthy service converts a dead one into recurring bursts instead of steady load — worse for shared infrastructure, not better, because bursts collide with other tenants' real traffic at the exact moment they fire.
What we do about it
Watch each unit's own restart-limit signal. Three within a trailing window trips a breaker:
- Mask the unit — nothing, including the next webhook call, restarts it.
- Return a clear, deliberate response instead of a raw error.
- Notify the owner once, not on every future occurrence.
Recovery is a deploy, not a manual reset. A successful deploy is the owner's own signal that they believe it's fixed; if it isn't, the breaker trips again on its own.