Automation without a runbook fails silently and surfaces in front of a customer. What belongs in a runbook, which alerts are genuinely needed, and thresholds without noise.
Key takeaways
- The expensive failure is not the one that alerts but the one that passes silently.
- A runbook is one page: what broke, how to spot it, what to do manually, who handles it.
- An alert with no defined recipient is identical to not alerting at all.
- Most alerts are unnecessary; two correct ones beat twenty nobody reads.
- You also have to alert when nothing happened - a process that stopped running produces no error.
The question nobody asks during an automation project is what happens when it fails. Not if - when, because any process depending on an external system will fail eventually. The difference between an hour of disruption and a week-long problem is whether somebody knew, knew what to do, and was defined as the owner.
Three kinds of failure, and the dangerous one
| Failure type | What it looks like | The risk |
|---|---|---|
| Visible error | The process stopped and reported | Low - you know |
| Partial failure | Some records went through | Medium - partial data |
| Silent failure | The process stopped running | High - nobody knows |
The third is the one that hurts. A process that did not run produces no error - it simply does not happen, and the result surfaces when a customer asks why they did not receive something, sometimes weeks later. The only way to catch it is an alert on the absence of activity rather than on an error, explained fully in silent failures in Zapier and Make.
What belongs in a runbook
- The process name and what it does, in one line.
- When it runs - trigger or schedule.
- How you know it worked - where to look.
- What to do manually if it did not run today.
- Who handles it - a name, not a role.
- Who to update if it lasts beyond a day.
The fourth point is what makes a runbook useful. Most business processes have a reasonable manual fallback for a day or two, and whoever knows it does not need to wake anybody at 2am. What creates panic is not the failure but the absence of a known alternative.
Which alerts are genuinely needed
Three, for most processes:
- The process failed - an error stopped the run.
- The process did not run when it should have - no run in the expected window.
- Volume anomaly - it ran but handled far fewer or far more records than usual.
The third is what catches partial failure. A process that usually moves 40 records and moved 3 today did not technically fail, but something there is broken - and that is exactly the alert that prevents late discovery.
Setting a threshold that does not create noise
An alert arriving daily stops being an alert within a week. So the threshold has to be based on what was measured rather than on a feeling: run for two weeks, observe the normal range, and set the threshold outside it.
Two rules help: alert on a trend rather than a single event (three errors in an hour, not one), and suppress repeat alerts for the same fault until it is resolved. The general logic of thresholds without noise is in business event alerting thresholds.
Where the alert goes
Not to a shared mailbox. An alert sent to info@ is an alert nobody reads, and that is one of the most common reasons for a silent failure despite monitoring existing.
What works: a channel the owner genuinely reads, with their name clear in the message, and the required action in the body of the alert itself. An alert saying "Scenario 14 failed" without saying what that is or what to do will reach somebody who does not know what to do with it, and get postponed to tomorrow.
Who handles it when there is no IT person?
In a small business the owner will usually be the business owner or whoever is closest to the process, and that is fine - provided the runbook is written so it needs no technical knowledge. That is why the "what to do manually" line is the main one: it lets the business keep functioning even when the fix waits for a business day.
What does need defining in advance is who to call for the fix - the supplier who built it, whoever maintains it, or an external person - and what response time is agreed. The moment of the fault is not the moment to work that out, and the broader ownership logic is in who maintains the automation.
Why retries are not a complete answer
The first reflex after a failure is to add a retry, and that is right for most faults: a system that did not respond for a second will respond on the next attempt. But a retry only solves one class of failure - the temporary one - and it can make the second class worse.
Two situations are worth knowing. First: a process that writes a record and fails after the write but before the confirmation. A retry will create the same record a second time, and that is how the duplicates somebody cleans up by hand later are born. The prevention is a unique identifier per operation, so the other side can recognise a repeat.
Second: a permanent failure - an expired permission, a changed field, a revoked key. Here a retry only delays the alert and multiplies the errors in the log. So cap the number of attempts and alert after the last one, rather than retrying forever in silence.
A quarterly check worth running
Once a quarter it is worth verifying the monitoring itself works: deliberately switch off one non-critical process and see whether the alert arrived, and to whom. It is a ten-minute check, and it finds problems there is no other way to find - a changed address, an expired alerting integration, a channel nobody reads any more.
In businesses that have never checked, this test frequently fails the first time. That is not a sign of negligence but of the fact that monitoring ages like everything else - people change, channels change, and nobody is notified that an alert stopped being delivered.
Sources
Frequently asked questions
Does every automation need a runbook?
Every one that stops something when it fails. A convenience process nobody would miss for a day does not need one, but it is worth recording that it is such a process - so it is not mistakenly treated as critical when something happens.
How long does writing a runbook take?
Ten minutes per process. It is one page of six lines, not a document. A runbook longer than a page is not read at the moment it is needed, which is the only moment it was written for.
What do you do when alerts arrive constantly?
Do not mute them - fix the threshold or the cause. An alert arriving daily is usually a wrong threshold or a standing fault nobody addressed, and both produce the same result: total deafness to every alert.
Is the automation tool enough as monitoring?
Partly. Most tools alert on errors but not on an absent run, which is precisely the dangerous failure. A check outside the tool - even a daily query confirming records were created - closes that gap.
Keep reading
Related service
Business Automation
I build custom automations that remove repetitive work end to end.
About the author
Yehonatan Saadia
Freelance automation, web & MVP developer
I'm Yehonatan Saadia, a senior developer who builds business automation, custom websites, and MVPs for small and mid-sized companies across the US, Europe, and Israel. These guides come from real client work, not theory.
Work with meHave a project like this?
Tell me what you're trying to automate or build and I'll tell you the fastest reliable way to ship it.