Spotting a Failing Server Before Your Users Do

The phone call I want to avoid is the one from a member of the public saying “the form on your website is not working.” By then the disk has been full for six hours, the certificate expired last Tuesday, and the service has been returning 502s since the morning rush.

Monitoring is not glamorous. It is also the difference between a fifteen-minute outage and a six-hour one.

The Failure Modes That Actually Happen

In the small and mid-sized networks I look after, the same handful of things cause almost every incident:

• Disk fills up. Logs, backups, or a queue nobody cleaned. The system does not crash — it gets slow, then it stops accepting writes.
• A service dies quietly. A daemon exits at 3am. Nothing restarts it. Nothing tells you.
• A certificate expires. TLS renewals fail silently more often than people expect.
• The link drops or degrades. A fibre fault, a saturated uplink, a misconfigured QoS policy.
• Someone changes something. A config push at 5pm that nobody tested.

Every one of these is trivially detectable, and none of them announce themselves.

Layer 1: Does Anyone Notice?

The baseline question is uncomfortable to ask: if the website went down right now, how long before you knew? If the honest answer is “when a customer tells me”, you have no monitoring — you have a customer relationship with your monitoring.

The floor is a simple external check. An HTTP request to your site every few minutes from a location outside your network, with an alert when it fails. This is not optional and it is not the same as monitoring from inside, because a failure inside your network can hide itself from inside probes.

Layer 2: The Things That Actually Break

Once the outside-in check exists, add checks for the specific components that have taken you down before. There is no point monitoring everything; there is no point monitoring nothing.

For Windows servers and workstations:

• Disk free space on the system and data volumes. Alert below a threshold that is generous enough to leave you time to act.
• Service state for the services that matter — the web server, the database, the backup agent. A stopped service is a page-load error waiting to happen.
• Event log entries for critical errors, filtered to the handful of event IDs that actually indicate trouble in your environment. Unfiltered, the event log is noise; filtered to your own baseline, it is signal.
• Uptime and process health using the Sysinternals Autoruns and PsService utilities when auditing what a machine should be running.

For Linux and network devices:

• CPU, memory, load average and disk I/O wait on hosts. A high wa (iowait) on a database server explains a slow site better than any application log.
• Interface errors and discards on switches. A rising error counter on a fibre uplink is a failing SFP or a dirty connector — find it before the link drops entirely.
• SNMP polling for interface utilisation, uptime and device temperature. On a multi-site network this is the cheapest way to see the whole estate on one screen.
• Service and process checks via SSH, for the handful of services that must always be running.

For everything:

• Certificate expiry, checked with a 30-day warning. This one catches people out constantly and it is trivially preventable.
• Backup job status. Already covered in the backup article, but it belongs in the monitoring list as well as the backup list.

Layer 3: Knowing What Normal Looks Like

The first two layers tell you that something is broken. The third layer tells you something is wrong before it breaks.

Baselining is unfashionable and it pays for itself. Record the normal range for each system: typical memory use, typical CPU, typical request volume, normal bandwidth at 9am and at 9pm. Then set alerts on the deviation, not on an absolute number.

Absolute thresholds produce noise. A file server that normally sits at 30% disk and occasionally jumps to 80% during a month-end process is not an incident; an alert at 80% would train everyone to ignore the channel. A file server that normally sits at 30% and quietly climbs to 85% over three weeks is a genuine problem, and a trend alert catches it in week one instead of week four.

Log aggregation with a query interface (Loki, Graylog, or similar) is the practical way to do this at low cost. The value is not the storage — it is that you can ask “show me every authentication failure on this server in the last hour” across twenty machines in one query, instead of logging into each one.

Alerting: The Part Everyone Gets Wrong

More alerts is not safer. It is the opposite. When a channel fires forty times a day, it gets muted, and then it misses the one that mattered.

Three rules I apply to every monitoring setup:

1. Every alert must have a documented response. If nobody knows what to do when it fires, it is noise. If the response is “nothing, this is known”, it is not an alert.
2. Severity means urgency. Route critical alerts to a channel that gets checked immediately, and informational ones to a channel that gets reviewed daily. Mixing them guarantees the critical one is buried.
3. Every alert must be actionable by one person. “Investigate” is not an action. “Check whether the backup disk on the file server has free space, and free it if it is below 20%” is.

On-call outside business hours is the part that distinguishes a real setup from a demo. If nobody is paged at 2am, the honest thing is to say so and design for it: make the automatic recovery paths good enough that a human is the second line of defence, not the first. Automatic restart on a health-check failure, failover to a secondary host, and a circuit breaker that degrades gracefully are all cheaper than a wake-up call.

The First Weekend

You do not need a monitoring stack. You need a starting point, and you can have a useful one in a day.

Day one: an external uptime check with an alert to a real inbox, plus disk-space and service-state checks on the two or three systems that would hurt most.

Week one: certificate expiry monitoring, and a daily report of the metrics so you start learning the baseline.

Month one: log aggregation, trend-based alerts, and one written restore test.

Quarter one: a review of every alert against the three rules above, deleting the ones that no longer have a documented response.

What to Automate vs. What to Keep Manual

The Honest Test

Monitoring is working if you can answer these without checking a screen:

• What broke, and when did it break?
• Who was affected, and how many people?
• Why did it break?
• How long until you knew?
• What changed so it does not happen the same way again?

If the fourth question makes you hesitate, that is the gap. Everything else is downstream of it.

Want your infrastructure watched properly? I set up monitoring and alerting that matches your actual risk, with documented response steps, so the phone rings for the right reasons. Talk to me about a review.