The Rota Doesn’t Lie: A UK Team’s On-Call Resilience Audit

Most cloud resilience plans are written in daylight. They describe architectures, backup windows and a reassuring diagram with arrows. The on-call rota is different. It is the part of the plan that gets tested when a phone lights up at 3am, the laptop is upstairs, and the person on duty has a toddler asleep next door. If you want to know how your UK team would cope with a real incident, don’t start with the runbook. Start with the rota.

An honest audit hiding in plain sight

A rota tells you who is expected to respond, how often, and whether the organisation has thought beyond the first name on the list. It also exposes the gaps that diagrams tend to miss: decision rights, access, local knowledge and fatigue.

Ask a simple question: when the pager goes off tonight, who wakes up, what can they do before anyone else joins, and how long will it take them to find the right dashboard? If the answer is vague, you have found a resilience issue before an outage has even happened.

A rota that lists names but not responsibilities is a phone tree, not an escalation path. A rota that depends on one senior engineer every night is not a rota either. It is a single point of failure with a mobile phone.

Pager fatigue is a leading indicator

Noise is the quiet killer of on-call culture. If the same person gets woken for a disk warning that self-clears, a certificate that has months left, or a test alert from a forgotten staging environment, they will eventually sleep through the one that matters. Not because they don’t care, but because human attention is finite.

UK teams often run lean. A handful of engineers may cover cloud platforms, networking, identity and a legacy system nobody wants to own. In that setup, every unnecessary page has a cost. It erodes trust in the alerting system and makes people less likely to respond quickly when the alert is real.

A useful review question is not ‘how many alerts did we get?’ but ‘how many pages required a human action within 15 minutes?’ If the answer is very few, the pager is doing too much talking.

Handover is where resilience leaks

Incidents rarely respect the working day. A problem that starts at 16:55 on a Friday can drift into the weekend if ownership is unclear. A bank holiday Monday can turn a manageable issue into a Monday morning surprise because nobody knows who is holding the baton.

Good handover is boring and explicit. It names the current owner, the known symptoms, what has already been tried, what not to try, and the next decision point. It also says who is covering if the primary contact is unavailable. Without that, the on-call rota is only a timetable. It is not a system of ownership.

Try a handover test: ask the person coming on call to explain the state of a live issue in two minutes using only the notes. If they cannot, the notes are not ready. If they have to call someone who is now off duty, the handover is not ready either.

The UK working-time reality

On-call is not just a calendar problem. It sits alongside the Working Time Regulations 1998, rest requirements and the simple fact that people need sleep. Whether on-call time counts as working time can depend on how restricted the person is while waiting. That is a legal and HR conversation, not one to settle in a Slack thread.

What you can do is look for obvious strain. Is the same person covering too many nights in a row? Are shifts followed by a normal working day? Does the rota assume someone will answer during a school run, a commute or a hospital appointment? If the answer is yes, the rota is relying on goodwill rather than design.

The practical fix is often unglamorous: a wider rotation, clearer secondary cover, fewer non-actionable alerts, and a rule that the person who handled an overnight incident is not also expected to lead the 9am review. That is not softness. It is how you keep the rota sustainable.

Runbooks that survive contact with reality

A runbook is not resilient just because it exists. It has to be usable at 3am by someone who is half awake and not the person who wrote it.

Check the basics. Do the links work? Are the credentials available in the right vault? Does the runbook say which account, which region, and which environment? A runbook that says ‘log into the console’ without saying which console is a riddle, not a procedure. A runbook that assumes a permission the on-call engineer does not have is a trap.

The best test is a dry run. Give the rota to someone who has not handled that service before and ask them to walk through the first three steps. You will learn more about resilience in 20 minutes than you will from a month of status meetings.

Escalation paths outside office hours

Escalation is not just a list of names. It is a set of rights. Who can declare an incident? Who can wake up a vendor? Who can approve emergency spend? Who can communicate with customers? If those answers only exist during office hours, your escalation path has a business-hours-shaped hole in it.

Vendor support deserves a separate look. A 24/7 support line may be a ticket intake desk with an automated reply, not an engineer who can act. Before you need it, find out what happens after you log a priority-one case at 2am. Does someone call you back? Do they have access to your environment? Can they escalate internally? The time to discover this is not during an outage.

Measure what matters

Metrics should make the rota better, not turn it into a leaderboard. Useful measures include time to acknowledge, time to first meaningful action, the percentage of pages that led to a change, false-positive rate, and how often handover notes were complete enough to be used.

Avoid reducing incident response to a single number. A fast acknowledgement that leads nowhere is not success. A slow acknowledgement that comes with a clear plan may be better. The point is to find the weak links: noisy alerts, missing access, unclear ownership, or a rota that quietly depends on one person.

A simple review you can run this month

Pick one service and one week. Map who was on call, what they were paged for, and what they could actually do. Then ask five questions:

• Could the on-call person take the first meaningful action without waking someone else?
• Were any pages non-actionable, duplicated or self-clearing?
• Was handover written down and understood by the next person?
• Did the escalation path work outside UK office hours, including bank holidays?
• Did anyone carry an unfair share of nights or follow an overnight incident with a full day?

The answers will not give you a perfect resilience score. They will give you something more useful: a list of fixes that make the next 3am less dramatic. And that is the whole point of an on-call review. The rota does not lie. It just waits for you to read it.