The Bill That Grows When Nothing Breaks: A UK Guide to Observability Costs

There is a strange moment in the life of many UK cloud estates when the bill stops tracking the business. The website is not busier, the API is not doing anything new, and nobody has launched a major product. Yet the observability line keeps climbing. Somewhere between a helpful debug log and a dashboard nobody opens, telemetry has become its own workload.

That is not a reason to switch everything off. Logs and traces are how you find out why the checkout button went quiet at 09:12 on a Tuesday. But they are also billable events. Every line you ship, index, query and keep is a small commercial decision, whether or not somebody made it deliberately.

Why the quiet line item gets loud

Observability pricing is rarely one number. You might pay for ingestion, indexing, storage, retention, queries, seats, dashboards, alerts, or some combination that only becomes clear halfway down page four of the invoice. Some vendors bill per host, some per gigabyte, some per million events, and some per active user who wanders into the console after a quiet lunch.

The practical effect is that a team can optimise its compute and database spend for months, then discover that log ingestion has quietly overtaken the lot. It is not usually malice. It is defaults. Applications ship verbose logs in development, then keep shipping them in production because nobody changed the setting. Metrics get labelled with request IDs. Traces get sampled at 100% because someone wanted a complete picture during a migration and never turned it down afterwards.

In the UK, this often surfaces after a cloud migration or a merger. Two estates arrive with two observability habits, and the combined bill looks like a third, more expensive company has moved in.

Retention is a dial, not a moral choice

Most teams treat retention as a compliance question, which it partly is. But it is also a pricing lever. Hot storage is expensive because it is built for fast queries. Warm and cold tiers are cheaper but slower, and sometimes come with retrieval charges that surprise people during an incident. Keeping everything forever in the hot tier is not cautious. It is just expensive.

A better starting point is to sort telemetry into three rough buckets. First, the short-lived debugging data you need for a week or two. Second, operational data you might need for a quarter when a customer complaint arrives. Third, audit or security records that genuinely need a longer life, ideally in a cheaper tier.

The trick is to make that choice per stream, not per vendor. A payments service and a marketing microsite do not need the same retention policy. Put the decision in writing, tag the streams, and revisit it when the invoice changes shape. "We keep everything for a year" is a policy, but it is not always a good one.

Sampling and cardinality: the two levers most teams leave alone

Sampling is not cheating. It is choosing how much evidence you need. Head sampling at the start of a request is simple and cheap. Tail sampling, which decides after seeing the whole trace, is better at keeping the interesting failures while dropping the boring successes. Either way, the point is to stop treating every successful health check as a historical artefact.

Cardinality is the quieter trap. A metric labelled with a user ID, request ID, container ID, or full URL path can create an enormous number of unique time series. The dashboard looks fine. The bill does not. Logs have a similar problem when every line includes a session token or a query string that changes on every request.

You do not need a specialist to spot this. Ask your observability tool for the top metrics by series count and the top log streams by volume. If the winner is something like http_request_duration_seconds with a label called request_id, you have found a likely culprit. Fix the label, not the invoice.

Invoice quirks to read twice

A few pricing habits are worth knowing before renewal. Ingestion and indexing are often separate charges, so sending data is only half the story. Retention overages can be billed per gigabyte per month, which adds up faster than the headline rate suggests. Query charges may depend on how much data a search scans, so a broad dashboard refresh can cost more than a narrow alert. And some platforms charge for seats or advanced features that sit outside the core usage commitment.

None of this is automatically bad value. It is just easy to compare two vendors on the wrong number. When you renew, ask for a worked example using your own volumes: this many gigabytes ingested, this many retained, this many queries, this many users. If the vendor cannot produce it, that is useful information too.

A monthly observability review you can finish

This does not need to become a standing committee. Give it thirty to forty-five minutes, once a month, with one person from engineering and one from finance or procurement. The goal is not to slash spend. It is to stop accidental growth.

1. Pull the observability invoice and split it by service, environment and team tag.
2. List the top ten log streams by volume. Ask which ones are still needed at current verbosity.
3. Check retention for each major stream. Move anything that does not need hot storage.
4. Find the top five metrics by cardinality. Look for IDs, URLs or free-text labels.
5. Confirm trace sampling is deliberate. If it is still at 100%, ask why.
6. Review dashboards and alerts with no views in the last thirty days. Retire or archive them.
7. Check whether any queries or scheduled reports scan more data than they need.
8. Make one change, write it down, and note the expected bill effect.

That last step matters. Observability costs are easy to discuss in the abstract and hard to manage without a baseline. A simple note saying "we reduced debug logs on the payments API and expect to save £X a month" gives the next review something to test.

What good looks like

Good observability spend is not the lowest number on the page. It is spend you can explain. Each team knows which streams it owns, why the retention period is what it is, and what would happen if the data disappeared. The finance team can see the bill without needing a translator. And nobody is paying to store a year of successful health checks just because that was the default in 2022.

The workload should be the thing that costs money. Telemetry should be the thing that helps you understand it. When those two swap places, it is time for a quiet monthly review, a sensible retention policy, and perhaps one fewer request ID in your metrics.