When Your SLO Is Really an Invariant
Report a thing that must never happen as 99.98%, and you have already turned an incident into a statistic.
Consider the system that counts votes in a presidential election. It reads each ballot, records the choice on it, and adds it to a tally that decides who takes office. Speed matters on election night, and so does staying up while the whole country keeps refreshing the page, but neither is the reason the system exists. It exists to count every vote correctly.
Now imagine reporting it the way you would report any service, as a few numbers on a dashboard. Latency, 99.9% of results in under a second. Availability, 99.99% through election night. Accuracy, 99.98%. Three green rows, all in the high nineties, each one reading as a strong result. Sounds reasonable, right? Not quite.
Accuracy, though, is not like the other two. When latency dips below target, some results come back slow, and a slow result is a nuisance you can wait out. When availability slips, the system goes unreachable for a while, and a short outage is something you recover from. Both are failures you can live with in small amounts, which is exactly what a percentage is for. Accuracy is a different kind of thing. Across a hundred million ballots, 99.98% works out to twenty thousand votes counted wrong, and in a close race that is not a rounding error, it can be the outcome. But reported as a percentage, the one thing the system must never get wrong looks pretty good next to the other two.
A correct vote is an invariant. It is a property the system has to hold every single time, not most of the time and definitely not to two decimal places. The distinction between the two is pretty straightforward: an invariant is about correctness, whether the system does the right thing at all, while a service level is about quality of service, how well it does it. Those are not the same conversation, and they do not belong in the same kind of report. But the difference is easy to miss at first, and so a property that is really an invariant often gets filed as one more service level and reported, month after month, as a percentage.
It helps to hold the two apart from the very start. A service level is built on tolerance. It sets a target, budgets the failures allowed below it, and treats a spell under target as something to correct rather than something to panic about. An invariant is completely different. Here are a few examples of invariants you deal with every day:
in banking, money is not created, destroyed, or lost in a transfer
in a shop, your card is charged once for an order, never twice
in messaging, a message reaches the person it was sent to, and no one else
in any app you sign into, you see your own account and never someone else’s
None of these is a property you want to be true most of the time. They are what makes the system that system rather than a broken one, and the day one of them isn’t true any more, you have a very bad day, whatever the latency and availability happen to say.
The word invariant comes from software correctness, and its lineage explains why it must hold every time. In domain modelling, an invariant is what makes an object valid in the first place: an order whose total does not equal the sum of its line items is not really an order any more. In concurrent programming, it is the property you have to restore before you release the lock. In formal verification, it is what holds in every reachable state of the system. Underneath all three is the same idea, that the invariant is what “valid” is defined to mean, so a violation is a bug to find and fix.
There is a simple test for telling one from the other. Ask whether there is some number of failures per period the business would knowingly accept in return for shipping a little faster. For latency the answer is obviously yes, and figuring out that number is more or less the whole SLO conversation. For a corrupted balance or a leaked record the answer is a big NO, and any target short of a hundred percent is faintly absurd. When the only honest target is a hundred, what you are holding is an invariant, and a percentage is the wrong instrument for it.
And the confusion is expensive in both directions. Put an invariant on the SLO list and it turns into 100% target with no error budget, and since the error budget is the engine an SLO runs on, you have quietly jammed it. Worse is what the percentage does to the way people see the thing. “We ran at 99.98% accuracy this month” frames it as a rate to be managed rather than a line never to be crossed, so each wrong vote becomes a number in an average instead of an incident to declare and fix. A percentage on a must-never is also a strange thing to watch. It sits near the top of the dashboard, always green, so after a while it stops carrying any relevant information, and so people stop looking at it altogether. A violation that should have set off an alarm just nudges the average, and by the time one really matters, nobody is watching the number that would have shown it. The mistake runs the other way too, and costs just as much. If you try to push a proper service level all the way to a 100%, the organisation will burn itself out chasing a last fraction of a nine no customer would ever notice or care about.
So you do not budget an invariant, you enforce it and then keep checking it. Enforcement is mostly done through design that makes a violation either impossible or immediately visible: idempotency, checksums, double-entry, immutable logs, access controls. And you use tests to prove the property holds before each release. In production, you can use reconciliation to keep comparing the real data against what has to be true, so anything that slips through is caught fast. The invariant itself never gets an error budget, but the machinery that watches it can, and probably should. For example, “Every vote has an audit record” is the invariant, and “reconciliation finds any missing record within fifteen minutes” is a control with a target you can measure, and that target makes a perfectly good SLO.
So what do you report instead of a percentage? Two things. First, every violation is an incident, one you declare and fix. Second, in place of a success rate, you report on the machinery that keeps the invariant true: the design controls, the tests, the reconciliation that would catch a breach. Does it still cover everything, and how fast would it notice? That second part is the one organizations typically skip, and unfortunately it is the one that matters most. When there are no incidents to declare, the reason is often not that the invariant held, but that nothing is monitoring it in the first place. It is the old dogs-not-barking problem: a quiet guard dog reassures you only if you know the dog is there and awake. So what you want to know is not that the dashboard stayed quiet, but that something was there to bark.
There is one case worth mentioning that looks like an exception but isn’t. At very large scale, engineers do sometimes put a number on an invariant. AWS S3, for instance, is designed for eleven nines of durability, per object, per year. But that number is not an accepted rate of loss, and nobody spends it down the way you spend an error budget. It is a description of how thoroughly loss has been engineered out of the system in the first place.
AWS has treated durable storage as an invariant rather than a target for as long as I can remember. In its paper on how it uses formal methods, the company states the core safety property in those words, “at no time can the system have lost or corrupted any committed data,” and describes using TLA+, a formal specification language, to model-check the fault-tolerant algorithms inside S3, where it found real bugs that ordinary testing would not have reached. More recently, the team building ShardStore, the storage node that actually holds S3 object data on disk, carried the same idea down into the code. They wrote a small executable model of what correct behaviour looks like and checked the real implementation against it with property-based testing and model checking, on every deployment, across hundreds of millions of scenarios that include power loss and crashes. That work caught sixteen issues before they reached production, among them crash-consistency bugs that normal testing had missed.
So none of that is a failure rate anyone is managing. It is an invariant being held by design and re-proven every time the code changes.
It is tempting to use the percentage, because it has become the standard way to report any metric. It’s easy to read, understand, and put on a slide or dashboard, it rounds up, it goes green. But it is important to keep the two apart and decide which of your properties are correctness conditions and which are quality-of-service targets. Hold the first with design and reconciliation instead of a number, and give the second the budget it needs.
So the next time you see a thing that must never happen reported to two decimal places, read those decimal places for what they really are. Somewhere behind them is an incident that nobody ever declared.
// Adrian