SLIs, SLOs & Error Budgets

A service-level objective is a reliability decision expressed in measurable terms. It says which user outcome matters, what fraction must be good, and over which population and time window. The objective becomes operational only when a written policy uses the resulting error budget to change engineering or release decisions.

Resource health is evidence about causes; the SLI is evidence about outcomes. CPU, link utilization, queue depth, and replica count may explain why a service is slow, but none is automatically a service-level indicator. Start at the user boundary, then use the dependency graph to localize the failure.

Keep the terms separate #

A service-level indicator (SLI) is a quantitative measure of a service outcome, such as the fraction of valid requests completed successfully within 300 ms.
A service-level objective (SLO) is a target for that indicator over a defined window, such as at least 99.9% over a rolling 28 days.
An error budget is the allowed amount of non-good service implied by an SLO. At a 99.9% objective, the allowed bad fraction is 0.1%.
A service-level agreement (SLA) is an agreement whose breach may have contractual or business consequences. An internal SLO may be tighter than an external SLA to leave operating margin.

Availability is not a synonym for “process is up.” A process can answer health checks while returning wrong data, timing out for one region, or failing a critical operation. Define the outcome from the consumer's perspective.

Specification before implementation #

An SLI specification states the outcome independently of instrumentation:

The proportion of checkout attempts by eligible customers that complete with the correct order within 800 ms.

An SLI implementation adds a measurement source and exact classification:

Count load-balancer requests to the checkout completion endpoint, excluding authenticated load tests; good means HTTP success with the order-confirmed application status and total latency below 800 ms.

Separating them exposes measurement bias. A backend log excludes requests that never reach the backend. A load balancer sees transport outcomes but may not know whether the response was semantically correct. A client sees user experience but may report late or not at all when completely offline. Choose the best feasible observation point and document the blind spots.

Event-based SLIs #

Many SLIs fit a common ratio:

\text{SLI} = \frac{\text{good events}}{\text{valid events}}.

The denominator is part of the product definition. Decide how to treat malformed requests, user cancellations, client disconnects, dependency failures, retries, load tests, and operations outside the supported contract. Excluding a failure because another team caused it usually makes the indicator less representative of the user's experience.

Common event classifications include:

Outcome	Good event	Valid-event population
Availability	request returns an acceptable result	supported requests that reached the chosen observation point
Latency	valid request completes below a threshold	valid requests, often partitioned by operation class
Freshness	response uses data no older than a limit	reads for which freshness is promised
Correctness	result passes a domain-specific check	operations whose truth can be independently determined
Throughput	offered work accepted at or above a promised rate	defined load range and request class

Latency is usually clearer as a threshold ratio—“99% below 300 ms”—than as a mean. A mean can improve while a harmed minority grows. Multiple thresholds can represent an ordinary target and a severe tail without reducing the distribution to one average.

Time-based SLIs #

When discrete user events are not observable, measure good time intervals:

\text{SLI}_{time} = \frac{\text{good intervals}}{\text{valid intervals}}.

The interval width changes the meaning. If a minute is bad whenever any probe fails, one two-second failure consumes a full bad minute. If a minute is good whenever most probes pass, short complete outages may be hidden. State the aggregation rule and use an interval fine enough for the service's failure and response dynamics.

Synthetic probes are controlled observations, not a substitute for all real traffic. They test the path, identity, data, and operation encoded in the probe. A shallow GET /health does not represent a checkout transaction.

From SLO to error budget #

For objective $O$ and $N$ valid events in the window, the allowed number of bad events is

B_{allowed} = (1-O)N.

If a service has a 99.9% success SLO and receives 3,000,000 valid requests in 28 days, it may have

(1-0.999)\times 3{,}000{,}000 = 3{,}000

bad requests. An incident that produces 1,200 bad requests consumes 40% of the window's budget.

For request-based objectives, the final number of valid events in a calendar window may not be known in advance. Compute budget as a fraction and report both counts and ratios. “Minutes of downtime” is only equivalent when the SLI itself is time-based and the badness rule is defined per interval.

Burn rate #

Burn rate compares the observed bad fraction with the allowed bad fraction:

\text{burn rate} = \frac{1-\text{observed SLI}}{1-O}.

At a 99.9% SLO, an observed bad fraction of 1% burns budget at $1\%/0.1\%=10\times$ the sustainable rate. If that rate persisted, the full-window budget would be exhausted in one tenth of the window.

Burn-rate alerts should combine windows. A short window catches severe new failures; a longer window confirms sustained consumption and reduces paging on brief noise. The alert threshold, windows, and required overlap should be derived from how much budget may be spent before a human can act. A raw device threshold such as “CPU over 80%” may be diagnostically useful but is not an SLO alert unless it reliably represents user harm.

Windows and their trade-offs #

Window	Strength	Failure mode
Calendar month or quarter	simple reporting and contractual alignment	resets at an arbitrary boundary; traffic total is unknown until the end
Rolling window	continuously represents recent experience	calculations and explanations are less intuitive
Short alert window	detects a rapid outage	noisy and blind to slow budget leaks
Long alert window	detects sustained degradation	responds late to a catastrophic new failure

Do not silently reset operational memory at a reporting boundary. A service that spends the entire budget on the last day of one month and the first day of the next has harmed users across one continuous incident.

Dependencies belong in the outcome #

If the service cannot complete because a database, identity provider, network, or downstream API is unavailable, the user-visible SLI is bad. Attribution can identify the responsible dependency, but attribution should not repair the numerator. Otherwise every team can report green by excluding failures caused elsewhere while the end-to-end service remains broken.

Use the dependency map for engineering:

find which dependency consumed the most budget;
identify hard dependencies that could be cached, replicated, bypassed, or degraded by an explicitly designed product behavior;
compare dependency objectives with the parent service objective;
test whether redundant dependencies share network, power, identity, control-plane, or deployment failure domains.

A fallback or degraded response changes the definition of good. Specify what the user sees, which data may be stale or incomplete, when the behavior activates, and how budget accounting treats it before implementation.

Error-budget policy #

An error budget without a policy is a report. A policy should name:

service owner, SLO approvers, review date, and scope;
exact SLI implementation and data-quality checks;
budget calculation and window;
actions at warning, critical, and exhausted states;
exceptions, who can authorize them, and how they are recorded;
exit criteria for resuming normal change;
escalation when stakeholders dispute the measurement or action.

Actions should reduce user risk. Examples include stopping a rollout, prioritizing a recurring failure class, or requiring a postmortem after one event consumes a large fraction of the budget. The policy is not a punishment and should not reward hiding valid bad events.

Worked design: checkout latency and success #

Suppose checkout has two objectives over a rolling 28-day window:

99.95% of eligible attempts return a correct confirmed-or-declined result; infrastructure errors and timeouts are bad.
99% of valid attempts complete within 800 ms.

Implementation steps:

Define eligibility and distinguish business declines from infrastructure failure.
Measure near the client-facing edge and propagate a request identity into traces.
Keep success and latency as separate indicators; a fast error is not a good latency event.
Partition only when the product contract differs, such as interactive checkout versus batch reconciliation.
Compare bad-event exemplars with traces and the service dependency map.
Page on multi-window burn rate; use resource alerts as diagnostic or ticket signals unless immediate action is independently justified.
Review excluded events and missing telemetry as part of SLI quality.

Review checklist #

Can the SLI be explained as a user outcome without naming the monitoring tool?
Are good and valid events mechanically classifiable?
Is the measurement point stated, including what it cannot see?
Are thresholds, population, and time window part of the objective?
Are dependency failures counted from the user's perspective?
Are missing, late, and duplicate telemetry handled explicitly?
Does burn-rate alerting give enough time for an effective action?
Does a written policy change decisions when the budget is consumed?
Is the SLO reviewed when product behavior, traffic, dependencies, or instrumentation changes?