Testing Strategy: Scope, Cost & Evidence

Automated tests do not prove that software is correct. Each test establishes a narrower claim: under a particular setup, through a particular interface, an observation agreed with an oracle. A testing strategy is therefore not a target percentage or a fixed pyramid. It is a portfolio of experiments whose combined evidence covers the failures that matter while returning useful feedback quickly enough to change decisions.

Describe the test before naming it #

Terms such as unit, integration, functional and acceptance are used inconsistently. A name is useful only after five properties are explicit:

A test that calls one repository against PostgreSQL is narrow in application scope but real at the database boundary. A browser test against five stubbed services is broad in UI scope but synthetic below the HTTP boundary. Calling the first an “integration test” and the second an “end-to-end test” says less than declaring those facts.

Scope trades diagnosis for realism #

flowchart LR
  U["Small scope<br/>one behavior"] --> C["Component boundary"] --> I["Real integration"] --> E["End-to-end journey"]
  U -. "faster, more specific" .-> C
  E -. "broader, more production-like" .-> I

Smaller tests usually run faster, isolate failures better and control more variables. Broader tests exercise more wiring, configuration and runtime behavior, but a failure has more possible causes and more sources of nondeterminism. Neither end is intrinsically superior. The engineering rule is:

This produces common scopes, not a universal ladder:

• a unit test exercises a deliberately small behavior with controlled collaborators;
• a component or service test crosses the component's public boundary while replacing dependencies outside it;
• an integration test exercises a real boundary between selected parts, such as application code and its production database engine;
• a system or end-to-end test follows a meaningful journey through the assembled system;
• an acceptance test evaluates an example of required behavior. It may execute at any technical scope, including below the UI.

Functional versus non-functional is a different axis. Latency, capacity, security, recovery and usability tests can each be narrow or system-wide. “Black box” versus “white box” describes what the test knows, not how much it executes.

Allocate tests by failure mechanism #

Start from a failure and ask where it first becomes observable.

The familiar test pyramid is a cost heuristic: many cheap, focused checks; fewer expensive, broad checks. It is not a quota. A compiler, data pipeline, mobile client and distributed control plane have different dominant risks, so their healthy portfolios have different shapes.

A feedback pipeline, not one giant suite #

A useful suite is staged by information value:

1. During editing, run deterministic tests directly related to the changed behavior.
2. Before merge, run the fast portfolio plus real boundary integrations affected by the change.
3. In a hermetic CI environment, run broader component and system checks.
4. Against a deployed candidate, verify configuration, migrations and a small set of critical journeys.
5. In production, use canaries, synthetic probes and observability for properties that a pre-production environment cannot reproduce.

Each stage should preserve the evidence needed to diagnose failure: the seed and input, expected and actual values, version/configuration, relevant logs, traces and state. A bare red status from a broad test is weak evidence because it identifies neither the broken claim nor the responsible boundary.

Reliability is part of correctness #

A test that can pass or fail without a relevant change has an invalid signal. Larger tests correlate strongly with flakiness because they include more processes, resources and timing relationships. Common causes are shared mutable state, real wall-clock sleeps, unordered assumptions, unpinned environments, external services and eventual consistency observed without a bounded condition.

Retries are useful for measuring and containing this damage, not for converting failure into success. Record the first failure and classify a pass-on-retry as flaky. Otherwise the suite teaches the team that red means “run it again,” and genuine regressions become routine noise.

Worked portfolio: placing an order #

Suppose checkout validates inventory, prices a basket, stores an order, charges a payment provider and shows confirmation.

• Unit tests cover price arithmetic, promotion precedence and idempotency-state transitions.
• A database integration test applies real migrations and proves uniqueness and transaction behavior.
• A payment contract test checks request/response compatibility; application tests use a controllable provider stub for declines and timeouts.
• A component test drives the checkout API with real serialization and the selected database.
• One browser journey proves that the assembled UI can place an order and display the resulting identifier.
• A deployment probe confirms that the running environment reaches its database and payment sandbox.

Writing twenty browser variants for pricing rules would slow diagnosis without adding realism to the arithmetic. Omitting the browser journey would leave routing, cookies, form submission and UI wiring unproved. The portfolio is complete when each important risk has a credible observation point, not when a diagram has the “right” proportions.