Network Telemetry & Observability

Network telemetry is evidence about what a network and its protocols are doing. It includes familiar device counters, but it also includes routing state, flow records, logs, active measurements, sampled or captured packets, and observations produced inside the forwarding path. These sources answer different questions. A dashboard becomes trustworthy only when it preserves the observation point, time semantics, selection policy, and known loss or sampling limits of each one.

The durable mental model is source → acquisition → transport → normalization → storage → analysis. Protocol names belong to particular arrows in that pipeline; no single protocol supplies complete network observability.

Start with the question and observation point #

A signal can be accurate and still answer the wrong question. Interface counters observed on a router can show congestion or discards at that interface, but cannot establish whether a user request completed. A server-side latency metric misses requests that never reached the server. A packet capture at one link sees neither traffic routed elsewhere nor processing that occurs after the capture point.

Write an observation contract before selecting a tool:

1. Object: interface, queue, route, adjacency, flow, packet, endpoint exchange, or service outcome.
2. Observation point: device CPU, forwarding ASIC, collector, client, server, or a particular link direction.
3. Time: event time, export time, receive time, clock source, precision, and acceptable skew.
4. Selection: full population, filtered subset, sampled population, or aggregation.
5. Failure semantics: what disappears when the device, transport, exporter, or collector fails?

This contract prevents a common mistake: treating “not present in the collector” as “did not happen.” Absence may instead mean filtering, sampling, packet loss, exporter restart, template loss, clock error, or an uninstrumented path.

Four planes of evidence #

RFC 9232 organizes telemetry by where data originates:

The planes overlap but are not interchangeable. A BGP route in the RIB does not prove the FIB programmed it. A programmed FIB entry does not prove packets arrived. A rising interface counter does not prove the intended application used that interface. Strong diagnosis links evidence across the planes.

Measurement families are complementary #

Managed objects and structured state #

SNMP exposes managed objects through MIB modules and supports command/response plus asynchronous notifications. It remains useful for broadly implemented counters and status, but polling cadence, counter width, security model, and vendor MIB semantics matter. SNMPv3 adds message security and access control; a community string in an older deployment is not equivalent protection.

YANG-modeled state retrieved or subscribed through NETCONF, RESTCONF, YANG-Push, or gNMI gives machines typed paths and schema context. Structured data improves automation, but a path name alone does not guarantee identical device support, update cadence, or implementation quality. Capability discovery and schema versions belong in the evidence.

Events and logs #

Syslog carries discrete reports with facility, severity, timestamp, hostname, application, process, message identity, and optional structured data. It records what an originator chose to announce; it is not a reliable state database. RFC 5424 separates message format from transport and recommends the TLS transport defined by RFC 5425. UDP transport can lose, reorder, or truncate evidence precisely during overload.

Events need stable identities and state transitions. Repeated “link down” messages without the corresponding interface identity, prior state, sequence, and later recovery cannot support dependable lifecycle reconstruction.

Flow export #

IPFIX exports Flow Records from a Metering Process through an Exporting Process to a Collecting Process. Templates define each record's fields, so the collector must associate data with the correct template and observation domain. Flow export is compact enough for traffic accounting and conversation analysis, but aggregation, active/inactive timeouts, sampling, exporter clock, and lost records bound what can be inferred.

A flow is an exporter-defined aggregation, not a transport connection. Several TCP connections may be grouped when keys omit distinguishing fields; one connection may produce multiple records after timeout or path changes.

Packets and active probes #

Packet capture gives protocol-level evidence at a specific observation point. It is the strongest source for malformed messages, retransmissions, handshakes, and on-wire timing, but volume, privacy, encryption, offload behavior, and capture loss limit it. Active probes supply a controlled workload and can measure reachability, delay, loss, and path behavior even when production traffic is absent. A probe is representative only if its source, destination, policy, packet size, class, and timing match the question.

Pull, push, periodic, and on-change #

Polling samples a state at client-selected times. If a fault appears and clears between polls, it is invisible. Faster polling reduces that blind interval but increases device, network, and collector load. Independent pollers can also synchronize into bursts.

Subscription protocols move the scheduling decision into an explicit stream. YANG-Push supports periodic and on-change datastore subscriptions. gNMI supports ONCE, POLL, and long-lived STREAM subscriptions; an individual streamed path can use target-defined, periodic sampling, or on-change behavior. Streaming removes repeated request overhead, but it does not remove correctness work:

• establish an initial synchronized snapshot before applying deltas;
• retain source timestamps rather than replacing them with collector arrival time;
• detect sequence gaps, reconnects, exporter resets, and resynchronization;
• bound queues and define what happens under backpressure;
• verify whether “on change” is supported for every selected path;
• avoid requesting an update rate that the source cannot sustain.

Counter arithmetic #

A monotonically increasing counter becomes a rate only after comparing two valid observations from the same counter instance:

$$r = \frac{C_2-C_1}{t_2-t_1}.$$

Before using the result, check:

• $t_2>t_1$ and both timestamps use compatible clocks;
• the device, interface, or process did not restart;
• the counter did not wrap;
• its unit is known: packets, octets, events, or time;
• the interface identity did not get reused;
• discontinuity metadata, if provided, did not change.

For an interface whose line rate is $B$ bits/s, a first approximation to transmit utilization over the interval is

$$U = \frac{8\,(O_2-O_1)}{B\,(t_2-t_1)}.$$

This measures octets counted by that interface, not necessarily application goodput. Framing overhead, retransmission, encapsulation, compression, asymmetric traffic, and short queue bursts may make the operational question require other signals.

Worked interval #

An interface octet counter increases from 1,250,000,000 to 1,400,000,000 over 10 seconds on a 1 Gbit/s link:

$$U = \frac{8(150{,}000{,}000)}{10^9(10)} = 0.12.$$

The mean utilization is 12%. That does not rule out microbursts: a queue can overflow during short peaks while the interval average stays low. Queue-drop counters, finer-grained telemetry, or packet evidence are needed to test that hypothesis.

Freshness, cardinality, and aggregation form a budget #

Higher resolution is not free. Let $N$ be the number of observed series, $f$ their update frequency, and $s$ the encoded bytes per update. The raw ingress rate is approximately

$$R = Nfs.$$

Increasing labels, paths, or interfaces raises $N$; faster sampling raises $f$; verbose encodings raise $s$. The pipeline must budget device work, management-network capacity, collector memory, storage, query cost, and retention together. When overload forces data loss, the collector must expose the loss rather than silently presenting a sparse stream as complete.

Aggregation should follow the diagnostic question. Keep enough dimensions to distinguish direction, interface, queue, protocol, and failure domain when those facts drive action. Remove dimensions that identify users or create unbounded cardinality without operational value.

Correlation is a claim, not a join #

Shared timestamps and labels make two signals comparable; they do not make one the cause of the other. A service latency spike and interface discards on the same path support a network hypothesis when:

1. the dependency map says the affected requests used that path;
2. the discard interval overlaps the user-impact interval after clock uncertainty is considered;
3. direction and traffic class match;
4. an independent signal—queue depth, packet loss, retransmission, or an active probe—agrees;
5. alternative shared causes, such as load or collector delay, are tested.

Use telemetry to reduce uncertainty in a causal story. Do not convert a coincident graph into an automatic root-cause verdict.

Security and privacy #

Management data reveals topology, software versions, interface descriptions, peer addresses, traffic volumes, and sometimes user-identifying flow or packet fields. Protect management endpoints with authenticated and encrypted transport, least-privilege authorization, isolated reachability, credential rotation, and audit logs. Treat collectors as sensitive infrastructure.

RFC 9232 warns that large-scale collection of end-user data can become pervasive monitoring. Define a purpose, minimize identifying fields, restrict access, and set retention before collection. Packet payloads and fine-grained flow records deserve stricter handling than aggregate device health.

Diagnostic checklist #

• State the operational question and the exact observation point.
• Separate desired configuration, control-plane decision, programmed forwarding state, and observed packet outcome.
• Record source identity, schema/model version, event time, receive time, sampling, aggregation, and known loss.
• Detect restarts, counter discontinuities, template changes, sequence gaps, and stale subscriptions.
• Use user-facing measurements for service health and network signals for localization.
• Confirm a suspected fault with evidence from another plane before acting.
• Make telemetry-pipeline health observable; an empty collector is not evidence of a healthy network.