Industrial Data Historians & Time-Series Integrity

An industrial data historian is an operational time-series archive optimized for process values, discrete states, events, quality, and retrieval over long periods. Its defining job is not “put tag values in a database.” It is to preserve enough evidence that a future reader can tell what the source reported, when the process value applied, whether it was trustworthy, and which collection rules shaped the record.

The evidence pipeline #

A point normally passes through several clocks and filters:

1. the device or controller acquires or calculates a value;
2. an interface receives it by scan, subscription, event, or message;
3. a local buffer retains it through temporary archive/network loss;
4. exception logic decides whether to send it onward;
5. archive compression decides whether it adds meaningful shape;
6. storage partitions and indexes the event;
7. query logic returns raw, interpolated, or aggregate results.

The value displayed by a dashboard is therefore a product of a pipeline. Diagnose the stage at which evidence was lost instead of assuming the archive contains every controller update.

A historical event needs context #

A useful stored event contains or resolves to:

• point identity and configuration version;
• value and application data type;
• source timestamp and, where available, server/interface timestamp;
• quality or status;
• engineering unit and interpretation metadata;
• provenance for substituted, manually entered, recalculated, or corrected data.

Tag names are human conveniences, not durable identities by themselves. Renaming a point, reusing a name for another instrument, changing scale, or moving the source address can make a continuous-looking trend semantically discontinuous. Preserve stable identity and effective-dated configuration.

Source time, receive time, and archive time #

Source time estimates when the value was acquired or became valid at the source. Receive time marks arrival at the interface/server. Archive time marks persistence. Network queues and store-and-forward can separate them by seconds or hours.

Sorting solely by arrival time distorts a reconnect: old buffered samples appear as a sudden present-time burst. Sorting solely by source time without validating clocks lets a misconfigured device write into the future or far past. Store the available timestamps, synchronize clocks, define acceptance windows, and flag late or out-of-order events.

For duplicated timestamps, the system needs a documented rule: retain multiple values, replace a prior event with an audit trail, or choose by source sequence/quality. Silent last-write-wins makes incident reconstruction unreliable.

Scan, exception, and compression are separate #

An exception threshold does not mean the process was constant between events; it means no observed change crossed the configured reporting rule. Archive compression commonly evaluates whether an intermediate point lies close enough to the line or corridor between retained points. It preserves shape within an error model, not every sample.

Discrete and enumerated states should normally be stored on every transition and interpreted as stepped, not linearly interpolated. Counters need rollover/reset semantics. Totals and laboratory values may be sparse by nature. One compression template cannot preserve every signal class.

Gaps are data #

A missing interval can mean source failure, interface disconnect, archive outage, bad quality rejection, a disabled point, clock rejection, or a query that excluded bad events. These causes are operationally different. Preserve interface status, buffer state, point configuration changes, and quality transitions so the gap can be explained.

Do not draw an unqualified line across a long absence. A visualization may connect points for orientation only when it marks the gap and the interpolation rule. A calculation should state how incomplete intervals affect its result rather than silently filling them with zero or last value.

Local buffering protects against transient archive/network loss only within its capacity and durability. Engineer queue size, disk persistence, ordering, duplicate handling, backpressure, and what happens when the buffer fills. “Store and forward enabled” is not a losslessness proof.

Raw, interpolated, and aggregate results #

A raw query returns stored events, which may already have passed exception and compression. An interpolated query asks for estimates at selected times according to point semantics. An aggregate query summarizes intervals—average, minimum, maximum, count, duration-in-state, total, and so on.

For a continuously varying analog, a time-weighted average over $[t_0,t_1]$ has the conceptual form

$$\bar{x}=\frac{1}{t_1-t_0}\int_{t_0}^{t_1}x(t)\,dt.$$

The result depends on whether $x(t)$ is treated as stepped or linearly interpolated, how interval boundaries are estimated, and how bad or uncertain spans are weighted or excluded. An arithmetic mean of stored events is generally not a time average because compression deliberately makes event spacing irregular.

An aggregate result should carry interval boundaries, calculation type, coverage, and quality. OPC UA aggregate semantics distinguish good, uncertain, and bad contributions and define status information for calculated values; a client should not discard that evidence.

Corrections and auditability #

Historical values sometimes require annotation, replacement, or deletion: a laboratory correction, a bad instrument interval, a clock repair, or a legally required removal. The archive should record who changed what, when, why, and whether the returned value is original or modified. Recalculation should identify its algorithm, inputs, and effective period.

Separate immutable acquisition evidence from curated analytical series where the use case requires both. A corrected business report should not erase the raw sequence needed to diagnose the instrument or collection path.

Capacity and retrieval #

A first-pass uncompressed event estimate is

$$N_{day}=\sum_i \frac{86{,}400}{T_i},$$

where $T_i$ is the observation interval in seconds for point $i$. Storage then depends on event encoding, indexes, replication, quality/timestamp overhead, compression behavior, and retention tiers. Do not size from a promised compression ratio alone; replay representative signals through the actual rules and include bursts, state transitions, bad quality, reconnect backfill, and index overhead.

Query performance is shaped by time range, point count, event density, aggregation, partitioning, and concurrency. Precomputed summaries can accelerate common dashboards, but their interval, algorithm, quality policy, and rebuild behavior become part of the data contract.

Historian integrity checklist #

• For each point, define identity, type, unit, source, timestamp capability, and quality mapping.
• Validate observation, exception, and compression settings against the signal's important phenomena.
• Preserve source time and quality through buffering, late arrival, duplicates, and reconnect.
• Make gaps, substitutions, corrections, and configuration changes visible and auditable.
• Distinguish raw stored events, interpolation, and aggregates in APIs and displays.
• Test stepped versus sloped semantics, interval boundaries, bad-quality spans, clock jumps, and counter resets.
• Verify recovery by disconnecting the archive path and comparing source evidence with the backfilled result.