Optimal Digital Detection: Matched Filters, Correlators & BER

A digital receiver does not need to preserve the whole sampled waveform forever. It needs statistics that retain every distinction among the allowed transmitted signals and discard dimensions containing only noise. In additive white Gaussian noise (AWGN), projection onto the signal space does exactly that; the matched filter and correlator are two implementations of the same projection.

“Optimal” is conditional. Maximum-likelihood (ML) detection minimizes symbol error for equally likely hypotheses under the assumed observation model. Maximum a posteriori (MAP) detection includes prior probabilities, and Bayes detection can include unequal decision costs. A circuit can implement the equations perfectly and still be wrong because its noise is colored, timing is offset, or its channel estimate is stale.

From a waveform to hypotheses #

During one symbol interval $0\le t<T$, suppose the receiver observes

$$r(t)=s_i(t)+n(t), \qquad i\in\{0,\ldots,M-1\},$$

where $n(t)$ is zero-mean AWGN with two-sided power spectral density $N_0/2$. Expand the $M$ candidate signals in an orthonormal basis $\{\phi_k(t)\}_{k=1}^{N}$ spanning their signal space:

$$s_i(t)=\sum_{k=1}^{N}s_{ik}\phi_k(t),\qquad r_k=\int_0^T r(t)\phi_k(t)\,dt.$$

The vector $\mathbf r=(r_1,\ldots,r_N)$ is sufficient: components orthogonal to every possible $s_i$ contain noise but no information about which hypothesis was sent. Under equal priors, ML chooses the nearest signal vector,

$$\hat i=\arg\min_i\|\mathbf r-\mathbf s_i\|^2.$$

Expanding the square and dropping $\|\mathbf r\|^2$, which is common to every candidate, gives the practical score

$$\Lambda_i=\langle r,s_i\rangle-\frac{E_i}{2},\qquad E_i=\langle s_i,s_i\rangle.$$

The energy term matters when candidate signals have unequal energy. “Choose the largest correlation” is correct without qualification only when the energies are equal.

Why the matched filter maximizes sampled SNR #

Let an LTI filter with impulse response $h(t)$ be sampled at $T$. Its signal output is

$$y_s(T)=\int_0^T h(T-\tau)s(\tau)\,d\tau,$$

while the output-noise variance in AWGN is

$$\sigma_n^2=\frac{N_0}{2}\int_{-\infty}^{\infty}|h(t)|^2dt.$$

Cauchy–Schwarz bounds $|y_s(T)|^2$ by the product of signal energy and filter energy. Equality holds when the filter is proportional to the time-reversed conjugate signal:

$$h(t)=K s^*(T-t).$$

At the sample instant, convolution with this filter becomes correlation with $s(t)$. Thus a correlator multiplying by $s^*(t)$ and integrating over $[0,T]$ and a matched filter followed by a sample produce the same statistic. The filter does not “remove all noise”; it concentrates signal energy at one instant while admitting exactly the noise bandwidth required by that signal.

flowchart LR
  R["received r(t)"] --> F0[matched to s0]
  R --> F1[matched to s1]
  R --> FM[matched to sM−1]
  F0 --> S[sample at T]
  F1 --> S
  FM --> S
  S --> C[add −Ei/2 and log prior]
  C --> D[arg max]

Binary geometry and the likelihood ratio #

For two candidates, subtracting their scores reduces the decision to one axis:

$$z=\langle r,s_1-s_0\rangle \underset{H_0}{\overset{H_1}{\gtrless}} \gamma.$$

With equal priors and costs,

$$\gamma=\frac{E_1-E_0}{2}.$$

If the signals have equal energy, the threshold is zero. The distance $d=\|s_1-s_0\|$ determines the Gaussian tail crossing the boundary:

$$P_e=Q\!\left(\frac{d}{\sqrt{2N_0}}\right).$$

This picture explains why antipodal signalling is power-efficient. For BPSK, $s_1=-s_0$ and $d=2\sqrt{E_b}$, so

$$P_b=Q\!\left(\sqrt{\frac{2E_b}{N_0}}\right).$$

Orthogonal equal-energy binary signals have $d=\sqrt{2E_b}$ and therefore a 3 dB smaller squared distance at the same bit energy.

Worked BPSK decision and BER #

Take coherent BPSK with $E_b/N_0=6$ dB. Convert the ratio before using the formula:

$$\frac{E_b}{N_0}=10^{6/10}=3.981,qquad P_b=Q(\sqrt{2\cdot3.981})=Q(2.822)\approx2.39\times10^{-3}.$$

That is about 2390 erroneous bits per million before coding, assuming independent AWGN, correct coherent phase, correct symbol timing, and no implementation loss. If a simulation at this point measures $10^{-2}$, merely increasing the trial count cannot explain the gap; the model, normalization, or receiver is different.

For one received bit, normalize the matched-filter output so the hypotheses have means $\pm1$. A sample $z=+0.18$ is a hard one, but it is weak evidence; $z=+2.1$ is the same hard bit with much greater confidence. A hard slicer discards this distinction.

Priors and costs move the boundary #

The log-likelihood ratio (LLR) is

$$L(r)=\log\frac{p(r\mid H_1)}{p(r\mid H_0)}.$$

MAP detection decides $H_1$ when

$$L(r)>\log\frac{P(H_0)}{P(H_1)}.$$

If false positives and false negatives have unequal costs, the right side also contains the appropriate cost ratio. A rare alarm event should therefore not use the same threshold as an equiprobable data bit unless the costs happen to compensate for the prior.

For BPSK with observation $y=x+n$, $x\in\{-A,+A\}$, and $n\sim\mathcal N(0,\sigma^2)$, the channel LLR is

$$L(y)=\frac{2Ay}{\sigma^2}.$$

This value is the right interface to a soft-input channel decoder. Its sign is the hard decision; its magnitude is reliability. Getting $A$, $\sigma^2$, or the sign convention wrong does not merely rescale an internal number: it miscalibrates the decoder's evidence.

Colored noise changes the metric #

If sampled noise has covariance $\mathbf C_n$ rather than $\sigma^2\mathbf I$, Euclidean distance is no longer ML. The correct quadratic metric is

$$d_i^2=(\mathbf r-\mathbf s_i)^T\mathbf C_n^{-1}(\mathbf r-\mathbf s_i).$$

Equivalently, whiten the observation and candidate signals, then apply ordinary matched filtering in the whitened space. Narrowband interference, analogue filtering, oversampled front ends, and equalizers commonly color the noise. Treating their output samples as independent can make a detector confidently wrong.

Unknown amplitude, phase, frequency, or timing introduces nuisance parameters. A coherent receiver estimates them and plugs estimates into the likelihood. A noncoherent receiver marginalizes or maximizes over them and pays a performance or complexity price. The comparison is only fair when both receivers use the same available information.

BER measurements need statistical evidence #

For $N$ tested bits and $K$ errors, the estimate is $\hat p=K/N$. The count, not only the ratio, determines confidence. One error in $10^6$ bits and one thousand errors in $10^9$ bits have the same estimate but very different sensitivity to a single burst.

When no errors occur, the result is not “BER = 0”. For independent trials, the 95% upper bound is approximately $3/N$ (the rule of three). Demonstrating BER below $10^{-6}$ with zero observed errors therefore requires roughly three million independent bits under the stated condition—and burst correlation can invalidate the independence assumption.

Failure signatures #

A trustworthy detector test therefore has three layers: verify the sufficient statistics against a direct inner product, verify decisions against the stated ML/MAP metric, and only then compare BER with theory. “Optimal” is not a label attached to a matched-filter block; it is a reproducible claim about the complete observation model and decision rule.