Speech Coding & Source-Filter Models

Speech is structured: vocal-fold excitation and turbulent noise drive a time-varying vocal tract whose resonances shape the spectrum. A speech codec can represent that source, the tract filter, or a transform of the resulting waveform more efficiently than sending every sample independently. The best family depends on whether the task is conversational voice, mixed speech/music, archival fidelity, or machine analysis.

Coding is only one contributor to end-to-end voice quality. Capture, echo control, denoising, frame buffering, network jitter, packet loss concealment, clock drift, playback, and acoustic feedback can dominate a nominal codec difference.

The source-filter approximation #

Over a short frame, linear prediction models a speech sample as

$$\hat s[n]=\sum_{k=1}^{p}a_k s[n-k], \qquad e[n]=s[n]-\hat s[n].$$

The synthesis filter is

$$H(z)=\frac{1}{1-\sum_{k=1}^{p}a_kz^{-k}}.$$

Its spectral envelope approximates vocal-tract resonances; the residual $e[n]$ represents excitation and modeling error. Voiced speech has quasi-periodic excitation, while fricatives and plosives have noise-like or transient excitation. The approximation is local: phonemes and pitch change, so codecs update parameters frame by frame.

flowchart TB
  subgraph ENC["Encode"]
    direction LR
    S["Mic samples"] --> A["Pitch / LPC / transform"] --> Q["Quantize"] --> P["Packet"]
  end
  subgraph DEC["Receive and decode"]
    direction LR
    J["Jitter + loss handling"] --> D["Synthesis"] --> O["Playback"]
  end
  P --> J

Linear predictive coefficients are not usually quantized directly because small coefficient errors can destabilize the synthesis filter. Codecs use representations such as line spectral frequencies/pairs with better interpolation and stability behavior.

Three coding families #

Waveform coding preserves the waveform without an explicit speech-production model. ITU-T G.711 compands 8 kHz speech samples into 8-bit A-law or μ-law PCM at 64 kbit/s. It has low algorithmic delay and robust simple decoding, but uses more bitrate than model-based speech codecs and carries only narrowband audio.

Parametric/vocoder coding sends a compact model—spectral envelope, pitch, voicing, gain—then synthesizes speech. It reaches very low bitrates but model mismatch can sound synthetic and handles non-speech poorly.

Analysis-by-synthesis CELP searches a codebook for excitation that, after the predictive synthesis and perceptual weighting filters, best matches the input. Algebraic CELP makes the codebook search practical. It preserves speech naturalness at low bitrates but introduces frame/look-ahead delay and degrades on music or unusual signals if the model dominates.

Transform coding groups samples into frequency-like coefficients, quantizes them according to signal/perceptual importance, and entropy-codes the result. It suits music and mixed content and can use low-overlap transforms to control delay, but transient handling and packet loss need explicit design.

Modern codecs are hybrids rather than ideological choices. Opus contains a speech-oriented SILK mode, a low-delay MDCT-based CELT mode, and a hybrid mode; the encoder can switch by content, bitrate, bandwidth, and configuration. 3GPP EVS similarly spans speech and audio operating points for mobile services.

Framing converts compression into latency #

End-to-end one-way delay includes

$$D=D_{capture}+D_{frame}+D_{lookahead}+D_{encode}+D_{packet} +D_{network}+D_{jitter}+D_{decode}+D_{playout}.$$

Packet duration is not the same as algorithmic delay. A codec may require look-ahead; a sender may aggregate multiple frames; the receiver adds a jitter buffer. RFC 6716 defines Opus configurations from 2.5 to 60 ms frame duration and reports total algorithmic delay depending on mode; 20 ms is common, not mandatory.

Longer frames usually improve coding efficiency and reduce packet overhead, but each lost packet removes more audio and framing latency rises. Short frames improve interactivity and loss granularity while spending more bitrate and processing.

Bitrate is not the only rate #

Network rate includes headers and protection. Sending a 20-byte payload every 20 ms is 8 kbit/s of codec payload, but IPv4/UDP/RTP adds at least 40 bytes before link framing: header rate alone is 16 kbit/s without compression or bundling. Secure transport and FEC add more.

Compare codecs with all of these fixed:

Packet loss and recovery #

Packet loss concealment predicts missing audio from decoder state; it cannot recover arbitrary information. In-band FEC sends a coarse representation of a previous frame in a later packet. It helps only if playout waits long enough for that later packet and if the later packet arrives. Redundant transport or duplicate packets make different delay/bandwidth trades.

Losses in bursts are harder than isolated losses because predictor and pitch history disappear together. A jitter buffer trades late-packet loss against latency. Therefore evaluate a codec over timestamped packet traces, not by deleting independent frames at one average percentage.

Tandem coding and processing #

Decoding and re-encoding a lossy speech stream accumulates distortion because the next encoder treats synthesis artifacts as input. Sample-rate conversion, noise suppression, automatic gain control, and echo cancellation between transcodes can change the signal further. Preserve one compressed representation end to end when possible, or decode once to a lossless working format for editing.

Voice activity detection and discontinuous transmission save rate by suppressing silence and sending comfort-noise parameters. Poor thresholds clip quiet phonemes or produce an unnatural noise-floor jump. Echo cancellers need consistent timing and a linear-enough playback path; clipping or rapid nonlinear processing makes the echo path hard to model.

Selecting and verifying a voice path #

1. Define conversational delay, bandwidth, loss pattern, devices, and whether music/non-speech must survive.
2. Choose a standardized codec supported end to end; avoid transcoding at gateways.
3. Configure frame size, bitrate, VBR constraints, FEC, DTX, and jitter policy as one system.
4. Verify timestamp and sample-clock handling over long calls; clock mismatch otherwise drains or fills buffers.
5. Test clean and noisy talkers, multiple languages, plosives, double-talk, music-on-hold, and silence transitions.
6. Replay real packet-delay/loss traces and measure both objective metrics and controlled listening outcomes.
7. Preserve original captures when codec output will feed recognition, forensics, or future reprocessing.

Speech coding succeeds when intelligibility, naturalness, delay, robustness, and interoperability all meet the use case. A lower nominal bitrate is not an improvement if it moves cost into jitter buffering, retransmission, user fatigue, or gateway transcoding.