Image Compression & Transform Coding

An image codec does not merely “run an FFT and remove high frequencies.” It models redundancy—neighboring pixels, predictable edges, repeated colors—and then represents the remaining error efficiently. A lossy codec also quantizes information, making an irreversible rate–distortion decision. Entropy coding makes symbols shorter; quantization is the usual source of loss.

Format choice is a systems decision. Content type, alpha, bit depth, color management, animation, random access, decode memory, hardware support, progressive delivery, and repeated editing can matter more than one benchmark's bytes or score.

The common lossy pipeline #

flowchart TB
  subgraph MODEL["Signal model"]
    direction LR
    RGB["Pixels + color metadata"] --> C["Color transform"] --> SS["Chroma sampling"] --> P["Prediction"]
  end
  subgraph CODE["Residual coding"]
    direction LR
    T["Transform"] --> Q["Quantization"] --> E["Entropy coding"] --> B["Bitstream + metadata"]
  end
  P --> T

Not every format contains every box. Baseline JPEG uses an 8×8 DCT without inter-pixel prediction inside a component; lossy WebP and AV1-derived AVIF use block prediction followed by transform coding of residuals. JPEG 2000 uses wavelets. PNG is lossless and uses scanline filters before DEFLATE rather than transform quantization.

Decoder output is the reverse model, not the original after quantization:

$$\hat x = \text{prediction} + T^{-1}(Q^{-1}(\hat q)).$$

Bits saved by better prediction reduce residual energy before quantization. Bits saved by coarser quantization increase distortion. Entropy coding cannot repair a poor predictor or recover discarded coefficients.

Color comes before compression quality #

RGB channels are correlated and do not align well with visual sensitivity. Lossy codecs commonly transform to a luma-like component $Y'$ and two chroma-difference components. The prime matters: codec signals are usually nonlinear gamma-coded components, not physical luminance.

Chroma subsampling exploits lower sensitivity to fine chroma detail:

• 4:4:4 keeps chroma at every luma sample;
• 4:2:2 halves chroma horizontally;
• 4:2:0 halves it horizontally and vertically.

Subsampling is independent of transform quantization and can be the dominant damage for colored text, UI screenshots, diagrams, and synthetic edges. Chroma sample siting and reconstruction filters must match; otherwise colors shift by half a pixel or bleed across edges.

Color primaries, transfer function, matrix coefficients, range, ICC profile, and mastering metadata determine how code values become displayed color. Stripping or mislabelling metadata can produce a large visible error even when decoded pixel numbers are exact.

Prediction removes local redundancy #

Images have strong local correlation. A lossless predictor estimates a pixel from already decoded neighbors and encodes the residual. PNG chooses among five simple scanline filters; DEFLATE then compresses the filtered bytes. WebP lossless uses multiple predictors, reversible color transforms, color indexing, and LZ77-style backward references before entropy coding.

Modern lossy block codecs predict a block from reconstructed neighboring pixels using directional modes. Smooth areas yield small residuals; edges favor a direction aligned with the boundary. Because the decoder uses previously reconstructed neighbors, the encoder must run a decoder model during mode search. Predicting from original pixels would drift.

Block size trades modeling flexibility against signaling cost. Large blocks describe smooth regions efficiently; small blocks follow texture and edges. Encoders search partitions and modes under a rate–distortion objective such as

$$J=D+\lambda R,$$

where $D$ is distortion, $R$ is estimated bits, and $\lambda$ expresses the current quality/rate trade. Different encoders can emit very different files for the same format and quality label.

Transforms compact residual energy #

For a block residual vector $r$, an orthogonal transform produces $c=Tr$. Transform coding is useful when energy concentrates in a few coefficients that can be represented precisely while many small coefficients are coarsely quantized or zeroed.

The 8×8 DCT-II behind classic JPEG uses cosine basis functions with increasing horizontal and vertical spatial frequency. The DC coefficient records the block mean; AC coefficients record variation. The DCT is related to the DFT of an even extension, but its boundary condition avoids explicit complex values and often compacts natural-image blocks well.

JPEG 2000's wavelet transform represents an image across scales without independent 8×8 block boundaries. It supports progressive resolution/quality and precise lossless or lossy paths, at the cost of a different complexity and ecosystem profile.

Transforms do not by themselves compress: without quantization and entropy coding, an invertible transform rearranges information. Their job is to make coefficient statistics easier to code.

Quantization is the irreversible step #

A scalar quantizer maps a coefficient to a discrete index, for example

$$q_i=\mathop{\mathrm{round}}\left(\frac{c_i}{\Delta_i}\right), \qquad \hat c_i=q_i\Delta_i.$$

Larger step $\Delta_i$ creates more zeros and repeated small indices but larger reconstruction error. Frequency-dependent matrices spend less precision where the codec's visual model expects error to be less visible. Quality settings are encoder-specific mappings to quantizers and tools, not cross-format physical units.

Quantization artifacts reveal the representation:

Repeated lossy saves compound error and can lock later encoders onto earlier artifacts. Keep a lossless master and encode delivery derivatives once.

Entropy coding removes statistical redundancy #

After prediction and quantization, symbols are nonuniform: zeros, short runs, small magnitudes, and common modes occur frequently. Huffman, arithmetic/range, or context-adaptive coders assign shorter representations to more probable events. This stage is lossless with respect to the symbol stream.

The decoder must reproduce contexts exactly, so bitstream syntax, scan order, probability update, and error behavior are normative. A truncated or corrupt stream may lose one block, one restart interval, or the remainder of the file depending on resynchronization structure. Security testing matters because image decoders parse attacker-controlled dimensions, partitions, and entropy states before allocating large buffers.

What the formats optimize #

AVIF is a file-format mapping around AV1-coded image items, not simply “an AV1 frame with another extension.” Item properties, transforms, alpha association, sequences, and color metadata participate in correct display.

Evaluate a codec as a deployment #

Rate–distortion curves compare multiple bitrates, not one hand-picked quality. PSNR measures squared sample error and is useful for regression but correlates imperfectly with perception. SSIM-like and learned metrics add perceptual models and their own failure modes. For product decisions, combine metrics with controlled visual inspection at actual display sizes and with task-specific tests such as text readability or machine-vision accuracy.

Measure:

• encoded bytes, dimensions, bit depth, chroma format, alpha, and metadata;
• encode/decode latency, peak memory, and energy on target devices;
• quality across photographs, gradients, line art, text, transparency, and adversarial dimensions;
• browser/library support for every required feature, not only the file extension;
• progressive behavior and failure under truncation/corruption;
• color output against known reference values and displays.

A selection workflow #

1. Preserve the source in a lossless, color-managed representation with enough bit depth.
2. Classify content: photograph, UI/text, illustration, scientific array, HDR, alpha, or animation.
3. Enumerate required decoders and feature constraints before choosing a format.
4. Generate rate–distortion sweeps with production encoder settings.
5. Inspect failure regions—edges, gradients, saturated colors, transparency, and metadata—not only average scenes.
6. Validate actual decode time, memory, and pixel/color output on every target class.
7. Store the codec configuration beside the derivative so it can be reproduced without treating a quality number as a specification.

Compression is successful when the decoded representation remains fit for its downstream purpose at the chosen system cost. Smallest file at one score is evidence about one operating point, not a universal codec ranking.