Architectures: Convolutional, Recurrent & Transformer

An architecture decides which parameters are shared and which inputs can interact directly. These are inductive biases: assumptions that make some functions easy to learn with finite data and compute. A fully connected network can represent many of the same mappings, but it does not get locality, sequence state, or content-addressed interaction for free.

Convolution shares a local detector #

For a one-dimensional input, a convolutional channel computes

$$y_{t,c}=\phi\left(b_c+\sum_{\tau=0}^{k-1}\sum_d W_{\tau,d,c}x_{t+\tau,d}\right).$$

The same kernel $W$ is applied at every position. Parameter sharing says a local pattern should mean the same thing wherever it occurs; locality says nearby inputs interact before distant ones. A shift in input produces a corresponding shift in feature map (translation equivariance, up to boundaries/stride).

Pooling, stride, or aggregation can make later outputs less sensitive to exact position, but invariance is not automatic. Padding changes edge behavior. For medical images, maps, or signals, absolute position may matter and coordinate/context features may be required.

Stacking stride-1 kernels expands receptive field. Two layers of width 3 let an output depend on five input positions, not six, because receptive fields overlap. Dilation expands the field without equally increasing parameters. Effective influence can still concentrate near the center even when the theoretical field is large.

CNNs suit images, audio, spatial grids, and local sequence motifs when the stationarity assumption is reasonable. They struggle when arbitrary distant relationships dominate unless depth, dilation, or explicit global mechanisms connect them.

Recurrence carries state through time #

A simple recurrent neural network updates

$$h_t=\phi(W_xx_t+W_hh_{t-1}+b),$$

sharing the same transition at every time step. The hidden state summarizes the prefix, enabling variable-length sequences and streaming inference. Backpropagation through time unfolds the recurrence; gradients contain repeated products of transition Jacobians and can vanish or explode exponentially with lag. LSTM and GRU add gated, partly additive state paths that make retention easier to learn, while sequential dependence still restricts parallel training. See Recurrent Sequence Models for BPTT, gate equations, truncation, state handling, teacher forcing, and encoder–decoder behavior.

Attention routes by content #

Given token representations, self-attention forms queries, keys, and values:

$$Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,$$

$$\mathop{\mathrm{Attention}}(Q,K,V)= \mathop{\mathrm{softmax}}\left(\frac{QK^T}{\sqrt{d_k}}+M\right)V.$$

Each query takes a weighted combination of values according to query–key compatibility. The $\sqrt{d_k}$ scaling keeps dot-product variance from driving softmax into saturation. Mask $M$ blocks padding or future positions.

Multiple heads learn different projections and routing patterns; feedforward blocks transform each position; residual connections and normalization make deep stacks trainable. Because attention without position is permutation-equivariant, positional encodings or relative-position mechanisms supply order.

Every token can reach every other in one layer, and sequence positions train in parallel. Dense self-attention forms an $n\times n$ score matrix, costing quadratic memory/compute in sequence length. Sparse, local, recurrent, or state-space variants trade connectivity for efficiency.

A comparison by bias #

These are not exclusive families. Vision transformers use patch locality in preprocessing; conformers mix convolution and attention; encoder–decoder systems may combine recurrent or convolutional front ends with attention.

Choose from the information path #

Start with the causal/availability graph. A forecasting model must not attend to future measurements. A streaming recognizer has latency and memory bounds. An image model may need rotation rather than translation invariance. Architecture follows these constraints, input resolution, dataset size, and hardware—not fashion.

Failure modes shared across architectures #

• Shortcut features can dominate the intended mechanism regardless of architecture.
• Padding, crop, resize, or tokenization can encode labels or remove signal.
• More context can add distractors and training-serving differences.
• Attention weights are internal routing coefficients, not automatically faithful explanations.
• Parameter count alone does not measure capacity, compute, data requirement, or generalization.

A good architecture makes the desired information flow short and repeatable while making forbidden information unavailable by construction.