Recurrent Sequence Models: RNN, LSTM, GRU & BPTT

A recurrent neural network (RNN) processes an ordered stream by carrying a state forward. Unlike a fixed context window, recurrence can run for any number of steps and can operate online with bounded state. The same reuse that gives this flexibility also couples memory, stability, and optimization: the transition is applied repeatedly in the forward pass, and its Jacobian is multiplied repeatedly in the backward pass.

State is a learned summary, not a lossless history #

A basic recurrent transition is

$$h_t=\phi(W_xx_t+W_hh_{t-1}+b), \qquad y_t=g(h_t).$$

The model shares $W_x$ and $W_h$ across time. Its state $h_t$ can be interpreted as a task-dependent compression of the prefix $x_{1:t}$, but the network has no guarantee of retaining every earlier fact. Training decides what survives because state capacity, activation saturation, noise, and the loss all favor some summaries over others.

The output arrangement follows the task:

Causality is separate from recurrence. A forward RNN is causal; a bidirectional RNN combines past and future context and therefore cannot be used unchanged in a strict streaming system.

BPTT is ordinary backpropagation through shared transitions #

Backpropagation through time (BPTT) unfolds the computation graph and accumulates parameter gradients from every use of the shared transition. The influence of state $h_k$ on a later state $h_t$ contains

$$\frac{\partial h_t}{\partial h_k} =\prod_{i=k+1}^{t}\frac{\partial h_i}{\partial h_{i-1}}.$$

Products of matrices can contract most directions toward zero or expand some directions rapidly. A vanishing long-range contribution means the loss cannot tell an early transition how it should change. An exploding contribution produces a steep update dominated by a few directions and may make the loss non-finite.

The spectral radius of $W_h$ alone is not a universal test: nonlinear derivatives and the visited states also enter every Jacobian. It is nevertheless a useful linearized intuition. Saturated sigmoid or tanh units multiply by small derivatives; a transition with an expanding singular direction can amplify errors.

Global-norm clipping rescales a gradient $g$ when $\lVert g\rVert>\tau$:

$$g\leftarrow \tau\frac{g}{\lVert g\rVert}.$$

This preserves direction while limiting one update's magnitude. It contains exploding gradients; it does not recover a long-range signal that already vanished, and repeated clipping is a diagnostic that the dynamics or learning rate need inspection.

Truncated BPTT detaches the graph after a finite number of steps. State may continue forward beyond that boundary, but credit cannot travel backward through it. “The model saw 10,000 steps” is therefore not equivalent to “the loss trained a 10,000-step dependency.”

LSTM builds a gated additive state path #

One common LSTM formulation computes

$$\begin{aligned} i_t&=\sigma(W_ix_t+U_ih_{t-1}+b_i),\\ f_t&=\sigma(W_fx_t+U_fh_{t-1}+b_f),\\ o_t&=\sigma(W_ox_t+U_oh_{t-1}+b_o),\\ \tilde c_t&=\tanh(W_cx_t+U_ch_{t-1}+b_c),\\ c_t&=f_t\odot c_{t-1}+i_t\odot\tilde c_t,\\ h_t&=o_t\odot\tanh(c_t). \end{aligned}$$

The cell update is additive. The forget gate $f_t$ controls retention, the input gate $i_t$ controls writing, and the output gate $o_t$ controls exposure of the stored state. Along the direct cell path,

$$\frac{\partial c_t}{\partial c_{t-1}}=f_t$$

before accounting for indirect recurrent dependencies. When relevant forget gates remain near one, information and gradient can persist without repeatedly crossing a saturating candidate-state nonlinearity. This makes long time scales learnable; it does not make memory unlimited or eliminate explosions elsewhere in the graph.

GRU merges the storage and exposed state #

A gated recurrent unit (GRU) uses an update gate $z_t$ and reset gate $r_t$:

$$\begin{aligned} z_t&=\sigma(W_zx_t+U_zh_{t-1}),\\ r_t&=\sigma(W_rx_t+U_rh_{t-1}),\\ \tilde h_t&=\tanh(W_hx_t+U_h(r_t\odot h_{t-1})),\\ h_t&=(1-z_t)\odot h_{t-1}+z_t\odot\tilde h_t. \end{aligned}$$

The interpolation supplies an additive carry path without a separate cell/output pair. LSTM versus GRU is not a general quality ranking. Compare validation quality, latency, memory, state behavior, and run-to-run variation under the same data and budget.

Encoder–decoder changes the sequence boundary #

An encoder can consume $x_{1:T}$ and initialize a decoder that factorizes

$$p(y_{1:U}\mid x_{1:T}) =\prod_{u=1}^{U}p(y_u\mid y_{<u},x_{1:T}).$$

The original fixed-vector sequence-to-sequence formulation compressed the source into the encoder's final state. Attention later provided a direct, content-dependent path to all encoder states and relieved that bottleneck. The historical fixed-vector model remains useful for understanding conditioning and autoregressive decoding, but it is not a default current architecture for high-capacity translation.

During training, teacher forcing supplies the true previous target to predict the next one. During generation, the model consumes its own sampled or selected output. That train–inference mismatch is exposure bias: early errors move the decoder into prefixes absent from training. Beam search changes the search approximation, not the learned probability model, and a larger beam can prefer short sequences unless length treatment is explicit.

CTC is a different contract for unsegmented inputs: it sums over monotonic alignments containing blanks and repeated labels. It suits speech, handwriting, and other ordered alignments, but cannot represent arbitrary output reordering.

State handling is part of model correctness #

• Reset state between independent sequences; otherwise one user's or recording's tail contaminates the next.
• When carrying state across batches, preserve sequence identity and detach it at the intended BPTT boundary.
• Mask padded timesteps in both loss and recurrence when padding would otherwise update state.
• Never deploy a bidirectional or unmasked model when future inputs are unavailable.
• Define how state survives gaps, reconnects, missing samples, and changes of sampling rate.
• Evaluate by dependency length and streaming conditions, not only aggregate accuracy.

Useful diagnostics include gradient-norm distributions, clipping frequency, gate saturation, performance versus sequence length, perturbation of early inputs, hidden-state drift across long blank spans, and explicit reset/no-reset tests. The right question is not whether an architecture “has memory,” but which information path carries a particular dependency and whether training provides usable credit along that path.