Sequence to Sequence Learning with Neural Networks
The 2014 sequence-to-sequence paper demonstrated an end-to-end LSTM encoder–decoder for variable-length translation, exposing the fixed-vector conditioning contract, autoregressive factorization, beam search, and the optimization effect of shortening dependency paths by reversing source order.
The method uses one LSTM to encode a variable-length input into its final state and another LSTM to factor the output probability left to right until an end token. It established that a direct neural system could perform a large sequence-to-sequence task without a hand-designed alignment pipeline.
Contributions and limits #
- The encoder and decoder have separate parameters and support input/output sequences of different lengths.
- The decoder defines a normalized conditional distribution over the next token and uses approximate beam search at inference.
- Reversing only the source sequence shortened several early source–target dependency paths and materially improved optimization in the reported setup.
- Gradient norm clipping remained necessary despite LSTM gating.
- The single fixed-dimensional encoder vector is a bottleneck. Attention-based models supplied direct access to encoder states soon afterward; the paper is historical architecture evidence, not a current translation baseline.
Visual audit #
All nine pages were rendered and inspected. The encoder–decoder schematic, PCA projections, length/frequency plots, examples, and result tables were reviewed. None is retained: the recurrent GTK chapter explains the probability and information paths directly, and copying historical benchmark plots would not improve the mechanism explanation.