Language Models are Unsupervised Multitask Learners

GPT-2 is a clean historical case study of the decoder-only language-model pattern. The report frames every document, task description, example, input, and output as one sequence of symbols and optimizes the autoregressive likelihood

$$p(x_1,\ldots,x_T)=\prod_{t=1}^{T}p(x_t\mid x_{<t}).$$

This converts many natural-language tasks into conditional continuation without adding a task-specific encoder, decoder, or output head.

Architecture and representation #

The models use a Transformer language-model architecture with byte-level BPE, learned token and position embeddings, pre-normalization, and a 1,024-token context. The four reported sizes range from 117 million to 1.542 billion parameters. The released reference implementation makes the causal mechanism concrete: a lower-triangular attention mask removes future keys, repeated attention/MLP blocks transform the prefix, and the final hidden states are projected to vocabulary logits using the token-embedding matrix.

The report is evidence for the decoder-only pattern, not for the exact internals of current proprietary ChatGPT models.

What the experiments show #

Without task-specific gradient updates, larger GPT-2 variants improved across language-modeling, cloze, reading-comprehension, summarization, translation, and question-answering evaluations. Natural-language cues such as TL;DR: or a sequence of question/answer examples could induce task-like behavior. This supports the idea that broad next-token training can acquire reusable conditional behaviors.

The paper also contains its own warnings:

• summarization could look plausible while confusing details;
• question answering remained far below retrieval-based systems;
• WebText overlapped evaluation datasets, producing a small but consistent benefit;
• samples reproduced repeated training strings in some conditions;
• fluent generations included confident factual inventions.

The appendix samples are especially useful counterevidence to equating fluency with truth: the larger model maintains genre and local coherence while fabricating people, dates, events, and technical instructions.

Current interpretation #

GPT-2 establishes the architectural and objective lineage. It does not establish current scale, context length, tokenizer, attention variant, training corpus, serving stack, or post-training method. Those details must come from model-specific current documentation where disclosed.

Visual audit #

All 24 physical pages were rendered and inspected. The model-scaling plots, overlap CDF, benchmark tables, and sample-comparison pages support the historical experiments but are not required to understand the causal decoder mechanism. None is retained. The GTK article replaces them with a source-independent generation loop and uses prose to preserve the crucial lesson from the samples: probability-ranked continuation is not a truth guarantee.