Training Language Models to Follow Instructions with Human Feedback
Ouyang et al. document a 2022 instruction-tuning pipeline—supervised demonstrations, a preference-trained reward model, and PPO fine-tuning—and show both substantial preference gains and the limits of treating a small labeler group's judgments as universal alignment.
This paper explains how the 2022 InstructGPT models changed a pretrained GPT-3 model's behavior without changing its basic GPT architecture. It is primary evidence for one published OpenAI post-training pipeline, not a specification of current ChatGPT training.
Three-stage pipeline #
- Supervised fine-tuning (SFT): labelers write desired responses to prompts; a pretrained model is fine-tuned to imitate them.
- Reward modeling (RM): labelers rank several model responses to the same prompt; a model learns a scalar score whose differences predict those preferences.
- Policy optimization: PPO updates the SFT policy to increase the learned reward, with a KL penalty limiting departure from the reference policy. The reported PPO-ptx variant also mixes in pretraining updates to reduce capability regressions.
These stages alter which continuations the model tends to produce. They do not turn the model into a database or replace autoregressive token generation with a separate symbolic reasoner.
Evidence and numbers #
On the paper's held-out API-prompt distribution, labelers preferred 175B InstructGPT outputs to 175B GPT-3 outputs of the time. The 1.3B PPO-ptx model was preferred to the 175B GPT-3 baseline despite having over 100 times fewer parameters. On closed-domain tasks, the paper reports hallucination rates of 21% for InstructGPT and 41% for GPT-3. These are results for the paper's models, prompts, labelers, and 2022 protocol—not generic rates for later systems.
The paper's training data included roughly 13,000 SFT prompts, 33,000 reward-model prompts, and 31,000 PPO prompts. Around 40 contractors supplied demonstrations, comparisons, and evaluations; inter-labeler agreement was about 73%.
What is actually being optimized #
The learned reward approximates the preferences expressed by a particular labeling process. The authors explicitly separate that from universal “human values.” Labeler demographics, researcher-written instructions, customer prompt distribution, interface design, and aggregation choices all shape the target.
Reward optimization can also trade away other capabilities. PPO without the pretraining mix regressed on several public NLP datasets; mixing pretraining gradients mitigated but did not eliminate every regression. More reward optimization was not monotonically better.
Limitations that remain durable #
The models still fabricated facts, followed false premises, over-hedged, failed multi-constraint instructions, generated biased or toxic output, and could comply with harmful requests. The paper's labeler group was mostly English-speaking and not representative of all affected users. Better instruction following can also increase misuse capability.
Visual audit #
All 68 physical pages were rendered and inspected. Figure 2 is the load-bearing SFT→RM→PPO flow; Figures 12 and 39–50 reveal labeling context and important failure behavior. None is retained. The pipeline is described directly because copying Figure 2 could imply it is the current ChatGPT recipe, while the UI and benchmark plots are source-specific evidence rather than durable architecture. The GTK article marks this pipeline as historical and keeps current-product claims separate.