Reinforcement Learning from Human Feedback

Nathan Lambert · Manning, 2026

How language models are tuned after pretraining: instruction tuning, reward models, policy-gradient methods, direct alignment, and the preference data behind them. Each chapter pairs the ideas with the practical choices that make them work.

Start readingChapter 1 · IntroductionPDF

20 chapters

Chapters

Introductions

  1. Chapter 1: Introduction
  2. Chapter 2: A Tiny History of RLHF
  3. Chapter 3: Training Overview

Core training pipeline

  1. Chapter 4: Instruction Fine-Tuning
  2. Chapter 5: Reward Modeling
  3. Chapter 6: Reinforcement Learning
  4. Chapter 7: Reasoning and Inference-Time Scaling
  5. Chapter 8: Direct-Alignment Algorithms
  6. Chapter 9: Rejection Sampling

Data and preferences

  1. Chapter 10: The Nature of Preferences
  2. Chapter 11: Preference Data
  3. Chapter 12: Synthetic Data & Distillation

Practical considerations

  1. Chapter 13: Tool Use and Function Calling
  2. Chapter 14: Over-Optimization
  3. Chapter 15: Regularization
  4. Chapter 16: Evaluation
  5. Chapter 17: Crafting Model Character and Products

Appendices

  1. Appendix A: Definitions
  2. Appendix B: Beyond “Just Style”
  3. Appendix C: Practical Issues