{"id":1045,"date":"2026-07-06T11:15:14","date_gmt":"2026-07-06T11:15:14","guid":{"rendered":"https:\/\/learnerbox.net\/blog\/?p=1045"},"modified":"2026-07-07T10:04:02","modified_gmt":"2026-07-07T10:04:02","slug":"how-large-language-models-work-part-4-training-scaling-laws-and-rlhf","status":"publish","type":"post","link":"https:\/\/learnerbox.net\/blog\/ai-theory\/how-large-language-models-work-part-4-training-scaling-laws-and-rlhf\/","title":{"rendered":"How Large Language Models Work \u2014 Part 4: Training, Scaling Laws, and RLHF"},"content":{"rendered":"\n
This is Part 4 of a five-part series on the internal mechanics of large language models. Part 3<\/a> covered the complete transformer block and architectural variants. Part 4 addresses how models are trained: the pre-training objective, loss functions, the Chinchilla scaling laws, and the alignment pipeline of instruction tuning and RLHF that transforms a raw language model into a useful assistant.<\/mark><\/em><\/h6>\n\n\n\n

The Pre-Training Objective<\/h4>\n\n\n\n

The entire capability of a modern LLM emerges from one deceptively simple training objective: predict the next token. Given a sequence of tokens [<\/mo>t<\/mi>1<\/mn><\/msub>,<\/mo>t<\/mi>2<\/mn><\/msub>,<\/mo>\u2026<\/mo>,<\/mo>t<\/mi>n<\/mi><\/msub>]<\/mo><\/mrow>[t_1, t_2, \\ldots, t_n]<\/annotation><\/semantics><\/math>, the model is trained to maximise the log-probability of each token given all preceding tokens:L<\/mi>PT<\/mtext><\/msub>=<\/mo>\u2212<\/mo>1<\/mn>n<\/mi><\/mfrac>\u2211<\/mo>i<\/mi>=<\/mo>1<\/mn><\/mrow>n<\/mi><\/munderover>log<\/mi>\u2061<\/mo>P<\/mi>(<\/mo>t<\/mi>i<\/mi><\/msub>\u2223<\/mo>t<\/mi>1<\/mn><\/msub>,<\/mo>\u2026<\/mo>,<\/mo>t<\/mi>i<\/mi>\u2212<\/mo>1<\/mn><\/mrow><\/msub>;<\/mo>\u2009<\/mtext>\u03b8<\/mi>)<\/mo><\/mrow>\\mathcal{L}_{\\text{PT}} = -\\frac{1}{n} \\sum_{i=1}^{n} \\log P(t_i \\mid t_1, \\ldots, t_{i-1};\\, \\theta)<\/annotation><\/semantics><\/math><\/p>\n\n\n\n

This is the negative log-likelihood loss, or equivalently, cross-entropy between the model’s predicted distribution and the one-hot true distribution over the vocabulary. Minimising L<\/mi>PT<\/mtext><\/msub><\/mrow>\\mathcal{L}_{\\text{PT}}<\/annotation><\/semantics><\/math> is equivalent to maximising the likelihood of the training corpus under the model.<\/p>\n\n\n\n

The reason this objective is so productive is that predicting the next token accurately requires solving an enormous range of implicit sub-problems. To predict the next word in a chemistry paper, the model must understand chemistry. To predict dialogue in a novel, it must model character motivation and narrative consistency. To predict the output of a code snippet, it must simulate execution. All of these capabilities emerge as instrumental sub-goals of the single next-token prediction objective, a phenomenon sometimes called the “bitter lesson” of AI: simple objectives applied at scale consistently outperform hand-crafted inductive biases.<\/p>\n\n\n\n

The training corpus for frontier models is correspondingly vast. GPT-3 was trained on roughly 300 billion tokens drawn from Common Crawl, WebText2, Books, and Wikipedia. Llama 3 used over 15 trillion tokens. Assembling, filtering, deduplicating, and quality-scoring a corpus at this scale is itself a major engineering undertaking, and corpus quality is widely understood to be among the most important determinants of downstream model capability, arguably more important than architectural choices at equivalent parameter counts.<\/p>\n\n\n\n

The Training Loop<\/h4>\n\n\n\n

Pre-training proceeds through the standard deep learning training loop, applied at extreme scale:<\/p>\n\n\n\n

Forward pass:<\/strong> A batch of token sequences is sampled from the corpus. Each sequence is processed through the full transformer stack consisting of embedding, N<\/mi><\/mrow>N<\/annotation><\/semantics><\/math>N transformer blocks, and unembedding. This produces a probability distribution over the vocabulary at each position.<\/p>\n\n\n\n

Loss computation:<\/strong> Cross-entropy loss is computed by comparing the model’s predicted distribution at each position with the true next token.<\/p>\n\n\n\n

Backward pass:<\/strong> Gradients of the loss with respect to all \u03b8<\/mi><\/mrow>\\theta<\/annotation><\/semantics><\/math>\u03b8 parameters are computed via backpropagation through the entire network.<\/p>\n\n\n\n

Parameter update:<\/strong> An optimiser, called Adam or AdamW in virtually all modern LLMs, applies the gradient update:\u03b8<\/mi>\u2190<\/mo>\u03b8<\/mi>\u2212<\/mo>\u03b7<\/mi>\u22c5<\/mo>m<\/mi>^<\/mo><\/mover>t<\/mi><\/msub>\/<\/mi>(<\/mo>v<\/mi>^<\/mo><\/mover>t<\/mi><\/msub><\/msqrt>+<\/mo>\u03f5<\/mi>)<\/mo><\/mrow>\\theta \\leftarrow \\theta – \\eta \\cdot \\hat{m}_t \/ (\\sqrt{\\hat{v}_t} + \\epsilon)<\/annotation><\/semantics><\/math><\/p>\n\n\n\n

where m<\/mi>^<\/mo><\/mover>t<\/mi><\/msub><\/mrow>\\hat{m}_t<\/annotation><\/semantics><\/math>\u200b and v<\/mi>^<\/mo><\/mover>t<\/mi><\/msub><\/mrow>\\hat{v}_t<\/annotation><\/semantics><\/math>\u200b are bias-corrected first and second moment estimates of the gradient, and \u03b7<\/mi><\/mrow>\\eta<\/annotation><\/semantics><\/math> is the learning rate. AdamW adds a decoupled weight decay term \u03bb<\/mi>\u03b8<\/mi><\/mrow>\\lambda \\theta<\/annotation><\/semantics><\/math> directly to the parameter update, separate from the gradient. This is important because standard L2 regularisation interacts badly with Adam’s adaptive scaling, while decoupled weight decay does not.<\/p>\n\n\n\n

The learning rate schedule typically involves a linear warmup phase (to avoid instability at the start of training when gradients are large and parameters are random) followed by a cosine decay to a minimum learning rate of roughly \u03b7<\/mi>min<\/mtext><\/msub>\u2248<\/mo>0.1<\/mn>\u03b7<\/mi>max<\/mtext><\/msub><\/mrow>\\eta_{\\text{min}} \\approx 0.1 \\eta_{\\text{max}}<\/annotation><\/semantics><\/math>\u200b:\u03b7<\/mi>t<\/mi><\/msub>=<\/mo>\u03b7<\/mi>min<\/mtext><\/msub>+<\/mo>1<\/mn>2<\/mn><\/mfrac>(<\/mo>\u03b7<\/mi>max<\/mtext><\/msub>\u2212<\/mo>\u03b7<\/mi>min<\/mtext><\/msub>)<\/mo>(<\/mo>1<\/mn>+<\/mo>cos<\/mi>\u2061<\/mo>\u2009\u2063<\/mtext>(<\/mo>\u03c0<\/mi>t<\/mi><\/mrow>T<\/mi><\/mfrac>)<\/mo><\/mrow>)<\/mo><\/mrow><\/mrow>\\eta_t = \\eta_{\\text{min}} + \\frac{1}{2}(\\eta_{\\text{max}} – \\eta_{\\text{min}})\\left(1 + \\cos\\!\\left(\\frac{\\pi t}{T}\\right)\\right)<\/annotation><\/semantics><\/math><\/p>\n\n\n\n

where T<\/mi><\/mrow>T<\/annotation><\/semantics><\/math> is the total number of training steps. The cosine schedule has become the near-universal choice because it empirically outperforms linear decay and step schedules across model sizes and tasks.<\/p>\n\n\n\n

At frontier scale, training runs on clusters of thousands of GPUs or TPUs, employing a combination of data parallelism (each device processes a different batch, gradients are aggregated), tensor parallelism (each layer’s weight matrices are sharded across devices), and pipeline parallelism (different layers run on different devices simultaneously). GPT-3’s training required approximately 3.14 \u00d7 10\u00b2\u00b3 FLOPs and ran on 10,000 V100 GPUs. A GPT-4-class training run is estimated at well over 10\u00b2\u2075 FLOPs.<\/p>\n\n\n\n

Scaling Laws and the Chinchilla Result<\/h4>\n\n\n\n

How should a fixed compute budget be allocated between model size (number of parameters N<\/mi><\/mrow>N<\/annotation><\/semantics><\/math>) and training data (number of tokens D<\/mi><\/mrow>D<\/annotation><\/semantics><\/math>)? This is the central question addressed by neural scaling law research.<\/p>\n\n\n\n

Kaplan et al. (OpenAI, 2020)<\/a> established that loss scales as a power law in both N<\/mi><\/mrow>N<\/annotation><\/semantics><\/math> and D<\/mi><\/mrow>D<\/annotation><\/semantics><\/math>, and derived the now-famous result that, for a fixed compute budget C<\/mi><\/mrow>C<\/annotation><\/semantics><\/math>, loss is minimised by scaling N<\/mi><\/mrow>N<\/annotation><\/semantics><\/math> and D<\/mi><\/mrow>D<\/annotation><\/semantics><\/math> in a roughly fixed ratio, with N<\/mi><\/mrow>N<\/annotation><\/semantics><\/math> growing considerably faster than D<\/mi><\/mrow>D<\/annotation><\/semantics><\/math>. This led to the prevailing practice of training very large models on relatively modest amounts of data, exemplified by GPT-3 (175B parameters, 300B tokens).<\/p>\n\n\n\n

Hoffmann et al. (DeepMind, 2022) in the “Chinchilla” paper<\/a> challenged this conclusion. Using a broader range of model sizes and more careful experimental design, they derived a different optimal allocation:N<\/mi>opt<\/mtext><\/msub>\u221d<\/mo>C<\/mi>0.5<\/mn><\/msup>,<\/mo><\/mspace>D<\/mi>opt<\/mtext><\/msub>\u221d<\/mo>C<\/mi>0.5<\/mn><\/msup><\/mrow>N_{\\text{opt}} \\propto C^{0.5}, \\quad D_{\\text{opt}} \\propto C^{0.5}<\/annotation><\/semantics><\/math><\/p>\n\n\n\n

That is, compute-optimal training requires scaling parameters and tokens in a roughly 1:1 ratio. Their finding: for a given compute budget, the optimal model size is significantly smaller than previously believed, but must be trained on significantly more data. The compute-optimal ratio is approximately 20 training tokens per parameter.<\/p>\n\n\n\n

The practical implication was dramatic. GPT-3, by the Chinchilla analysis, was massively undertrained. A 175B parameter model should be trained on approximately 3.5 trillion tokens to be compute-optimal, not 300 billion. Chinchilla itself (70B parameters, 1.4 trillion tokens) outperformed Gopher (280B parameters) on nearly every benchmark despite using a quarter of the parameters, simply by training longer on more data.<\/p>\n\n\n\n

The Chinchilla result reshaped subsequent model development: Llama 1 and 2 trained smaller models on far more tokens; Llama 3 trained an 8B model on 15 trillion tokens. The scaling law itself takes the form:L<\/mi>(<\/mo>N<\/mi>,<\/mo>D<\/mi>)<\/mo>=<\/mo>E<\/mi>+<\/mo>A<\/mi>N<\/mi>\u03b1<\/mi><\/msup><\/mfrac>+<\/mo>B<\/mi>D<\/mi>\u03b2<\/mi><\/msup><\/mfrac><\/mrow>L(N, D) = E + \\frac{A}{N^\\alpha} + \\frac{B}{D^\\beta}<\/annotation><\/semantics><\/math><\/p>\n\n\n\n

where E<\/mi><\/mrow>E<\/annotation><\/semantics><\/math> is the irreducible entropy of the data distribution (a lower bound on achievable loss), and A<\/mi><\/mrow>A<\/annotation><\/semantics><\/math>, B<\/mi><\/mrow>B<\/annotation><\/semantics><\/math>, \u03b1<\/mi><\/mrow>\\alpha<\/annotation><\/semantics><\/math>, \u03b2<\/mi><\/mrow>\\beta<\/annotation><\/semantics><\/math> are empirically fitted constants. The three terms represent, respectively: the noise floor of the task, the contribution of model capacity, and the contribution of data quantity.<\/p>\n\n\n\n

From Language Model to Assistant: Instruction Tuning and RLHF<\/h4>\n\n\n\n

A model trained only on next-token prediction is not a useful assistant. It will complete any prompt in the style of its training distribution and it might respond to “How do I bake bread?” by generating more questions rather than an answer, because question-continuation is a common pattern in web text. Transforming a base language model into a helpful, honest, and harmless assistant requires a second training phase: alignment.<\/p>\n\n\n\n

The alignment pipeline used by OpenAI, Anthropic, Google DeepMind, and most frontier labs follows a three-stage process.<\/p>\n\n\n\n

Stage 1 \u2014 Supervised fine-tuning (SFT):<\/strong> Human annotators write demonstrations of ideal assistant behaviour through prompt-response pairs that exhibit the desired properties: helpfulness, accuracy, appropriate refusal of harmful requests, appropriate tone and format. The base model is then fine-tuned on this dataset using standard cross-entropy loss. SFT alone substantially improves instruction-following; InstructGPT’s SFT model already significantly outperformed the raw GPT-3 base in human evaluations.<\/p>\n\n\n\n

Stage 2 \u2014 Reward model training:<\/strong> Human annotators rank multiple model responses to the same prompt, from best to worst. A separate reward model R<\/mi>\u03d5<\/mi><\/msub><\/mrow>R_\\phi<\/annotation><\/semantics><\/math>\u200b, itself a transformer, is trained to predict these rankings. Given a prompt x<\/mi><\/mrow>x<\/annotation><\/semantics><\/math> and response y<\/mi><\/mrow>y<\/annotation><\/semantics><\/math>, the reward model produces a scalar score R<\/mi>\u03d5<\/mi><\/msub>(<\/mo>x<\/mi>,<\/mo>y<\/mi>)<\/mo>\u2208<\/mo>R<\/mi><\/mrow>R_\\phi(x, y) \\in \\mathbb{R}<\/annotation><\/semantics><\/math>. The training objective minimises a pairwise ranking loss:L<\/mi>RM<\/mtext><\/msub>=<\/mo>\u2212<\/mo>E<\/mi>(<\/mo>x<\/mi>,<\/mo>y<\/mi>w<\/mi><\/msub>,<\/mo>y<\/mi>l<\/mi><\/msub>)<\/mo><\/mrow><\/msub>\u2009\u2063<\/mtext>[<\/mo>log<\/mi>\u2061<\/mo>\u03c3<\/mi>\u2009\u2063<\/mtext>(<\/mo>R<\/mi>\u03d5<\/mi><\/msub>(<\/mo>x<\/mi>,<\/mo>y<\/mi>w<\/mi><\/msub>)<\/mo>\u2212<\/mo>R<\/mi>\u03d5<\/mi><\/msub>(<\/mo>x<\/mi>,<\/mo>y<\/mi>l<\/mi><\/msub>)<\/mo>)<\/mo><\/mrow>]<\/mo><\/mrow><\/mrow>\\mathcal{L}_{\\text{RM}} = -\\mathbb{E}_{(x, y_w, y_l)}\\!\\left[\\log \\sigma\\!\\left(R_\\phi(x, y_w) – R_\\phi(x, y_l)\\right)\\right]<\/annotation><\/semantics><\/math><\/p>\n\n\n\n

where y<\/mi>w<\/mi><\/msub><\/mrow>y_w<\/annotation><\/semantics><\/math> is the preferred response, y<\/mi>l<\/mi><\/msub><\/mrow>y_l<\/annotation><\/semantics><\/math>\u200b is the less preferred response, and \u03c3<\/mi><\/mrow>\\sigma<\/annotation><\/semantics><\/math> is the sigmoid function. This Bradley-Terry loss encourages the reward model to assign higher scores to preferred completions.<\/p>\n\n\n\n

Stage 3 \u2014 Reinforcement learning from human feedback (RLHF):<\/strong> The SFT model \u03c0<\/mi>\u03b8<\/mi><\/msub><\/mrow>\\pi_\\theta<\/annotation><\/semantics><\/math>\u200b is further fine-tuned using the reward model as a proxy for human preference. The policy gradient objective maximises expected reward while penalising deviation from the SFT model via a KL divergence constraint:J<\/mi>(<\/mo>\u03b8<\/mi>)<\/mo>=<\/mo>E<\/mi>x<\/mi>\u223c<\/mo>D<\/mi>,<\/mo>\u2009<\/mtext>y<\/mi>\u223c<\/mo>\u03c0<\/mi>\u03b8<\/mi><\/msub>(<\/mo>\u22c5<\/mo>\u2223<\/mi>x<\/mi>)<\/mo><\/mrow><\/msub>\u2009\u2063<\/mtext>[<\/mo>R<\/mi>\u03d5<\/mi><\/msub>(<\/mo>x<\/mi>,<\/mo>y<\/mi>)<\/mo>\u2212<\/mo>\u03b2<\/mi>\u22c5<\/mo>D<\/mi>KL<\/mtext><\/msub>\u2009\u2063<\/mtext>(<\/mo>\u03c0<\/mi>\u03b8<\/mi><\/msub>(<\/mo>\u22c5<\/mo>\u2223<\/mi>x<\/mi>)<\/mo>\u2009<\/mtext>\u2225<\/mi>\u2009<\/mtext>\u03c0<\/mi>SFT<\/mtext><\/msub>(<\/mo>\u22c5<\/mo>\u2223<\/mi>x<\/mi>)<\/mo>)<\/mo><\/mrow>]<\/mo><\/mrow><\/mrow>\\mathcal{J}(\\theta) = \\mathbb{E}_{x \\sim \\mathcal{D},\\, y \\sim \\pi_\\theta(\\cdot|x)}\\!\\left[R_\\phi(x, y) – \\beta \\cdot D_{\\text{KL}}\\!\\left(\\pi_\\theta(\\cdot|x) \\,\\|\\, \\pi_{\\text{SFT}}(\\cdot|x)\\right)\\right]<\/annotation><\/semantics><\/math><\/p>\n\n\n\n

The KL penalty, weighted by coefficient \u03b2<\/mi><\/mrow>\\beta<\/annotation><\/semantics><\/math>, is critical. Without it, the policy would rapidly overfit to the reward model’s blind spots, a phenomenon called reward hacking: the model learns to produce outputs that score highly according to R<\/mi>\u03d5<\/mi><\/msub><\/mrow>R_\\phi<\/annotation><\/semantics><\/math> while being nonsensical or subtly harmful in ways the reward model did not penalise. The KL term keeps the aligned model close to the original SFT distribution, preserving its language modelling capabilities while steering it toward preferred outputs.<\/p>\n\n\n\n

This optimisation problem is solved using the Proximal Policy Optimisation (PPO) algorithm, which clips the policy gradient update to prevent excessively large steps. PPO is not the only approach; Direct Preference Optimisation (DPO), introduced in 2023, reformulates the RLHF objective as a supervised learning problem that bypasses the reward model entirely:L<\/mi>DPO<\/mtext><\/msub>(<\/mo>\u03b8<\/mi>)<\/mo>=<\/mo>\u2212<\/mo>E<\/mi>(<\/mo>x<\/mi>,<\/mo>y<\/mi>w<\/mi><\/msub>,<\/mo>y<\/mi>l<\/mi><\/msub>)<\/mo><\/mrow><\/msub>\u2009\u2063<\/mtext>[<\/mo>log<\/mi>\u2061<\/mo>\u03c3<\/mi>\u2009\u2063<\/mtext>(<\/mo>\u03b2<\/mi>log<\/mi>\u2061<\/mo>\u03c0<\/mi>\u03b8<\/mi><\/msub>(<\/mo>y<\/mi>w<\/mi><\/msub>\u2223<\/mi>x<\/mi>)<\/mo><\/mrow>\u03c0<\/mi>ref<\/mtext><\/msub>(<\/mo>y<\/mi>w<\/mi><\/msub>\u2223<\/mi>x<\/mi>)<\/mo><\/mrow><\/mfrac>\u2212<\/mo>\u03b2<\/mi>log<\/mi>\u2061<\/mo>\u03c0<\/mi>\u03b8<\/mi><\/msub>(<\/mo>y<\/mi>l<\/mi><\/msub>\u2223<\/mi>x<\/mi>)<\/mo><\/mrow>\u03c0<\/mi>ref<\/mtext><\/msub>(<\/mo>y<\/mi>l<\/mi><\/msub>\u2223<\/mi>x<\/mi>)<\/mo><\/mrow><\/mfrac>)<\/mo><\/mrow>]<\/mo><\/mrow><\/mrow>\\mathcal{L}_{\\text{DPO}}(\\theta) = -\\mathbb{E}_{(x, y_w, y_l)}\\!\\left[\\log \\sigma\\!\\left(\\beta \\log \\frac{\\pi_\\theta(y_w|x)}{\\pi_{\\text{ref}}(y_w|x)} – \\beta \\log \\frac{\\pi_\\theta(y_l|x)}{\\pi_{\\text{ref}}(y_l|x)}\\right)\\right]<\/annotation><\/semantics><\/math><\/p>\n\n\n\n

DPO has become widely adopted because it is significantly simpler to implement and more stable to train than PPO-based RLHF, while achieving comparable alignment quality on most benchmarks.<\/p>\n\n\n\n

Why InstructGPT Outperformed GPT-3<\/h4>\n\n\n\n

The RLHF result that most concisely captures the importance of alignment is this: InstructGPT (1.3B parameters, RLHF-trained) was preferred by human evaluators over GPT-3 (175B parameters, base model) 85% of the time. A model with 100\u00d7 fewer parameters that is aligned to human preferences consistently produced more useful outputs than its massive unaligned ancestor.<\/p>\n\n\n\n

This result makes precise what “alignment” means in practice. A base model has vast latent capability distributed across its weights, but no particular disposition to deploy that capability in ways that are useful or safe. Alignment is the process of concentrating and directing that capability toward the response distribution humans actually want, without catastrophically degrading the underlying language modelling capability that makes the responses coherent.<\/p>\n\n\n\n

Conclusion<\/h4>\n\n\n\n

Pre-training on next-token prediction, at compute-optimal scale guided by Chinchilla’s laws, produces a powerful but raw language model. The alignment pipeline (SFT \u2192 reward model \u2192 RLHF or DPO) shapes that raw capability into something useful, honest, and controllable. The gap between these two stages is where most of the practical engineering work in deploying frontier models resides.<\/p>\n\n\n\n

Part 5 will close the series with inference: sampling strategies, the mechanics of context window management, emergent capabilities and what they imply, and the open research frontiers such as mechanistic interpretability, reasoning models, and multimodality. These will define the next phase of LLM development.<\/p>\n\n\n\n

Part 5<\/a>: Inference, Sampling, and Emergent Behaviour \u2014 the final instalment of this series.<\/mark><\/em><\/h6>\n\n\n\n

<\/p>\n","protected":false},"excerpt":{"rendered":"

This is Part 4 of a five-part series on the internal mechanics of large language models. Part 3 covered the complete transformer block and architectural variants. Part 4 addresses how models are trained: the pre-training objective, loss functions, the Chinchilla scaling laws, and the alignment pipeline of instruction tuning and RLHF that transforms a raw language model into a useful assistant. The Pre-Training Objective The entire capability of a modern LLM emerges from one deceptively simple training objective: predict the next token. Given a sequence of tokens [t1,t2,\u2026,tn][t_1, t_2, \\ldots, t_n], the model is trained to maximise the log-probability of each token given all preceding tokens:LPT=\u22121n\u2211i=1nlog\u2061P(ti\u2223t1,\u2026,ti\u22121;\u2009\u03b8)\\mathcal{L}_{\\text{PT}} = -\\frac{1}{n} \\sum_{i=1}^{n} \\log P(t_i \\mid t_1, \\ldots, t_{i-1};\\, \\theta) This is the negative log-likelihood loss, or equivalently, cross-entropy between the model’s predicted distribution and the one-hot true distribution over the vocabulary. Minimising LPT\\mathcal{L}_{\\text{PT}} is equivalent to maximising the likelihood of the training corpus under the model. The reason this objective is so productive is that predicting the next token accurately requires solving an enormous range of implicit sub-problems. To predict the next word in a chemistry paper, the model must understand chemistry. To predict dialogue in a novel, it must model character motivation and narrative consistency. To predict the output of a code snippet, it must simulate execution. All of these capabilities emerge as instrumental sub-goals of the single next-token prediction objective, a phenomenon sometimes called the “bitter lesson” of AI: simple objectives applied at scale consistently outperform hand-crafted inductive biases. The training corpus for frontier models is correspondingly vast. GPT-3 was trained on roughly 300 billion tokens drawn from Common Crawl, WebText2, Books, and Wikipedia. Llama 3 used over 15 trillion tokens. Assembling, filtering, deduplicating, and quality-scoring a corpus at this scale is itself a major engineering undertaking, and corpus quality is widely understood to be among the most important determinants of downstream model capability, arguably more important than architectural choices at equivalent parameter counts. The Training Loop Pre-training proceeds through the standard deep learning training loop, applied at extreme scale: Forward pass: A batch of token sequences is sampled from the corpus. Each sequence is processed through the full transformer stack consisting of embedding, NNN transformer blocks, and unembedding. This produces a probability distribution over the vocabulary at each position. Loss computation: Cross-entropy loss is computed by comparing the model’s predicted distribution at each position with the true next token. Backward pass: Gradients of the loss with respect to all \u03b8\\theta\u03b8 parameters are computed via backpropagation through the entire network. Parameter update: An optimiser, called Adam or AdamW in virtually all modern LLMs, applies the gradient update:\u03b8\u2190\u03b8\u2212\u03b7\u22c5m^t\/(v^t+\u03f5)\\theta \\leftarrow \\theta – \\eta \\cdot \\hat{m}_t \/ (\\sqrt{\\hat{v}_t} + \\epsilon) where m^t\\hat{m}_t\u200b and v^t\\hat{v}_t\u200b are bias-corrected first and second moment estimates of the gradient, and \u03b7\\eta is the learning rate. AdamW adds a decoupled weight decay term \u03bb\u03b8\\lambda \\theta directly to the parameter update, separate from the gradient. This is important because standard L2 regularisation interacts badly with Adam’s adaptive scaling, while decoupled weight decay does not. The learning rate schedule typically involves a linear warmup phase (to avoid instability at the start of training when gradients are large and parameters are random) followed by a cosine decay to a minimum learning rate of roughly \u03b7min\u22480.1\u03b7max\\eta_{\\text{min}} \\approx 0.1 \\eta_{\\text{max}}\u200b:\u03b7t=\u03b7min+12(\u03b7max\u2212\u03b7min)(1+cos\u2061\u2009\u2063(\u03c0tT))\\eta_t = \\eta_{\\text{min}} + \\frac{1}{2}(\\eta_{\\text{max}} – \\eta_{\\text{min}})\\left(1 + \\cos\\!\\left(\\frac{\\pi t}{T}\\right)\\right) where TT is the total number of training steps. The cosine schedule has become the near-universal choice because it empirically outperforms linear decay and step schedules across model sizes and tasks. At frontier scale, training runs on clusters of thousands of GPUs or TPUs, employing a combination of data parallelism (each device processes a different batch, gradients are aggregated), tensor parallelism (each layer’s weight matrices are sharded across devices), and pipeline parallelism (different layers run on different devices simultaneously). GPT-3’s training required approximately 3.14 \u00d7 10\u00b2\u00b3 FLOPs and ran on 10,000 V100 GPUs. A GPT-4-class training run is estimated at well over 10\u00b2\u2075 FLOPs. Scaling Laws and the Chinchilla Result How should a fixed compute budget be allocated between model size (number of parameters NN) and training data (number of tokens DD)? This is the central question addressed by neural scaling law research. Kaplan et al. (OpenAI, 2020) established that loss scales as a power law in both NN and DD, and derived the now-famous result that, for a fixed compute budget CC, loss is minimised by scaling NN and DD in a roughly fixed ratio, with NN growing considerably faster than DD. This led to the prevailing practice of training very large models on relatively modest amounts of data, exemplified by GPT-3 (175B parameters, 300B tokens). Hoffmann et al. (DeepMind, 2022) in the “Chinchilla” paper challenged this conclusion. Using a broader range of model sizes and more careful experimental design, they derived a different optimal allocation:Nopt\u221dC0.5,Dopt\u221dC0.5N_{\\text{opt}} \\propto C^{0.5}, \\quad D_{\\text{opt}} \\propto C^{0.5} That is, compute-optimal training requires scaling parameters and tokens in a roughly 1:1 ratio. Their finding: for a given compute budget, the optimal model size is significantly smaller than previously believed, but must be trained on significantly more data. The compute-optimal ratio is approximately 20 training tokens per parameter. The practical implication was dramatic. GPT-3, by the Chinchilla analysis, was massively undertrained. A 175B parameter model should be trained on approximately 3.5 trillion tokens to be compute-optimal, not 300 billion. Chinchilla itself (70B parameters, 1.4 trillion tokens) outperformed Gopher (280B parameters) on nearly every benchmark despite using a quarter of the parameters, simply by training longer on more data. The Chinchilla result reshaped subsequent model development: Llama 1 and 2 trained smaller models on far more tokens; Llama 3 trained an 8B model on 15 trillion tokens. The scaling law itself takes the form:L(N,D)=E+AN\u03b1+BD\u03b2L(N, D) = E + \\frac{A}{N^\\alpha} + \\frac{B}{D^\\beta} where EE is the irreducible entropy of the data distribution (a lower bound on achievable loss), and AA, BB, \u03b1\\alpha, \u03b2\\beta are empirically fitted constants. The three terms represent, respectively: the noise floor of the task, the contribution of model capacity, and the contribution of data quantity. From Language Model to Assistant: Instruction Tuning and RLHF A model trained only on next-token prediction is not a useful assistant. It will complete any prompt in the style of its training distribution and it might respond to “How do I bake bread?” by generating more questions rather than an answer, because question-continuation is a common pattern in web text. Transforming a base language model into a helpful, honest, and harmless assistant requires a second training phase: alignment. The alignment pipeline used by OpenAI, Anthropic, Google DeepMind, and most frontier labs follows a three-stage process. Stage 1 \u2014 Supervised fine-tuning (SFT): Human annotators write demonstrations of ideal assistant behaviour through prompt-response pairs that exhibit the desired properties: helpfulness, accuracy, appropriate refusal of harmful requests, appropriate tone and format. The base model is then fine-tuned on this dataset using standard cross-entropy loss. SFT alone substantially improves instruction-following; InstructGPT’s SFT model already significantly outperformed the raw GPT-3 base in human evaluations. Stage 2 \u2014 Reward model training: Human annotators rank multiple model responses to the same prompt, from best to worst. A separate reward model R\u03d5R_\\phi\u200b, itself a transformer, is trained to predict these rankings. Given a prompt xx and response yy, the reward model produces a scalar score R\u03d5(x,y)\u2208RR_\\phi(x, y) \\in \\mathbb{R}. The training objective minimises a pairwise ranking loss:LRM=\u2212E(x,yw,yl)\u2009\u2063[log\u2061\u03c3\u2009\u2063(R\u03d5(x,yw)\u2212R\u03d5(x,yl))]\\mathcal{L}_{\\text{RM}} = -\\mathbb{E}_{(x, y_w, y_l)}\\!\\left[\\log \\sigma\\!\\left(R_\\phi(x, y_w) – R_\\phi(x, y_l)\\right)\\right] where ywy_w is the preferred response, yly_l\u200b is the less preferred response, and \u03c3\\sigma is the sigmoid function. This Bradley-Terry loss encourages the reward model to assign higher scores to preferred completions. Stage 3 \u2014 Reinforcement learning from human feedback (RLHF): The SFT model \u03c0\u03b8\\pi_\\theta\u200b is further fine-tuned using the reward model as a proxy for human preference. The policy gradient objective maximises expected reward while penalising deviation from the SFT model via a KL divergence constraint:J(\u03b8)=Ex\u223cD,\u2009y\u223c\u03c0\u03b8(\u22c5\u2223x)\u2009\u2063[R\u03d5(x,y)\u2212\u03b2\u22c5DKL\u2009\u2063(\u03c0\u03b8(\u22c5\u2223x)\u2009\u2225\u2009\u03c0SFT(\u22c5\u2223x))]\\mathcal{J}(\\theta) = \\mathbb{E}_{x \\sim \\mathcal{D},\\, y \\sim \\pi_\\theta(\\cdot|x)}\\!\\left[R_\\phi(x, y) – \\beta \\cdot D_{\\text{KL}}\\!\\left(\\pi_\\theta(\\cdot|x) \\,\\|\\, \\pi_{\\text{SFT}}(\\cdot|x)\\right)\\right] The KL penalty, weighted by coefficient \u03b2\\beta, is critical. Without it, the policy would rapidly overfit to the reward model’s blind spots, a phenomenon called reward hacking: the model learns to produce outputs that score highly according to R\u03d5R_\\phi while being nonsensical or subtly harmful in ways the reward model did not penalise. The KL term keeps the aligned model close to the original SFT distribution, preserving its language modelling capabilities while steering it toward preferred outputs. This optimisation problem is solved using the Proximal Policy Optimisation (PPO) algorithm, which clips the policy gradient update to prevent excessively large steps. PPO is not the only approach; Direct Preference Optimisation (DPO), introduced in 2023, reformulates the RLHF objective as a supervised learning problem that bypasses the reward model entirely:LDPO(\u03b8)=\u2212E(x,yw,yl)\u2009\u2063[log\u2061\u03c3\u2009\u2063(\u03b2log\u2061\u03c0\u03b8(yw\u2223x)\u03c0ref(yw\u2223x)\u2212\u03b2log\u2061\u03c0\u03b8(yl\u2223x)\u03c0ref(yl\u2223x))]\\mathcal{L}_{\\text{DPO}}(\\theta) = -\\mathbb{E}_{(x, y_w, y_l)}\\!\\left[\\log \\sigma\\!\\left(\\beta \\log \\frac{\\pi_\\theta(y_w|x)}{\\pi_{\\text{ref}}(y_w|x)} – \\beta \\log \\frac{\\pi_\\theta(y_l|x)}{\\pi_{\\text{ref}}(y_l|x)}\\right)\\right] DPO has become widely adopted because it is significantly simpler to implement and more stable to train than PPO-based RLHF, while achieving comparable alignment quality on most benchmarks. Why InstructGPT Outperformed GPT-3 The RLHF result that most concisely captures the importance of alignment is this: InstructGPT (1.3B parameters, RLHF-trained) was preferred by human evaluators over GPT-3 (175B parameters, base model) 85% of the time. A model with 100\u00d7 fewer parameters that is aligned to human preferences consistently produced more useful outputs than its massive unaligned ancestor. This result makes precise what “alignment” means in practice. A base model has vast latent capability distributed across its weights, but no particular disposition to deploy that capability in ways that are useful or safe. Alignment is the process of concentrating and directing that capability toward the response distribution humans actually want, without catastrophically degrading the underlying language modelling capability that makes the responses coherent. Conclusion Pre-training on next-token prediction, at compute-optimal scale guided by Chinchilla’s laws, produces a powerful but raw language model. The alignment pipeline (SFT \u2192 reward model \u2192 RLHF or DPO) shapes that raw capability into something useful, honest, and controllable. The gap between these two stages is where most of the practical engineering work in deploying frontier models resides. Part 5 will close the series with inference: sampling strategies, the mechanics of context window management, emergent capabilities and what they imply, and the open research frontiers such as mechanistic interpretability, reasoning models, and multimodality. These will define the next phase of LLM development. Part 5: Inference, Sampling, and Emergent Behaviour \u2014 the final instalment of this series.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[38],"tags":[],"class_list":["post-1045","post","type-post","status-publish","format-standard","hentry","category-ai-theory"],"_links":{"self":[{"href":"https:\/\/learnerbox.net\/blog\/wp-json\/wp\/v2\/posts\/1045","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/learnerbox.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/learnerbox.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/learnerbox.net\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/learnerbox.net\/blog\/wp-json\/wp\/v2\/comments?post=1045"}],"version-history":[{"count":3,"href":"https:\/\/learnerbox.net\/blog\/wp-json\/wp\/v2\/posts\/1045\/revisions"}],"predecessor-version":[{"id":1052,"href":"https:\/\/learnerbox.net\/blog\/wp-json\/wp\/v2\/posts\/1045\/revisions\/1052"}],"wp:attachment":[{"href":"https:\/\/learnerbox.net\/blog\/wp-json\/wp\/v2\/media?parent=1045"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/learnerbox.net\/blog\/wp-json\/wp\/v2\/categories?post=1045"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/learnerbox.net\/blog\/wp-json\/wp\/v2\/tags?post=1045"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}