LearnerBox Insights

This is the final part of a five-part series on the internal mechanics of large language models. Parts 1–4 covered tokenization, embeddings, attention, the transformer block, and the training and alignment pipeline. Part 5 addresses inference: how a trained model generates text, the mathematics of sampling, context window management, emergent capabilities, and the open research frontiers shaping the next generation of LLMs.

From Trained Weights to Generated Text

A trained, aligned LLM is a fixed mathematical function: given a sequence of input tokens, it produces a probability distribution over the next token. Inference is the process of repeatedly applying this function to generate text, while the engineering decisions made at inference time have a surprisingly large impact on the quality, diversity, safety, and cost of model outputs.

At each generation step $t$, the model computes:$$P(t_{n+1} \mid t_1, t_2, \ldots, t_n;\, \theta) = \text{softmax}(W_U \cdot \mathbf{h}_n^{(N)})$$

where $\mathbf{h}_n^{(N)} \in \mathbb{R}^d$ is the final-layer hidden state at position $n$n, $W_U \in \mathbb{R}^{|V| \times d}$ is the unembedding matrix (often tied to the transpose of the input embedding matrix $W_E$WE​), and the resulting vector of logits over $|V|$ vocabulary entries is passed through softmax to produce a probability distribution. A token is then drawn from this distribution (called the sampling step) appended to the context, and the process repeats.

The Mathematics of Sampling

How a token is selected from the output distribution is not trivial. Naive greedy decoding that always selects the highest-probability token tends to produce repetitive, degenerate outputs. Real systems use one or more of the following strategies.

Temperature scaling. Before applying softmax, the logits are divided by a temperature parameter $T > 0$T>0:$$P_T(t_{n+1} = k) = \frac{\exp(z_k / T)}{\sum_j \exp(z_j / T)}$$

As $T \to 0$, the distribution concentrates on the single highest-logit token, equivalent to greedy decoding. As $T \to \infty$, the distribution approaches uniformity with maximum entropy, maximum randomness. Values of $T \in [0.6, 1.0]$ are typical for creative generation; $T \in [0.0, 0.3]$ for precise factual or code generation tasks. Temperature does not change which token is most probable; instead, it changes how peaked or flat the distribution is.

Top-k sampling. The distribution is truncated to only the $k$ highest-probability tokens, which are then renormalised and sampled from. A typical value is $k = 40$. Top-$k$ prevents the model from sampling tokens with negligible probability, but it is sensitive to the shape of the distribution. In a uniform distribution, $k = 40$ keeps 40 of many reasonable options; in a very peaked distribution, $k = 40$ may retain tokens with near-zero probability.

**Nucleus (top-$p$) sampling.** Rather than fixing the number of tokens, nucleus sampling fixes the cumulative probability mass: the smallest set of tokens $\mathcal{V}_p$​ such that $\sum_{k \in \mathcal{V}_p} P(t_k) \geq p$ is retained and renormalised. A typical value is $p = 0.9$ or $p = 0.95$. Nucleus sampling adapts naturally to the distribution’s shape. A peaked distribution retains few tokens, a flat one retains many – this is now the dominant strategy in production systems.

Min-p sampling. A newer variant, increasingly adopted in open-source inference stacks, sets a dynamic minimum probability threshold relative to the top token: any token with $P(t_k) < p_{\min} \times P(t_{\max})$ is discarded. This keeps tokens that are meaningfully competitive with the best option while discarding genuinely improbable ones, and empirically reduces repetition more effectively than top-$p$p at matched quality.

In practice, production inference systems combine multiple strategies: temperature scaling applied first to reshape the distribution, followed by top-$p$ or min-$p$ truncation, followed by top-$k$ as a hard ceiling. The specific combination and hyperparameters are treated as tunable properties of a deployment configuration, not fixed properties of the model itself.

Context Windows and Memory Management

Every inference call operates within a context window, which is the maximum sequence length the model can process in a single forward pass. For GPT-4, the context window is 128K tokens; for Gemini 1.5 Pro, it extended to 1 million tokens; Llama 3.1 supports 128K. Context window length is not a free parameter, rather, it is constrained by the quadratic $O(n^2)$ attention cost and, critically, by KV cache memory.

As established in Part 2, the KV cache stores the key and value tensors for every previous token at every layer, avoiding redundant recomputation during autoregressive generation. The memory footprint is:$$\text{KV cache memory} = 2 \times n \times N_{\text{layers}} \times n_{\text{heads}} \times d_k \times \text{bytes per element}$$

For a model with 96 layers, 96 heads, $d_k = 128$, processing a 128K-token context in FP16: $2 \times 128000 \times 96 \times 96 \times 128 \times 2 \approx 601$ GB. This exceeds the VRAM of any single current GPU by a factor of roughly 10, which is why long-context inference requires either model parallelism across multiple devices, quantisation of KV cache entries to lower precision (INT8 or INT4), or approximate attention methods that reduce what must be stored.

Several efficient attention variants address this scaling problem. Flash Attention (Dao et al., 2022) reorders the attention computation to avoid materialising the full $n \times n$ attention matrix in high-bandwidth memory, reducing memory complexity from $O(n^2)$ to $O(n)$ while maintaining identical numerical output. Grouped-query attention (GQA), used in Llama 2 and 3, shares key and value heads across multiple query heads, reducing KV cache size by a factor of $g$ (the group size) without significant quality degradation. Sliding window attention, used in Mistral, restricts each token to attend only within a fixed window of recent tokens, allowing arbitrarily long sequences at the cost of cross-window recall.

Emergent Capabilities and Phase Transitions

One of the most scientifically consequential, and practically important, observations in LLM scaling is the phenomenon of emergent capabilities: abilities that are essentially absent in smaller models and appear abruptly as scale crosses some threshold, without being explicitly trained for.

Wei et al. (2022) documented dozens of such capabilities: few-shot chain-of-thought reasoning, multi-step arithmetic, word unscrambling, and logical deduction all exhibit sharp phase transitions: near-zero performance at sub-threshold scales, near-human performance above it. The sharpness of these transitions distinguishes emergence from smooth capability growth and has profound implications for predicting what a model trained at a given scale will be able to do.

The mechanism underlying emergence is debated. One influential account (the “grokking” hypothesis, from Power et al. at Anthropic) suggests that models first memorise training examples, then undergo a phase transition in which they discover a compressed algorithmic solution that generalises to held-out data. Another account, from Anthropic’s interpretability research, suggests that emergent capabilities correspond to the model assembling multi-step circuits, consisting of sequences of attention heads and FFN layers that compose to implement a non-trivial algorithm, and that these circuits can only form once the model has sufficient depth and width to represent all required intermediate computations simultaneously.

From an engineering standpoint, emergence creates a prediction problem. The GPT-3 scaling curve gave no warning that a GPT-4-scale model would exhibit chain-of-thought reasoning, code execution, or structured tool use. This motivates careful capability elicitation and evaluation at each new model generation —(the practice of red-teaming) as a safety and capability discovery discipline.

Chain-of-Thought and Reasoning Models

The most practically significant emergent capability is multi-step reasoning — and the discovery that it can be dramatically improved by prompting or training the model to externalise its intermediate steps.

Chain-of-thought prompting (Wei et al., 2022) established that providing examples of step-by-step reasoning in the prompt elicits dramatically better performance on multi-step arithmetic, commonsense reasoning, and symbolic manipulation tasks without any additional training. The effect is robust and scales with model size: smaller models gain little from chain-of-thought, but models above approximately 100B parameters show large, consistent improvements.

Reasoning models — exemplified by OpenAI’s o1 and o3, and Anthropic’s Claude’s extended thinking mode — take this further by training models to produce extended internal reasoning traces before generating a final answer. The training procedure uses reinforcement learning on verifiable outcomes: the model is rewarded for producing correct final answers to problems where correctness can be checked algorithmically (mathematics, code execution, formal logic), and the RL signal drives the model to develop longer, more structured reasoning traces that are instrumentally useful for producing correct outputs.

The result is a qualitatively different inference regime. A standard GPT-style model generates tokens in a single forward pass per token, with effective reasoning depth bounded by model depth. A reasoning model generates a long internal monologue, often thousands of tokens of scratchpad, before committing to an answer, effectively trading inference-time compute for reasoning quality. This represents a fundamental shift in the compute profile of AI: capability is no longer a fixed function of model size, but can be increased at inference time by allocating more tokens to reasoning.

Open Frontiers: Mechanistic Interpretability, Multimodality, and Agents

Three research frontiers are most likely to define the next phase of LLM development.

Mechanistic interpretability is the program of reverse-engineering what computations specific model components implement. The goal is to go beyond behavioural evaluation — what the model does — to structural understanding — what algorithm it uses to do it. Anthropic’s superposition hypothesis (Elhage et al., 2022) established that neural networks represent more features than they have dimensions by encoding features as directions in a high-dimensional space and tolerating controlled interference between them. The subsequent Sparse Autoencoder (SAE) program — also led at Anthropic — has made it possible to decompose model activations into sparse, interpretable features: human-understandable concepts that each activate for a specific, semantically coherent set of inputs. SAE-based interpretability tools are now being applied to frontier models at production scale, with the goal of identifying circuits responsible for deception, sycophancy, and unsafe behaviours before deployment.

Multimodality extends the transformer architecture beyond text. Vision-language models (GPT-4V, Gemini, Claude 3) process images by encoding them through a vision encoder — typically a Vision Transformer (ViT) — into a sequence of patch embeddings that are projected into the LLM’s token embedding space and concatenated with text tokens before the first transformer layer. Audio-language models (GPT-4o’s audio mode) follow an analogous pattern with a spectrogram encoder. The frontier is native multimodality — a single model trained end-to-end on text, image, audio, and video tokens simultaneously, rather than modality-specific encoders bolted onto a text backbone.

Agentic systems extend the inference loop beyond token generation to include tool use, memory retrieval, and sequential decision-making. An agent wraps an LLM in a loop: the model generates text that includes structured tool calls (web search, code execution, database queries); the tool results are appended to the context; the model continues generating, possibly issuing further tool calls, until it produces a final response. Multi-agent systems extend this further: multiple LLM instances, each with different system prompts and tool access, communicate with each other through structured message passing. The engineering challenges are significant — context management, tool reliability, error recovery, and safety alignment in agentic settings are all active areas of research — but the capability ceiling of agentic LLMs is substantially higher than single-turn generation.

A Map of the Whole

Across this five-part series, we have traced the complete path from raw string to generated output:

1. Tokenization — BPE or SentencePiece converts text into integer token IDs from a vocabulary of 32K–256K entries.
2. Embedding — each token ID is projected into a $d$d-dimensional vector; positional information is added via learned embeddings, sinusoidal encoding, or RoPE.
3. Attention — $N$N transformer blocks each apply multi-head self-attention: Q, K, V projections; scaled dot products; causal masking; softmax; value aggregation.
4. Feed-forward network — each block applies a position-wise two-layer MLP with SwiGLU activation, implementing key-value memory over learned factual associations.
5. Training — next-token prediction loss over trillions of tokens; AdamW with cosine decay; Chinchilla-optimal compute allocation; distributed parallelism at scale.
6. Alignment — SFT on human demonstrations, reward model training on preference rankings, RLHF with PPO or DPO to concentrate capability toward preferred outputs.
7. Inference — temperature scaling, nucleus or min-$p$p sampling, KV cache management, and extended chain-of-thought reasoning to generate high-quality outputs efficiently.

The transformer is a simple, composable architectural primitive. What it does, when trained at sufficient scale on sufficient data, and aligned to human preferences, is not simple at all.

This concludes the five-part series on how large language models work. If you have found this series useful, the natural next steps are the mechanistic interpretability literature (Anthropic’s Transformer Circuits thread), the Chinchilla and scaling laws papers, and Neel Nanda’s open-source interpretability tooling at TransformerLens.

This is Part 4 of a five-part series on the internal mechanics of large language models. Part 3 covered the complete transformer block and architectural variants. Part 4 addresses how models are trained: the pre-training objective, loss functions, the Chinchilla scaling laws, and the alignment pipeline of instruction tuning and RLHF that transforms a raw language model into a useful assistant.

The Pre-Training Objective

The entire capability of a modern LLM emerges from one deceptively simple training objective: predict the next token. Given a sequence of tokens $[t_1, t_2, \ldots, t_n]$, the model is trained to maximise the log-probability of each token given all preceding tokens:$$\mathcal{L}_{\text{PT}} = -\frac{1}{n} \sum_{i=1}^{n} \log P(t_i \mid t_1, \ldots, t_{i-1};\, \theta)$$

This is the negative log-likelihood loss, or equivalently, cross-entropy between the model’s predicted distribution and the one-hot true distribution over the vocabulary. Minimising $\mathcal{L}_{\text{PT}}$ is equivalent to maximising the likelihood of the training corpus under the model.

The reason this objective is so productive is that predicting the next token accurately requires solving an enormous range of implicit sub-problems. To predict the next word in a chemistry paper, the model must understand chemistry. To predict dialogue in a novel, it must model character motivation and narrative consistency. To predict the output of a code snippet, it must simulate execution. All of these capabilities emerge as instrumental sub-goals of the single next-token prediction objective, a phenomenon sometimes called the “bitter lesson” of AI: simple objectives applied at scale consistently outperform hand-crafted inductive biases.

The training corpus for frontier models is correspondingly vast. GPT-3 was trained on roughly 300 billion tokens drawn from Common Crawl, WebText2, Books, and Wikipedia. Llama 3 used over 15 trillion tokens. Assembling, filtering, deduplicating, and quality-scoring a corpus at this scale is itself a major engineering undertaking, and corpus quality is widely understood to be among the most important determinants of downstream model capability, arguably more important than architectural choices at equivalent parameter counts.

The Training Loop

Pre-training proceeds through the standard deep learning training loop, applied at extreme scale:

Forward pass: A batch of token sequences is sampled from the corpus. Each sequence is processed through the full transformer stack consisting of embedding, $N$N transformer blocks, and unembedding. This produces a probability distribution over the vocabulary at each position.

Loss computation: Cross-entropy loss is computed by comparing the model’s predicted distribution at each position with the true next token.

Backward pass: Gradients of the loss with respect to all $\theta$θ parameters are computed via backpropagation through the entire network.

Parameter update: An optimiser, called Adam or AdamW in virtually all modern LLMs, applies the gradient update:$$\theta \leftarrow \theta – \eta \cdot \hat{m}_t / (\sqrt{\hat{v}_t} + \epsilon)$$

where $\hat{m}_t$​ and $\hat{v}_t$​ are bias-corrected first and second moment estimates of the gradient, and $\eta$ is the learning rate. AdamW adds a decoupled weight decay term $\lambda \theta$ directly to the parameter update, separate from the gradient. This is important because standard L2 regularisation interacts badly with Adam’s adaptive scaling, while decoupled weight decay does not.

The learning rate schedule typically involves a linear warmup phase (to avoid instability at the start of training when gradients are large and parameters are random) followed by a cosine decay to a minimum learning rate of roughly $\eta_{\text{min}} \approx 0.1 \eta_{\text{max}}$​:$$\eta_t = \eta_{\text{min}} + \frac{1}{2}(\eta_{\text{max}} – \eta_{\text{min}})\left(1 + \cos\!\left(\frac{\pi t}{T}\right)\right)$$

where $T$ is the total number of training steps. The cosine schedule has become the near-universal choice because it empirically outperforms linear decay and step schedules across model sizes and tasks.

At frontier scale, training runs on clusters of thousands of GPUs or TPUs, employing a combination of data parallelism (each device processes a different batch, gradients are aggregated), tensor parallelism (each layer’s weight matrices are sharded across devices), and pipeline parallelism (different layers run on different devices simultaneously). GPT-3’s training required approximately 3.14 × 10²³ FLOPs and ran on 10,000 V100 GPUs. A GPT-4-class training run is estimated at well over 10²⁵ FLOPs.

Scaling Laws and the Chinchilla Result

How should a fixed compute budget be allocated between model size (number of parameters $N$) and training data (number of tokens $D$)? This is the central question addressed by neural scaling law research.

Kaplan et al. (OpenAI, 2020) established that loss scales as a power law in both $N$ and $D$, and derived the now-famous result that, for a fixed compute budget $C$, loss is minimised by scaling $N$ and $D$ in a roughly fixed ratio, with $N$ growing considerably faster than $D$. This led to the prevailing practice of training very large models on relatively modest amounts of data, exemplified by GPT-3 (175B parameters, 300B tokens).

Hoffmann et al. (DeepMind, 2022) in the “Chinchilla” paper challenged this conclusion. Using a broader range of model sizes and more careful experimental design, they derived a different optimal allocation:$$N_{\text{opt}} \propto C^{0.5}, \quad D_{\text{opt}} \propto C^{0.5}$$

That is, compute-optimal training requires scaling parameters and tokens in a roughly 1:1 ratio. Their finding: for a given compute budget, the optimal model size is significantly smaller than previously believed, but must be trained on significantly more data. The compute-optimal ratio is approximately 20 training tokens per parameter.

The practical implication was dramatic. GPT-3, by the Chinchilla analysis, was massively undertrained. A 175B parameter model should be trained on approximately 3.5 trillion tokens to be compute-optimal, not 300 billion. Chinchilla itself (70B parameters, 1.4 trillion tokens) outperformed Gopher (280B parameters) on nearly every benchmark despite using a quarter of the parameters, simply by training longer on more data.

The Chinchilla result reshaped subsequent model development: Llama 1 and 2 trained smaller models on far more tokens; Llama 3 trained an 8B model on 15 trillion tokens. The scaling law itself takes the form:$$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$$

where $E$ is the irreducible entropy of the data distribution (a lower bound on achievable loss), and $A$, $B$, $\alpha$, $\beta$ are empirically fitted constants. The three terms represent, respectively: the noise floor of the task, the contribution of model capacity, and the contribution of data quantity.

From Language Model to Assistant: Instruction Tuning and RLHF

A model trained only on next-token prediction is not a useful assistant. It will complete any prompt in the style of its training distribution and it might respond to “How do I bake bread?” by generating more questions rather than an answer, because question-continuation is a common pattern in web text. Transforming a base language model into a helpful, honest, and harmless assistant requires a second training phase: alignment.

The alignment pipeline used by OpenAI, Anthropic, Google DeepMind, and most frontier labs follows a three-stage process.

Stage 1 — Supervised fine-tuning (SFT): Human annotators write demonstrations of ideal assistant behaviour through prompt-response pairs that exhibit the desired properties: helpfulness, accuracy, appropriate refusal of harmful requests, appropriate tone and format. The base model is then fine-tuned on this dataset using standard cross-entropy loss. SFT alone substantially improves instruction-following; InstructGPT’s SFT model already significantly outperformed the raw GPT-3 base in human evaluations.

Stage 2 — Reward model training: Human annotators rank multiple model responses to the same prompt, from best to worst. A separate reward model $R_\phi$​, itself a transformer, is trained to predict these rankings. Given a prompt $x$ and response $y$, the reward model produces a scalar score $R_\phi(x, y) \in \mathbb{R}$. The training objective minimises a pairwise ranking loss:$$\mathcal{L}_{\text{RM}} = -\mathbb{E}_{(x, y_w, y_l)}\!\left[\log \sigma\!\left(R_\phi(x, y_w) – R_\phi(x, y_l)\right)\right]$$

where $y_w$ is the preferred response, $y_l$​ is the less preferred response, and $\sigma$ is the sigmoid function. This Bradley-Terry loss encourages the reward model to assign higher scores to preferred completions.

Stage 3 — Reinforcement learning from human feedback (RLHF): The SFT model $\pi_\theta$​ is further fine-tuned using the reward model as a proxy for human preference. The policy gradient objective maximises expected reward while penalising deviation from the SFT model via a KL divergence constraint:$$\mathcal{J}(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot|x)}\!\left[R_\phi(x, y) – \beta \cdot D_{\text{KL}}\!\left(\pi_\theta(\cdot|x) \,\|\, \pi_{\text{SFT}}(\cdot|x)\right)\right]$$

The KL penalty, weighted by coefficient $\beta$, is critical. Without it, the policy would rapidly overfit to the reward model’s blind spots, a phenomenon called reward hacking: the model learns to produce outputs that score highly according to $R_\phi$ while being nonsensical or subtly harmful in ways the reward model did not penalise. The KL term keeps the aligned model close to the original SFT distribution, preserving its language modelling capabilities while steering it toward preferred outputs.

This optimisation problem is solved using the Proximal Policy Optimisation (PPO) algorithm, which clips the policy gradient update to prevent excessively large steps. PPO is not the only approach; Direct Preference Optimisation (DPO), introduced in 2023, reformulates the RLHF objective as a supervised learning problem that bypasses the reward model entirely:$$\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}_{(x, y_w, y_l)}\!\left[\log \sigma\!\left(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} – \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)\right]$$

DPO has become widely adopted because it is significantly simpler to implement and more stable to train than PPO-based RLHF, while achieving comparable alignment quality on most benchmarks.

Why InstructGPT Outperformed GPT-3

The RLHF result that most concisely captures the importance of alignment is this: InstructGPT (1.3B parameters, RLHF-trained) was preferred by human evaluators over GPT-3 (175B parameters, base model) 85% of the time. A model with 100× fewer parameters that is aligned to human preferences consistently produced more useful outputs than its massive unaligned ancestor.

This result makes precise what “alignment” means in practice. A base model has vast latent capability distributed across its weights, but no particular disposition to deploy that capability in ways that are useful or safe. Alignment is the process of concentrating and directing that capability toward the response distribution humans actually want, without catastrophically degrading the underlying language modelling capability that makes the responses coherent.

Conclusion

Pre-training on next-token prediction, at compute-optimal scale guided by Chinchilla’s laws, produces a powerful but raw language model. The alignment pipeline (SFT → reward model → RLHF or DPO) shapes that raw capability into something useful, honest, and controllable. The gap between these two stages is where most of the practical engineering work in deploying frontier models resides.

Part 5 will close the series with inference: sampling strategies, the mechanics of context window management, emergent capabilities and what they imply, and the open research frontiers such as mechanistic interpretability, reasoning models, and multimodality. These will define the next phase of LLM development.

Part 5: Inference, Sampling, and Emergent Behaviour — the final instalment of this series.

This is Part 3 of a five-part series on the internal mechanics of large language models. Part 2 derived scaled dot-product attention and the KV cache. Part 3 traces the complete transformer block consisting of feed-forward networks, layer normalisation, and residual streams; it examines the architectural differences between decoder-only and encoder-decoder models.

From Attention Output to Transformer Block

At the end of Part 2, self-attention had produced an output matrix $O \in \mathbb{R}^{n \times d}$ or one updated vector per token, now containing contextual information drawn from every other position in the sequence. But this is only the first half of a single transformer block. Before the signal moves to the next layer, it passes through three more components: a residual connection, layer normalisation, and a position-wise feed-forward network. Together with multi-head attention, these four elements constitute the repeating unit of every transformer-based LLM, stacked $N$ times to form the full model.

Understanding each component precisely is important not just for implementation, but for diagnosing failure modes, interpreting mechanistic interpretability research, and making informed decisions about architectural variants when selecting or fine-tuning models.

Residual Connections

The residual connection, introduced in ResNets for computer vision and adopted into transformers from the outset, is deceptively simple. Rather than passing the attention output $O$ directly to the next component, the original input $X$ is added back:$$X’ = X + \text{MultiHead}(X)$$

This addition of the residual or skip connection has several important consequences.

First, it solves the vanishing gradient problem for very deep networks. During backpropagation, gradients flow backward through the network. Without residual connections, multiplying many Jacobians together through $N$N sequential non-linear transformations causes gradient magnitudes to decay exponentially, making early layers extremely slow to learn. The residual path provides a direct gradient highway that bypasses each block, so that gradients can flow backward through the addition operation unchanged, regardless of how many layers separate the loss from the input.

Second, and more subtly, residual connections give the network an inductive bias toward learning incremental transformations. Rather than learning to reconstruct the full representation from scratch at each layer, each block learns only the delta or the correction to add to the existing representation. In mechanistic interpretability, researchers conceptualise the transformer as a residual stream: a $d$-dimensional vector that is progressively written to and read from by successive attention heads and feed-forward layers, each contributing small structured updates. This framing, developed extensively by Anthropic, EleutherAI, and Neel Nanda’s research group, has proven highly productive for understanding what specific components of large models actually compute.

Layer Normalisation

Before and/or after each sub-component, layer normalisation is applied. For a vector $\mathbf{x} \in \mathbb{R}^d$:$$\text{LayerNorm}(\mathbf{x}) = \gamma \odot \frac{\mathbf{x} – \mu}{\sigma + \epsilon} + \beta$$

where $\mu = \frac{1}{d}\sum_i x_i$ is the mean, $\sigma = \sqrt{\frac{1}{d}\sum_i (x_i – \mu)^2}$ is the standard deviation, $\epsilon$ is a small constant for numerical stability, and $\gamma, \beta \in \mathbb{R}^d$ are learned scale and shift parameters.

Layer normalisation stabilises training by preventing the internal covariate shift problem, which is the tendency for the distribution of a layer’s inputs to change as upstream parameters update, forcing each layer to continuously readapt. By normalising each token’s representation independently across the feature dimension (as opposed to batch normalisation, which normalises across the batch dimension), layer norm is compatible with variable-length sequences and works correctly with batch size 1, making it well-suited for autoregressive inference.

Two conventions exist for where layer norm is placed within the block:

Post-norm (original Transformer, “Attention Is All You Need”): LayerNorm is applied after the residual addition:$$X’ = \text{LayerNorm}(X + \text{MultiHead}(X))$$

Pre-norm (GPT-2, GPT-3, most modern LLMs): LayerNorm is applied to the input before the sub-component:$$X’ = X + \text{MultiHead}(\text{LayerNorm}(X))$$

Pre-norm has become the dominant convention because it produces more stable training dynamics at large scale. Gradients flow more cleanly through the unmodified residual path, and the model is less sensitive to initialisation and learning rate choice. The architectural difference is subtle but has meaningful empirical consequences; models trained with post-norm often require careful learning rate warmup schedules that pre-norm models do not.

The Feed-Forward Network

After the attention sub-layer (with its own residual connection and layer norm), each transformer block applies a position-wise feed-forward network (FFN). The term “position-wise” means the same two-layer MLP is applied independently to each token’s vector, and the FFN has no cross-token interactions, unlike attention.

The standard formulation:$$\text{FFN}(\mathbf{x}) = W_2 \cdot \sigma(W_1 \mathbf{x} + \mathbf{b}_1) + \mathbf{b}_2$$

where $W_1 \in \mathbb{R}^{d_{\text{ff}} \times d}$, $W_2 \in \mathbb{R}^{d \times d_{\text{ff}}}$​, and $d_{\text{ff}}$​ is the intermediate dimension, typically $4d$. In GPT-3, $d = 12288$ and $d_{\text{ff}} = 49152$, giving each FFN roughly $4 \times 12288^2 \approx 603$ million parameters, more than the attention sub-layer at that layer.

The activation function $\sigma$σ has evolved across generations. The original transformer used ReLU. GPT-2 and GPT-3 used GeLU (Gaussian Error Linear Unit), which approximates ReLU with a smooth, non-zero gradient for slightly negative inputs:$$\text{GeLU}(x) = x \cdot \Phi(x) \approx 0.5x\!\left(1 + \tanh\!\left(\sqrt{\frac{2}{\pi}}\left(x + 0.044715x^3\right)\right)\right)$$

where $\Phi$ is the standard normal CDF. More recent models use SwiGLU, a gated variant introduced by Noam Shazeer in 2020 and adopted in PaLM, Llama, and most frontier models:$$\text{SwiGLU}(\mathbf{x}) = \text{SiLU}(W_1 \mathbf{x}) \odot (W_3 \mathbf{x})$$

where $\text{SiLU}(x) = x \cdot \sigma(x)$ is the Sigmoid Linear Unit and $W_3$​ is an additional learned gate projection. SwiGLU consistently outperforms GeLU on downstream benchmarks, though the underlying reason remains an active area of research. When using SwiGLU, $d_{\text{ff}}$​ is typically set to $\frac{8}{3}d$ rather than $4d$ to keep parameter counts comparable across architectures.

What does the FFN actually learn? Mechanistic interpretability work, particularly from Anthropic and the Neel Nanda group, has provided strong evidence that FFN layers function as key-value memories. Each row of $W_1$​ acts as a pattern detector (a “key”), and the corresponding row of $W_2$ stores an associated “value” that gets added to the residual stream when that pattern is detected. This explains why larger FFNs improve factual recall (they have more memory slots) and why deleting specific FFN neurons can surgically remove specific factual associations from a model’s behaviour.

The Complete Transformer Block

Assembling all components, the full pre-norm transformer block for a decoder-only model is:$$\mathbf{a} = \mathbf{x} + \text{MultiHead}(\text{LayerNorm}(\mathbf{x}))$$ $$\mathbf{x}’ = \mathbf{a} + \text{FFN}(\text{LayerNorm}(\mathbf{a}))$$

This two-step computation, attention followed by FFN, each with its own pre-norm and residual, is repeated $N$ times. In GPT-3, $N = 96$. In Llama 3 70B, $N = 80$. In a typical GPT-4-class model, $N$ is believed to be in the range of 96–120 layers. Each repetition allows the model to build increasingly abstract representations. Early layers handle low-level syntactic patterns, while later layers encode semantic and factual content.

Decoder-Only vs Encoder-Decoder Architecture

Two principal transformer architectures are in widespread use, differing in how they process input and generate output.

Decoder-only models (GPT, Llama, Mistral, Gemma, Falcon) use a single stack of transformer blocks with causal (masked) self-attention. Every token can attend only to previous tokens. The entire input prompt and the generated output share the same context window and are processed by the same stack. At inference time, the model autoregressively generates one token at a time, each conditioned on all prior tokens. This architecture is dominant for general-purpose language models because it scales efficiently and the training objective of next-token prediction across the entire sequence is simple, scalable, and empirically powerful.

Encoder-decoder models (T5, BART, mT5, the original Transformer for machine translation) use two distinct stacks. The encoder processes the full input with bidirectional attention. Every token can attend to every other token simultaneously, with no causal mask. The encoded representation is then passed to the decoder via a cross-attention mechanism: each decoder token’s query attends to the encoder’s key-value pairs in addition to previously generated decoder tokens. The cross-attention block takes the form:$$\text{CrossAttn}(Q_{\text{dec}},\ K_{\text{enc}},\ V_{\text{enc}}) = \text{softmax}\!\left(\frac{Q_{\text{dec}} K_{\text{enc}}^\top}{\sqrt{d_k}}\right)V_{\text{enc}}$$

This architecture was originally designed for sequence-to-sequence tasks such as translation, summarisation, or question answering, where the full input is available before generation begins. It has largely been supplanted by decoder-only models for general language tasks, primarily because the instruction-tuned decoder-only paradigm proved more flexible and scalable. However, encoder-decoder models retain advantages for tasks requiring deep understanding of a fixed input, such as structured prediction, constrained generation, and certain retrieval tasks.

A hybrid, the encoder-only model (BERT, RoBERTa, DeBERTa), uses bidirectional attention with no autoregressive generation. It is optimised for discriminative tasks such as classification, named entity recognition, or semantic similarity, rather than generation, and remains widely used in production embedding pipelines and retrieval systems.

Depth, Width, and the Scaling Hypothesis

The transformer block’s design raises a natural question: does it matter whether a model has more layers (depth) or larger hidden dimension (width)? Empirically, the two interact in complex ways, and the optimal allocation of a fixed parameter budget between depth and width is one of the central questions addressed by scaling law research, which is the subject of Part 4.

What is clear from both theory and experiment is that depth and width serve different functions. Width increases the model’s representational capacity at each layer by the size of the residual stream and the richness of the key-value memory in the FFN. Depth increases the number of sequential computational steps the model can take, which means the number of times attention and FFN transformations can be composed. Tasks requiring multi-step reasoning appear to benefit disproportionately from depth, while factual recall scales more uniformly with total parameter count regardless of how it is distributed between depth and width.

Conclusion

The transformer block is elegant in its modularity: a residual stream, two sub-components (attention and FFN), and two layer normalisation steps. The full model is simply this block repeated $N$ times. But the interactions between these components, including the gradient flow through residual connections, the factual storage in FFN key-value memories, and the progressive abstraction through depth, give rise to capabilities that are not predictable from any single component in isolation.

Part 4 will move from architecture to training: the next-token prediction objective, loss functions, the Chinchilla scaling laws, and the dramatic effect of RLHF and instruction tuning on model behaviour, explaining why InstructGPT, at a fraction of GPT-3’s parameter count, consistently outperformed it.

Coming next in the AI Engineering series is Part 4: Training, Scaling Laws, and RLHF.

This is Part 2 of a five-part series on the internal mechanics of large language models. Part 1 covered tokenization, embeddings, and positional encoding. Part 2 derives the self-attention mechanism from first principles, covers multi-head attention, and examines the KV cache.

The Problem Attention Solves

At the end of Part 1, we had a matrix $X \in \mathbb{R}^{n \times d}$, which was a one $d$d-dimensional vector per token, encoding both semantic identity and position. The first transformer layer receives this matrix and must do something critical: allow each token to incorporate information from every other token in the sequence before passing its updated representation to the feed-forward network.

This is the problem that self-attention solves. Earlier sequence models such as LSTMs and GRUs processed tokens sequentially, which meant that a word at position 1 had to wait for its influence to propagate through every intermediate hidden state to reach position 512. Long-range dependencies were difficult to learn because the gradient had to flow through hundreds of recurrent steps. Attention eliminates this bottleneck entirely by allowing any token to attend directly to any other token in a single operation, regardless of distance.

Queries, Keys, and Values

The first step of self-attention is to project each token’s embedding into three separate representations using learned weight matrices:$$Q = XW^Q, \quad K = XW^K, \quad V = XW^V$$where $W^Q, W^K, W^V \in \mathbb{R}^{d \times d_k}$​ are learned projection matrices, and $d_k$​ is the dimension of the query and key space (typically $d_k = d / h$ where $h$ is the number of attention heads, more on that shortly).

The resulting matrices $Q, K, V \in \mathbb{R}^{n \times d_k}$ are called the Query, Key, and Value matrices respectively.

The intuition behind this decomposition is often explained through an analogy: think of each token’s query vector as a question it is asking (“what context do I need?”), each token’s key vector as an advertisement of its content (“here is what I contain”), and each token’s value vector as the actual information it contributes when attended to (“here is what I give you if you attend to me”). The query-key interaction determines how much each token attends to every other; the values are what actually gets aggregated.

Scaled Dot-Product Attention

Given $Q$, $K$, and $V$, the attention output is computed as:$$\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V$$Let’s unpack this step by step.

Step 1 — Dot products: $QK^\top \in \mathbb{R}^{n \times n}$ is a matrix of raw attention scores. Entry $(i, j)$ is the dot product between the query vector of token $i$ and the key vector of token $j$:$$s_{ij} = \mathbf{q}_i \cdot \mathbf{k}_j$$A high dot product means token $i$ finds token $j$ highly relevant. This is the mechanism through which, for example, a pronoun “she” can attend strongly to the noun “Alice” it refers to three sentences earlier.

Step 2 — Scaling: The dot products are divided by $\sqrt{d_k}$​​. Without this scaling, when $d_k$​ is large, the dot products grow large in magnitude, pushing the softmax into regions where gradients become vanishingly small — a variant of the vanishing gradient problem. The $\sqrt{d_k}$​​ factor keeps the variance of the dot products approximately constant regardless of model dimensionality.

Step 3 — Causal masking (decoder-only models): In autoregressive models like GPT, token $i$ must not attend to any token $j > i$ meaning that it cannot look into the future. Before applying softmax, a mask is added:$$s_{ij} \leftarrow s_{ij} + m_{ij}, \quad m_{ij} = \begin{cases} 0 & \text{if } j \leq i \\ -\infty & \text{if } j > i \end{cases}$$The $-\infty$ values become zero after softmax, effectively zeroing out future positions.

Step 4 — Softmax: Each row of the masked score matrix is passed through softmax:$$\alpha_{ij} = \frac{\exp(s_{ij})}{\sum_{k=1}^{n} \exp(s_{ik})}$$The resulting matrix $A \in \mathbb{R}^{n \times n}$ is the attention weight matrix. Each row sums to 1 and can be interpreted as a probability distribution over the sequence: the probability that token $i$ attends to each position.

Step 5 — Value aggregation: The output for token $i$ is a weighted sum of all value vectors:$$\mathbf{o}_i = \sum_{j=1}^{n} \alpha_{ij} \mathbf{v}_j$$In matrix form: $O = AV \in \mathbb{R}^{n \times d_k}$​. Each output vector is a contextualised representation of its token, so that the same word “bank” will produce a different output vector in “river bank” versus “central bank” because its attention weights will be distributed differently across the surrounding context.

Multi-Head Attention

A single attention head computes one set of query-key-value interactions. But different aspects of meaning may require different attention patterns simultaneously: a token might need to attend to its syntactic head, its semantic antecedent, and its positional neighbours all at once. Multi-head attention runs $h$ attention operations in parallel:$$\text{head}_i = \text{Attention}(QW_i^Q,\ KW_i^K,\ VW_i^V)$$ $$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h)W^O$$where $W_i^Q, W_i^K, W_i^V \in \mathbb{R}^{d \times d_k}$​ are per-head projection matrices, each head operates in a $d_k = d/h$ dimensional subspace, and $W^O \in \mathbb{R}^{d \times d}$ is a final output projection that mixes the concatenated heads back into the full $d$-dimensional space.

In GPT-3, $d = 12288$ and $h = 96$, so each head operates in a $d_k = 128$ dimensional subspace. The 96 heads learn to specialize: empirical work in mechanistic interpretability has identified heads that track syntactic subject-verb agreement, heads that copy tokens from earlier in context, heads that attend to the most recent noun phrase, and heads that implement induction — recognizing when a pattern seen earlier in the context is repeating.

The computational cost of full self-attention is $O(n^2 d)$, or quadratic in sequence length. For a 128K-token context window, the attention matrix alone has $128000^2 \approx 16$ billion entries, making naive computation prohibitively expensive. Efficient attention variants such as Flash Attention, Sparse Attention, Sliding Window Attention address this scaling problem, which we’ll cover in Part 3 alongside the full transformer block.

The KV Cache

During inference, a decoder-only model generates one token at a time. At step $t$, the model processes the full sequence $[t_1, \ldots, t_t]$ and predicts $t_{t+1}$​. Without optimization, this would require recomputing the key and value matrices for all previous tokens at every step so thar cost grows as $O(t^2)$ per generation.

The KV cache eliminates this redundancy. Since the key and value representations of tokens $t_1, \ldots, t_{t-1}$​do not change between steps (they depend only on those tokens’ positions and embeddings, which are fixed once generated), they can be cached in memory and reused. At step $t$, only the new token’s $Q$, $K$, $V$ need to be computed; the cached $K$ and $V$ matrices from previous steps are concatenated and the attention is computed against the full cached sequence.

This reduces per-step inference cost from $O(t^2)$ to $O(t)$ in attention computation, but at the cost of memory. A KV cache for a 128K-token context with 96 heads, 128 layers, and FP16 precision occupies on the order of tens of gigabytes. Managing KV cache memory is one of the primary challenges in deploying frontier models efficiently at scale. It is why inference hardware for long-context models requires far more VRAM than a naive parameter count would suggest.

What Attention Actually Learns

It is tempting to describe attention as a mechanism that “understands” language. It is more precise, and more useful for engineering purposes, to say that attention is a differentiable, content-based memory retrieval system. The model learns, through gradient descent on the next-token prediction objective, to configure its $W^Q$, $W^K$, $W^V$ matrices such that the right values get retrieved for the right queries.

The mechanism has no built-in notion of syntax, coreference, or meaning. All of that emerges from training dynamics or from the statistical regularities of billions of documents. Understanding this distinction is important for debugging model failures: when an LLM makes a coreference error or loses track of a constraint stated early in a long context, the failure mode is almost always traceable to attention weights that were distributed incorrectly, either because the training distribution underrepresented that pattern, or because the KV cache compression strategy discarded the relevant context.

Conclusion

Self-attention is the architectural innovation that made modern LLMs possible. By allowing every token to attend directly to every other token in a single parallelisable matrix operation, it solves the long-range dependency problem that defeated earlier sequential architectures. Multi-head attention extends this by learning multiple independent attention patterns simultaneously, while the KV cache makes autoregressive inference tractable at scale.

Part 3 will take the attention output $O \in \mathbb{R}^{n \times d}$ and trace it through the rest of the transformer block: the feed-forward network, layer normalisation, residual connections, and the architectural choices (decoder-only versus encoder-decoder) that distinguish GPT-style models from BERT-style ones.

Coming next in the AI Engineering series is Part 3: The Transformer Block and Architecture.

This is Part 1 of a five-part deep-dive series on the internal mechanics of large language models. This series is written for engineers, ML practitioners, and technically advanced readers. Part 1 covers tokenization and embeddings — the first two stages through which raw text becomes a mathematical object a neural network can reason over.

Why Text Must Be Transformed

Neural networks are, at their core, mathematical functions that operate on tensors, which are multi-dimensional arrays of floating-point numbers. They cannot accept raw text strings as input. Before a language model can do anything useful, it must convert a sequence of characters into a sequence of numbers. This two-step process of tokenization followed by embedding is where all LLM processing begins, and understanding it deeply is prerequisite to understanding everything that follows.

Stage 1: Tokenization

What a Tokenizer Does

A tokenizer breaks a raw string into a sequence of discrete units called tokens, and maps each token to an integer index in a fixed vocabulary. Given the input string:

"The cat sat on the mat"

a tokenizer might produce the token sequence: [464, 5171, 7231, 322, 464, 8114]

Each integer is an index into a vocabulary table of size $|V|$, typically between 32,000 and 100,000 entries for modern models. GPT-4 uses a vocabulary of approximately 100,277 tokens. Llama 3 uses 128,256.

Byte-Pair Encoding (BPE)

The dominant tokenization algorithm used in modern LLMs is Byte-Pair Encoding (BPE), originally a data compression algorithm adapted for NLP by Sennrich et al. in 2016.

BPE builds its vocabulary through an iterative merging process:

1. Initialise the vocabulary with every individual character (or byte) in the training corpus.
2. Count all adjacent symbol pairs across the corpus.
3. Merge the most frequent pair into a single new symbol.
4. Repeat steps 2–3 until the vocabulary reaches the target size $|V|$∣V∣.

The result is a vocabulary that contains individual characters, common subwords, and frequent whole words. The word “tokenization” might be represented as three tokens: token, ization is further split into iz, ation depending on training corpus frequency.

This has a critical practical consequence: no word is ever truly unknown to a BPE tokenizer. Any string, including code, URLs, foreign scripts, or novel proper nouns, can be decomposed into its constituent bytes if nothing else matches. This is why BPE-based models generalise to inputs they have never seen verbatim.

Token Fertility and Efficiency

Not all languages tokenize equally efficiently. English text typically tokenizes at roughly 0.75 tokens per word. Languages with richer morphology, such as Finnish, Turkish, Arabic, or non-Latin scripts, tokenize less efficiently, sometimes requiring 3–5 tokens per word. This has practical implications: a context window of 128,000 tokens holds far more English prose than it does Thai or Arabic text, a bias baked into the model’s architecture from the vocabulary construction stage.

The fertility of a tokenizer, or the average number of tokens per word, is a meaningful measure of how well it serves a given language or domain. Code-optimised models like DeepSeek Coder use vocabularies with explicit code tokens to reduce the fertility of common programming constructs.

WordPiece and SentencePiece

Two notable alternatives to BPE are worth knowing:

WordPiece, used in BERT and its derivatives, is similar to BPE but merges pairs that maximise the likelihood of the training data under a language model, rather than simply the most frequent pair. This produces slightly different vocabulary distributions.

SentencePiece, used in models including Llama and T5, treats the input as a raw unicode byte stream with no pre-tokenization step (no whitespace splitting). This makes it language-agnostic and particularly well-suited for multilingual models.

Stage 2: Token Embeddings

The Embedding Matrix

Once tokenization has produced a sequence of integer indices $[t_1, t_2, \ldots, t_n]$, each index must be converted into a dense vector. This is done via an embedding matrix $W_E \in \mathbb{R}^{|V| \times d}$, where $d$ is the model’s hidden dimension (also called the embedding dimension or $d_{\text{model}}$​).

The embedding for token $t_i$ti​ is simply a row lookup:$$\mathbf{e}_i = W_E[t_i] \in \mathbb{R}^d$$In GPT-3, $d = 12288$. In Llama 3 8B, $d = 4096$. This single matrix, learned entirely from data during pre-training, is responsible for encoding the semantic relationships between all tokens in the vocabulary.

The number of parameters in the embedding matrix alone is $|V| \times d$. For GPT-4’s approximate configuration, that is $100{,}277 \times d_{\text{model}}$ which is often hundreds of millions of parameters just for this one component.

Why Dense Vectors Work: The Geometry of Meaning

The remarkable property of learned embeddings is that semantic relationships emerge as geometric relationships in $\mathbb{R}^d$. The classic demonstration is the linear analogy:

$$\mathbf{e}(\text{“king”}) – \mathbf{e}(\text{“man”}) + \mathbf{e}(\text{“woman”}) \approx \mathbf{e}(\text{“queen”})$$

This is not programmed. It emerges from the statistical structure of co-occurrence patterns in the training corpus. Words that appear in similar contexts end up with similar embedding vectors, measured by cosine similarity:$$\text{sim}(\mathbf{u}, \mathbf{v}) = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\| \|\mathbf{v}\|}$$In high-dimensional spaces, this geometry becomes extraordinarily rich. Directions in embedding space can encode syntactic roles, semantic fields, sentiment polarities, and factual associations — all simultaneously, in different orthogonal subspaces of the same $\mathbb{R}^d$.

Stage 3: Positional Encoding

Token embeddings encode what each token is, but they contain no information about where in the sequence each token appears. The sequence "dog bites man" and "man bites dog" would produce identical sets of token embeddings in a different order and would be catastrophic for a model that needs to understand syntax and word order.

Positional information must therefore be injected explicitly.

Sinusoidal Positional Encoding (Original Transformer)

The original transformer paper (Vaswani et al., 2017) proposed adding a fixed sinusoidal signal to each token embedding. For position $pos$pos and dimension $i$i:$$PE_{(pos, 2i)} = \sin\!\left(\frac{pos}{10000^{2i/d}}\right)$$ $$PE_{(pos, 2i+1)} = \cos\!\left(\frac{pos}{10000^{2i/d}}\right)$$

The input to the first transformer layer is then:$$\mathbf{x}_i = \mathbf{e}_i + PE_i$$

The sinusoidal formulation has a useful property: the positional encoding of position $pos + k$ can be expressed as a linear function of the encoding at $pos$pos, which makes it easy for attention heads to learn to attend to tokens at a fixed relative offset.

Learned Positional Embeddings

GPT-2 and GPT-3 replaced sinusoidal encodings with learned positional embeddings: a second matrix $W_P \in \mathbb{R}^{n_{\text{max}} \times d}$​, where $n_{\text{max}}$​ is the maximum context length. Position $pos$pos adds the row $W_P[pos]$ to the token embedding. These are learned end-to-end during pre-training and tend to slightly outperform sinusoidal encodings on benchmarks, at the cost of a hard context length ceiling — the model cannot generalise beyond $n_{\text{max}}$​ positions it has seen during training.

Rotary Positional Embedding (RoPE)

The current state of the art for most frontier models, including Llama, Mistral, GPT-4, and Gemini, is Rotary Positional Embedding (RoPE), introduced by Su et al. in 2021.

Rather than adding a positional vector to the token embedding, RoPE encodes position by rotating the Query and Key vectors in the attention mechanism (covered in Part 2) by a position-dependent angle. For a vector at position $m$m in dimension pair $(2i, 2i+1)$:$$\mathbf{q}_m^{(i)} = \begin{pmatrix} q_{2i} \\ q_{2i+1} \end{pmatrix} \begin{pmatrix} \cos m\theta_i & -\sin m\theta_i \\ \sin m\theta_i & \cos m\theta_i \end{pmatrix}$$

where $\theta_i = 10000^{-2i/d}$.

The critical advantage: the dot product between a query at position $m$m and a key at position $n$n depends only on their relative offset $m – n$, not their absolute positions. This gives the model a natural inductive bias toward relative position awareness, and with extensions like YaRN makes it possible to extend the effective context window well beyond what the model was trained on.

RoPE is why modern models can be fine-tuned to handle 128K or even 1M token context windows without retraining the entire model from scratch.

Putting It Together: The Input Representation

The final input to the first transformer layer for a sequence of $n$n tokens is a matrix $X \in \mathbb{R}^{n \times d}$:$$X = \begin{pmatrix} \mathbf{e}_1 + \mathbf{p}_1 \\ \mathbf{e}_2 + \mathbf{p}_2 \\ \vdots \\ \mathbf{e}_n + \mathbf{p}_n \end{pmatrix}$$where $\mathbf{p}_i$ is the positional encoding for position $i$ (sinusoidal, learned, or RoPE-derived). Each row of $X$ is a $d$-dimensional vector carrying both the semantic identity of the token and its position in the sequence. This matrix is what flows into the self-attention mechanism, which is the subject of Part 2.

---

Key Numbers to Anchor Your Intuition

Model	Vocabulary $\|V\|$	$d_{\text{model}}$	Positional scheme
GPT-2	50,257	768–1,600	Learned
GPT-3	50,257	12,288	Learned
Llama 3 8B	128,256	4,096	RoPE
Mistral 7B	32,000	4,096	RoPE
Gemma 2	256,000	2,304–3,584	RoPE

---

Conclusion

Before a single attention head fires, a large language model has already done substantial mathematical work. A raw string has been segmented into subword tokens by a BPE or SentencePiece algorithm, each token has been projected into a high-dimensional vector space where geometry encodes meaning, and position information has been woven in either additively or, in modern models, rotationally via RoPE. The quality of this input representation has an outsized effect on everything downstream: models with poorly constructed vocabularies or weak embedding initialisations are harder to train and less capable at convergence.

Part 2 will take this matrix $X \in \mathbb{R}^{n \times d}$ and walk through exactly what the self-attention mechanism does with it, including the full mathematical derivation of scaled dot-product attention, multi-head attention, and what different heads actually learn to represent.

Coming next in the AI Engineering series is Part 2: The Attention Mechanism.