How Large Language Models Work — Part 5: Inference, Sampling, Emergent Behaviour, and Open Frontiers

This is the final part of a five-part series on the internal mechanics of large language models. Parts 1–4 covered tokenization, embeddings, attention, the transformer block, and the training and alignment pipeline. Part 5 addresses inference: how a trained model generates text, the mathematics of sampling, context window management, emergent capabilities, and the open research frontiers shaping the next generation of LLMs.

From Trained Weights to Generated Text

A trained, aligned LLM is a fixed mathematical function: given a sequence of input tokens, it produces a probability distribution over the next token. Inference is the process of repeatedly applying this function to generate text, while the engineering decisions made at inference time have a surprisingly large impact on the quality, diversity, safety, and cost of model outputs.

At each generation step $t$, the model computes:$$P(t_{n+1} \mid t_1, t_2, \ldots, t_n;\, \theta) = \text{softmax}(W_U \cdot \mathbf{h}_n^{(N)})$$

where $\mathbf{h}_n^{(N)} \in \mathbb{R}^d$ is the final-layer hidden state at position $n$n, $W_U \in \mathbb{R}^{|V| \times d}$ is the unembedding matrix (often tied to the transpose of the input embedding matrix $W_E$WE​), and the resulting vector of logits over $|V|$ vocabulary entries is passed through softmax to produce a probability distribution. A token is then drawn from this distribution (called the sampling step) appended to the context, and the process repeats.

The Mathematics of Sampling

How a token is selected from the output distribution is not trivial. Naive greedy decoding that always selects the highest-probability token tends to produce repetitive, degenerate outputs. Real systems use one or more of the following strategies.

Temperature scaling. Before applying softmax, the logits are divided by a temperature parameter $T > 0$T>0:$$P_T(t_{n+1} = k) = \frac{\exp(z_k / T)}{\sum_j \exp(z_j / T)}$$

As $T \to 0$, the distribution concentrates on the single highest-logit token, equivalent to greedy decoding. As $T \to \infty$, the distribution approaches uniformity with maximum entropy, maximum randomness. Values of $T \in [0.6, 1.0]$ are typical for creative generation; $T \in [0.0, 0.3]$ for precise factual or code generation tasks. Temperature does not change which token is most probable; instead, it changes how peaked or flat the distribution is.

Top-k sampling. The distribution is truncated to only the $k$ highest-probability tokens, which are then renormalised and sampled from. A typical value is $k = 40$. Top-$k$ prevents the model from sampling tokens with negligible probability, but it is sensitive to the shape of the distribution. In a uniform distribution, $k = 40$ keeps 40 of many reasonable options; in a very peaked distribution, $k = 40$ may retain tokens with near-zero probability.

**Nucleus (top-$p$) sampling.** Rather than fixing the number of tokens, nucleus sampling fixes the cumulative probability mass: the smallest set of tokens $\mathcal{V}_p$​ such that $\sum_{k \in \mathcal{V}_p} P(t_k) \geq p$ is retained and renormalised. A typical value is $p = 0.9$ or $p = 0.95$. Nucleus sampling adapts naturally to the distribution’s shape. A peaked distribution retains few tokens, a flat one retains many – this is now the dominant strategy in production systems.

Min-p sampling. A newer variant, increasingly adopted in open-source inference stacks, sets a dynamic minimum probability threshold relative to the top token: any token with $P(t_k) < p_{\min} \times P(t_{\max})$ is discarded. This keeps tokens that are meaningfully competitive with the best option while discarding genuinely improbable ones, and empirically reduces repetition more effectively than top-$p$p at matched quality.

In practice, production inference systems combine multiple strategies: temperature scaling applied first to reshape the distribution, followed by top-$p$ or min-$p$ truncation, followed by top-$k$ as a hard ceiling. The specific combination and hyperparameters are treated as tunable properties of a deployment configuration, not fixed properties of the model itself.

Context Windows and Memory Management

Every inference call operates within a context window, which is the maximum sequence length the model can process in a single forward pass. For GPT-4, the context window is 128K tokens; for Gemini 1.5 Pro, it extended to 1 million tokens; Llama 3.1 supports 128K. Context window length is not a free parameter, rather, it is constrained by the quadratic $O(n^2)$ attention cost and, critically, by KV cache memory.

As established in Part 2, the KV cache stores the key and value tensors for every previous token at every layer, avoiding redundant recomputation during autoregressive generation. The memory footprint is:$$\text{KV cache memory} = 2 \times n \times N_{\text{layers}} \times n_{\text{heads}} \times d_k \times \text{bytes per element}$$

For a model with 96 layers, 96 heads, $d_k = 128$, processing a 128K-token context in FP16: $2 \times 128000 \times 96 \times 96 \times 128 \times 2 \approx 601$ GB. This exceeds the VRAM of any single current GPU by a factor of roughly 10, which is why long-context inference requires either model parallelism across multiple devices, quantisation of KV cache entries to lower precision (INT8 or INT4), or approximate attention methods that reduce what must be stored.

Several efficient attention variants address this scaling problem. Flash Attention (Dao et al., 2022) reorders the attention computation to avoid materialising the full $n \times n$ attention matrix in high-bandwidth memory, reducing memory complexity from $O(n^2)$ to $O(n)$ while maintaining identical numerical output. Grouped-query attention (GQA), used in Llama 2 and 3, shares key and value heads across multiple query heads, reducing KV cache size by a factor of $g$ (the group size) without significant quality degradation. Sliding window attention, used in Mistral, restricts each token to attend only within a fixed window of recent tokens, allowing arbitrarily long sequences at the cost of cross-window recall.

Emergent Capabilities and Phase Transitions

One of the most scientifically consequential, and practically important, observations in LLM scaling is the phenomenon of emergent capabilities: abilities that are essentially absent in smaller models and appear abruptly as scale crosses some threshold, without being explicitly trained for.

Wei et al. (2022) documented dozens of such capabilities: few-shot chain-of-thought reasoning, multi-step arithmetic, word unscrambling, and logical deduction all exhibit sharp phase transitions: near-zero performance at sub-threshold scales, near-human performance above it. The sharpness of these transitions distinguishes emergence from smooth capability growth and has profound implications for predicting what a model trained at a given scale will be able to do.

The mechanism underlying emergence is debated. One influential account (the “grokking” hypothesis, from Power et al. at Anthropic) suggests that models first memorise training examples, then undergo a phase transition in which they discover a compressed algorithmic solution that generalises to held-out data. Another account, from Anthropic’s interpretability research, suggests that emergent capabilities correspond to the model assembling multi-step circuits, consisting of sequences of attention heads and FFN layers that compose to implement a non-trivial algorithm, and that these circuits can only form once the model has sufficient depth and width to represent all required intermediate computations simultaneously.

From an engineering standpoint, emergence creates a prediction problem. The GPT-3 scaling curve gave no warning that a GPT-4-scale model would exhibit chain-of-thought reasoning, code execution, or structured tool use. This motivates careful capability elicitation and evaluation at each new model generation —(the practice of red-teaming) as a safety and capability discovery discipline.

Chain-of-Thought and Reasoning Models

The most practically significant emergent capability is multi-step reasoning — and the discovery that it can be dramatically improved by prompting or training the model to externalise its intermediate steps.

Chain-of-thought prompting (Wei et al., 2022) established that providing examples of step-by-step reasoning in the prompt elicits dramatically better performance on multi-step arithmetic, commonsense reasoning, and symbolic manipulation tasks without any additional training. The effect is robust and scales with model size: smaller models gain little from chain-of-thought, but models above approximately 100B parameters show large, consistent improvements.

Reasoning models — exemplified by OpenAI’s o1 and o3, and Anthropic’s Claude’s extended thinking mode — take this further by training models to produce extended internal reasoning traces before generating a final answer. The training procedure uses reinforcement learning on verifiable outcomes: the model is rewarded for producing correct final answers to problems where correctness can be checked algorithmically (mathematics, code execution, formal logic), and the RL signal drives the model to develop longer, more structured reasoning traces that are instrumentally useful for producing correct outputs.

The result is a qualitatively different inference regime. A standard GPT-style model generates tokens in a single forward pass per token, with effective reasoning depth bounded by model depth. A reasoning model generates a long internal monologue, often thousands of tokens of scratchpad, before committing to an answer, effectively trading inference-time compute for reasoning quality. This represents a fundamental shift in the compute profile of AI: capability is no longer a fixed function of model size, but can be increased at inference time by allocating more tokens to reasoning.

Open Frontiers: Mechanistic Interpretability, Multimodality, and Agents

Three research frontiers are most likely to define the next phase of LLM development.

Mechanistic interpretability is the program of reverse-engineering what computations specific model components implement. The goal is to go beyond behavioural evaluation — what the model does — to structural understanding — what algorithm it uses to do it. Anthropic’s superposition hypothesis (Elhage et al., 2022) established that neural networks represent more features than they have dimensions by encoding features as directions in a high-dimensional space and tolerating controlled interference between them. The subsequent Sparse Autoencoder (SAE) program — also led at Anthropic — has made it possible to decompose model activations into sparse, interpretable features: human-understandable concepts that each activate for a specific, semantically coherent set of inputs. SAE-based interpretability tools are now being applied to frontier models at production scale, with the goal of identifying circuits responsible for deception, sycophancy, and unsafe behaviours before deployment.

Multimodality extends the transformer architecture beyond text. Vision-language models (GPT-4V, Gemini, Claude 3) process images by encoding them through a vision encoder — typically a Vision Transformer (ViT) — into a sequence of patch embeddings that are projected into the LLM’s token embedding space and concatenated with text tokens before the first transformer layer. Audio-language models (GPT-4o’s audio mode) follow an analogous pattern with a spectrogram encoder. The frontier is native multimodality — a single model trained end-to-end on text, image, audio, and video tokens simultaneously, rather than modality-specific encoders bolted onto a text backbone.

Agentic systems extend the inference loop beyond token generation to include tool use, memory retrieval, and sequential decision-making. An agent wraps an LLM in a loop: the model generates text that includes structured tool calls (web search, code execution, database queries); the tool results are appended to the context; the model continues generating, possibly issuing further tool calls, until it produces a final response. Multi-agent systems extend this further: multiple LLM instances, each with different system prompts and tool access, communicate with each other through structured message passing. The engineering challenges are significant — context management, tool reliability, error recovery, and safety alignment in agentic settings are all active areas of research — but the capability ceiling of agentic LLMs is substantially higher than single-turn generation.

A Map of the Whole

Across this five-part series, we have traced the complete path from raw string to generated output:

1. Tokenization — BPE or SentencePiece converts text into integer token IDs from a vocabulary of 32K–256K entries.
2. Embedding — each token ID is projected into a $d$d-dimensional vector; positional information is added via learned embeddings, sinusoidal encoding, or RoPE.
3. Attention — $N$N transformer blocks each apply multi-head self-attention: Q, K, V projections; scaled dot products; causal masking; softmax; value aggregation.
4. Feed-forward network — each block applies a position-wise two-layer MLP with SwiGLU activation, implementing key-value memory over learned factual associations.
5. Training — next-token prediction loss over trillions of tokens; AdamW with cosine decay; Chinchilla-optimal compute allocation; distributed parallelism at scale.
6. Alignment — SFT on human demonstrations, reward model training on preference rankings, RLHF with PPO or DPO to concentrate capability toward preferred outputs.
7. Inference — temperature scaling, nucleus or min-$p$p sampling, KV cache management, and extended chain-of-thought reasoning to generate high-quality outputs efficiently.

The transformer is a simple, composable architectural primitive. What it does, when trained at sufficient scale on sufficient data, and aligned to human preferences, is not simple at all.

This concludes the five-part series on how large language models work. If you have found this series useful, the natural next steps are the mechanistic interpretability literature (Anthropic’s Transformer Circuits thread), the Chinchilla and scaling laws papers, and Neel Nanda’s open-source interpretability tooling at TransformerLens.