This is the final part of a five-part series on the internal mechanics of large language models. Parts 1–4 covered tokenization, embeddings, attention, the transformer block, and the training and alignment pipeline. Part 5 addresses inference: how a trained model generates text, the mathematics of sampling, context window management, emergent capabilities, and the open research frontiers shaping the next generation of LLMs. From Trained Weights to Generated Text A trained, aligned LLM is a fixed mathematical function: given a sequence of input tokens, it produces a probability distribution over the next token. Inference is the process of repeatedly applying this function to generate text, while the engineering decisions made at…

This is Part 4 of a five-part series on the internal mechanics of large language models. Part 3 covered the complete transformer block and architectural variants. Part 4 addresses how models are trained: the pre-training objective, loss functions, the Chinchilla scaling laws, and the alignment pipeline of instruction tuning and RLHF that transforms a raw language model into a useful assistant. The Pre-Training Objective The entire capability of a modern LLM emerges from one deceptively simple training objective: predict the next token. Given a sequence of tokens [t1,t2,…,tn][t_1, t_2, \ldots, t_n], the model is trained to maximise the log-probability of each token given all preceding tokens:LPT=−1n∑i=1nlog⁡P(ti∣t1,…,ti−1; θ)\mathcal{L}_{\text{PT}} = -\frac{1}{n} \sum_{i=1}^{n} \log…

This is Part 3 of a five-part series on the internal mechanics of large language models. Part 2 derived scaled dot-product attention and the KV cache. Part 3 traces the complete transformer block consisting of feed-forward networks, layer normalisation, and residual streams; it examines the architectural differences between decoder-only and encoder-decoder models. From Attention Output to Transformer Block At the end of Part 2, self-attention had produced an output matrix O∈Rn×dO \in \mathbb{R}^{n \times d} or one updated vector per token, now containing contextual information drawn from every other position in the sequence. But this is only the first half of a single transformer block. Before the signal moves to…

This is Part 2 of a five-part series on the internal mechanics of large language models. Part 1 covered tokenization, embeddings, and positional encoding. Part 2 derives the self-attention mechanism from first principles, covers multi-head attention, and examines the KV cache. The Problem Attention Solves At the end of Part 1, we had a matrix X∈Rn×dX \in \mathbb{R}^{n \times d}, which was a one ddd-dimensional vector per token, encoding both semantic identity and position. The first transformer layer receives this matrix and must do something critical: allow each token to incorporate information from every other token in the sequence before passing its updated representation to the feed-forward network. This is…

This is Part 1 of a five-part deep-dive series on the internal mechanics of large language models. This series is written for engineers, ML practitioners, and technically advanced readers. Part 1 covers tokenization and embeddings — the first two stages through which raw text becomes a mathematical object a neural network can reason over. Why Text Must Be Transformed Neural networks are, at their core, mathematical functions that operate on tensors, which are multi-dimensional arrays of floating-point numbers. They cannot accept raw text strings as input. Before a language model can do anything useful, it must convert a sequence of characters into a sequence of numbers. This two-step process of…