How AI in Drug Discovery Is Powerfully Reshaping Protein Science and Molecular Design — Part 2: AlphaFold, Generative Models, and Virtual Screening

This is Part 2 of a three-part series on AI in drug discovery. Part 1 covered the molecular foundations: chemical space, molecular representations, graph neural networks, and transfer learning for QSAR modelling. Part 2 covers protein structure prediction, generative molecular design, and deep learning-powered virtual screening. Part 3 will examine clinical trial optimisation, multi-omics integration, and the open challenges facing the field.

From Representing Molecules to Understanding Targets

Part 1 established how machine learning models can learn to reason about small molecules: how chemical structures are encoded as SMILES strings, molecular graphs, or 3D conformers, and how graph neural networks trained on large molecular databases can predict biological activity from structure. But AI in drug discovery does not operate only on the drug molecule side of the equation. The biological target, almost always a protein, must also be understood at a level of detail that makes rational drug design possible. And until recently, that understanding was one of the most significant bottlenecks in the entire field.

Proteins are chains of amino acids that fold into precise three-dimensional structures, and those structures determine their function. A drug molecule must fit into a specific region of a protein, called a binding site, with the geometric and chemical complementarity of a key fitting a lock. Without knowing the three-dimensional structure of the target protein, rational drug design is severely constrained. Experimental structure determination using X-ray crystallography, cryo-electron microscopy, or NMR spectroscopy is slow, expensive, and frequently fails for difficult protein classes. AI in drug discovery has fundamentally changed this situation.

AlphaFold and the Protein Folding Revolution

The protein folding problem, predicting the three-dimensional structure of a protein from its amino acid sequence alone, was considered one of the hardest open problems in biology for over fifty years. The Critical Assessment of Protein Structure Prediction (CASP) competition, held every two years since 1994, benchmarks progress against experimentally determined structures. For most of its history, progress was incremental.

In December 2020, DeepMind’s AlphaFold 2 entered CASP14 and produced predictions of accuracy comparable to experimental methods for the majority of protein targets. The architecture of AlphaFold 2 is worth examining in technical detail, because it represents one of the most sophisticated applications of deep learning to a biological problem and its design choices are directly relevant to AI in drug discovery.

AlphaFold 2 takes two primary inputs: the amino acid sequence of the target protein, and a multiple sequence alignment (MSA) of evolutionarily related sequences from other organisms. The evolutionary information in the MSA is critical: positions in the sequence that have co-evolved (changed together across species) are likely to be physically close in the folded structure, because mutations in one position that would destabilise the structure are compensated by mutations in the other.

The architecture processes these inputs through two coupled networks. The first, the Evoformer, operates on a two-dimensional representation consisting of the MSA representation (a matrix of shape $N_{seq} \times N_{res}$​ where $N_{seq}$​ is the number of aligned sequences and $N_{res}$ is the protein length) and a pairwise representation encoding information about relationships between each pair of residue positions. The Evoformer applies 48 blocks of attention-based processing that update both representations iteratively, allowing information to flow between the sequence-level and pairwise-level representations through a mechanism called triangle multiplication, which enforces geometric consistency by updating the $(i,j)$ pairwise representation using information from the $(i,k)$ and $(k,j)$ pairs:$$z_{ij} \leftarrow z_{ij} + \sum_k z_{ik} \cdot z_{kj}$$​

This operation has a direct geometric interpretation: if residue $i$i is close to residue $k$, and residue $k$ is close to residue $j$, the model should update its estimate of the $i$–$j$ distance accordingly. The triangle multiplication embeds a soft version of the triangle inequality directly into the network architecture.

The second network, the Structure Module, takes the pairwise representation produced by the Evoformer and uses it to iteratively update a set of rigid body frames, one per residue, representing the orientation and position of each amino acid in three-dimensional space. The Structure Module uses Invariant Point Attention, an attention mechanism designed to operate on geometric frames in a way that is equivariant to global rotations and translations of the entire protein.

AlphaFold 3, released in 2024, extended the architecture to predict the structure of complexes containing proteins, DNA, RNA, ligands, and cofactors simultaneously, using a diffusion-based structure generation process rather than the iterative frame refinement of AlphaFold 2. For AI in drug discovery, AlphaFold 3 is directly applicable to predicting how a drug candidate will bind to its target, which is the central computational task in structure-based drug design.

Structure-Based Virtual Screening

With accurate protein structures available, AI in drug discovery can proceed to virtual screening: computationally evaluating large libraries of candidate molecules for their likely binding to a target protein, using only computation rather than physical synthesis and experimental testing.

Classical virtual screening used physics-based docking algorithms such as AutoDock Vina and Glide, which sample the conformational space of a ligand within the protein binding site and score each pose using empirical energy functions. These methods are interpretable and physically motivated but computationally expensive per molecule and limited in accuracy by the simplifications in the scoring function.

Deep learning-based virtual screening replaces or augments the scoring function with a neural network trained on experimental binding affinity data. Models including PointVS, GNINA, and DiffDock use the three-dimensional structures of protein-ligand complexes as input and learn to predict binding affinities or generate bound poses directly. DiffDock, developed at MIT, frames molecular docking as a generative diffusion process: rather than searching the conformational space by sampling, it learns a diffusion model over the space of ligand positions, orientations, and torsion angles conditioned on the protein structure, and generates docked poses by running the reverse diffusion process. This approach achieves state-of-the-art pose prediction accuracy while being orders of magnitude faster than traditional docking for large-scale screening.

For AI in drug discovery at industrial scale, the practical impact is significant. A single GPU can evaluate millions of candidate molecules against a target in hours using a trained deep learning scoring function, whereas physics-based docking at the same scale would require weeks of computation. This enables genuinely exhaustive screening of large commercially available compound libraries, and increasingly, of entirely virtual libraries of compounds that have never been synthesised.

Generative Molecular Design

The most ambitious application of AI in drug discovery is generative molecular design: using generative models to propose entirely new molecules with desired properties, rather than selecting the best candidates from a pre-existing library. This shifts the paradigm from search within known chemical space to exploration and invention of new chemical space.

Several generative architectures have been applied to molecular design. The choice of architecture depends on the molecular representation, the type of property being optimised, and whether the generation is conditioned on the target protein structure.

Variational Autoencoders (VAEs) for molecular generation encode molecules into a continuous latent space and decode samples from that space into molecular structures. The key property of the continuous latent space is that it enables gradient-based optimisation: given a differentiable property predictor, the gradient of the predicted property with respect to the latent vector can be computed and used to navigate the latent space toward molecules with improved properties. The JTVAE (Junction Tree VAE) architecture decomposes molecules into tree-structured arrangements of chemical substructures called junction trees, encoding them as hierarchical latent variables that respect chemical validity constraints during decoding.

Generative Adversarial Networks (GANs) for AI in drug discovery pit a generator network against a discriminator trained to distinguish generated molecules from real ones. The ORGAN model extends this framework with reinforcement learning to additionally optimise chemical property objectives. GANs for molecular generation face significant training instability challenges, because the discrete nature of molecular graphs makes it difficult to backpropagate gradients through the generation process.

Autoregressive models generate molecules sequentially, one atom or bond at a time, using a probability distribution over the next structural element conditioned on what has been generated so far. Large language models trained on SMILES strings operate in this regime: MolGPT and related models apply the GPT architecture directly to SMILES token sequences, learning the conditional distribution $P(t_i \mid t_1, \ldots, t_{i-1})$ over SMILES tokens and sampling new molecules by autoregressive generation. These models can be conditioned on desired properties by fine-tuning on property-annotated SMILES datasets or by using classifier-free guidance at generation time.

Diffusion models for molecular generation are currently achieving state-of-the-art results across multiple benchmarks. EDM (Equivariant Diffusion Model) operates directly in 3D space, learning to generate atom positions and types simultaneously by reversing a diffusion process that progressively adds Gaussian noise to molecular coordinates. The equivariance of the denoising network to 3D rotations and translations ensures that generated molecules are physically reasonable regardless of their global orientation. DiffSBDD extends this to structure-based drug design, conditioning the molecular generation on the three-dimensional structure of the protein binding site and directly generating molecules shaped to fill and interact with the target.

The theoretical framework of diffusion-based molecular generation is closely related to the score matching formulation. The model learns the score function $\nabla_\mathbf{x} \log p_t(\mathbf{x})$, the gradient of the log-probability density at noise level $t$t, and uses it to guide the reverse diffusion trajectory:$$d\mathbf{x} = \left[-\frac{1}{2}\beta(t)\mathbf{x} – \beta(t)\nabla_\mathbf{x} \log p_t(\mathbf{x})\right]dt + \sqrt{\beta(t)}\,d\mathbf{W}$$

where $\beta(t)$ is the noise schedule and $\mathbf{W}$ is a Wiener process. For molecular generation, $\mathbf{x}$ encodes atom positions and types, and the learned score function guides the system from a Gaussian noise distribution toward the distribution of real drug-like molecules, optionally conditioned on protein structure or target property values.

Reinforcement Learning for Property Optimisation

Beyond purely generative approaches, reinforcement learning (RL) has been widely applied to AI in drug discovery as a framework for optimising molecular properties iteratively. In the RL formulation, a policy network generates molecules by sequential construction (adding atoms and bonds), and a reward function evaluates the generated molecule according to desired properties such as predicted binding affinity, drug-likeness (quantified by the QED score), synthetic accessibility, and selectivity.

The REINVENT model, developed at AstraZeneca and subsequently released as open source, uses a prior language model over SMILES strings as a starting point and trains an agent model to maximise a composite reward function using the REINFORCE algorithm. The KL divergence between the agent and the prior is included in the training objective to prevent the agent from drifting into chemically unreasonable regions of SMILES space:$$\mathcal{J}(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\!\left[R(\tau)\right] – \beta \cdot D_{KL}\!\left(\pi_\theta \,\|\, \pi_{\text{prior}}\right)$$

This formulation is directly analogous to the RLHF objective discussed in the LLM training series on this blog, with the prior language model playing the role of the SFT model and the property predictor playing the role of the reward model. The parallel is not coincidental: the problem of generating molecules with desired properties and the problem of generating text with desired qualities share a common mathematical structure, which is one reason that advances in LLM training methodology have transferred productively into AI in drug discovery.

Multi-Target Optimisation and ADMET Prediction

A practical constraint on all generative approaches to AI in drug discovery is that generating molecules with high predicted binding affinity to a single target is necessary but not sufficient. The molecule must also satisfy ADMET constraints: Absorption, Distribution, Metabolism, Excretion, and Toxicity properties that determine whether a candidate will behave safely and effectively in the body.

Deep learning models for ADMET prediction use the same molecular representation frameworks covered in Part 1. The challenge is that ADMET properties depend on a complex interplay of structural features that are not always intuitively related to the features that drive target binding. A molecule that binds its target with nanomolar affinity may be rapidly metabolised by liver enzymes, unable to cross the blood-brain barrier, or toxic to cardiac ion channels.

Multi-task learning, training a single neural network to predict multiple ADMET endpoints simultaneously, has been shown to outperform single-task models for most individual endpoints, because the shared representation learned across tasks captures general features of molecular behaviour that are relevant to multiple properties simultaneously. The Chemprop architecture, one of the most widely used open-source tools for molecular property prediction in AI in drug discovery, supports multi-task training with uncertainty quantification using ensembling and evidence-based deep learning methods.

Conclusion

Part 2 of this series has traced the flow of AI in drug discovery from the protein target through to candidate molecule generation: AlphaFold’s equivariant transformer architecture for protein structure prediction, deep learning-based virtual screening for rapid evaluation of large compound libraries, diffusion models for structure-conditioned molecular generation, and reinforcement learning for iterative property optimisation. Together, these approaches represent a comprehensive AI in drug discovery toolkit that operates across the full problem of finding a molecule that is potent, selective, and physically viable.

Part 3 will extend the analysis to the clinical phases of drug development: how AI is being applied to patient stratification and clinical trial design, how multi-omics data integration is enabling systems-level understanding of disease biology, and what the honest open challenges are that prevent AI in drug discovery from fulfilling its full theoretical potential.

Part 3: Multi-Omics, Clinical Trial Optimisation, and Open Challenges — the final instalment of this series.