LearnerBox logo LearnerBox Infosystems LLP
  • A RAG pipeline from scratch, showing the three main stages: chunking documents, embedding chunks into vector space, and data retrieval for a frontier large language model.
    AI Engineering

    Complete RAG Pipelines from Scratch in 4 Steps: Chunking, Embedding, and Retrieval

    The Problem RAG Solves

    Do you want to learn how to build a RAG Pipeline from Scratch? Every large language model has a knowledge cutoff. It knows what it was trained on, and nothing beyond that. Ask a frontier model about a document it has never seen, a database record updated this morning, or a policy that changed last week, and it will either hallucinate an answer or tell you it does not know. For the vast majority of real enterprise AI applications, this is a fundamental limitation.

    Retrieval-Augmented Generation (RAG) solves it. Rather than relying solely on what is baked into the model’s weights, RAG retrieves relevant content from an external knowledge source at query time and injects it into the model’s context window before generation. The model now has access to specific, current, verifiable information when it generates its answer. The result is a system that combines the reasoning capability of a frontier model with the factual grounding of a real knowledge base.

    Building a RAG pipeline from scratch requires understanding three distinct stages: chunking the source documents into retrievable units, embedding those chunks into a vector space, and retrieving the right chunks at query time. Each stage has engineering decisions that significantly affect the quality of the final system.

    Stage 1: Chunking

    Chunking is the process of splitting source documents into smaller pieces that can be individually embedded and retrieved. It sounds simple, but it is where many RAG pipelines fail silently.

    The core tension in chunking is between specificity and context. Small chunks are precise: a retrieval system can return exactly the passage relevant to a query without padding. But small chunks lose surrounding context, which means the model may receive a fragment that is accurate but uninterpretable without the sentences around it. Large chunks preserve context but dilute relevance: the retrieved passage contains the answer plus a lot of surrounding material that consumes precious context window tokens without contributing to the response.

    Fixed-size chunking splits documents by token or character count, typically 256 to 512 tokens per chunk, with an overlap of 10 to 20 percent between adjacent chunks. The overlap ensures that sentences spanning a chunk boundary are represented fully in at least one chunk. This approach is fast, predictable, and easy to implement:

    def fixed_size_chunks(text: str,
                          chunk_size: int = 512,
                          overlap: int = 64) -> list[str]:
        tokens = tokenizer.encode(text)
        chunks = []
        start = 0
        while start < len(tokens):
            end = min(start + chunk_size, len(tokens))
            chunks.append(tokenizer.decode(tokens[start:end]))
            start += chunk_size - overlap
        return chunks

    Semantic chunking splits on natural boundaries: paragraphs, sections, or sentences, then merges adjacent units until a target size is reached. This preserves the logical structure of the document far better than fixed-size splitting and is the preferred approach for documents with clear section structure such as contracts, technical documentation, and research papers.

    Hierarchical chunking maintains both a summary chunk and detailed sub-chunks for each section. At retrieval time, the system first retrieves summary-level chunks to identify the right section, then drills down to the detailed chunks within it. This approach produces significantly better results on long documents but requires more complex indexing infrastructure.

    A practical rule: for structured documents such as PDFs, legal text, and technical manuals, use semantic chunking. For conversational logs, emails, and unstructured text, fixed-size chunking with generous overlap is typically sufficient. Always add metadata to each chunk: the source document name, page number, section heading, and creation date. This metadata is invaluable for filtering at retrieval time and for attributing sources in the final response.

    Stage 2: Embedding

    Once documents are chunked, each chunk must be converted into a dense vector that captures its semantic meaning. This is done using an embedding model: a neural network trained to map text into a high-dimensional space where semantically similar passages are geometrically close to each other.

    The choice of embedding model matters significantly. The dominant options in 2026 are OpenAI’s text-embedding-3-large (3,072 dimensions), Cohere’s embed-v3, and open-source alternatives such as bge-large-en-v1.5 from BAAI and e5-mistral-7b-instruct from Microsoft. Open-source models have the advantage of running locally, which matters for data-sensitive applications.

    Embedding a corpus in Python using the OpenAI API looks like this:

    from openai import OpenAI
    import numpy as np
    
    client = OpenAI()
    
    def embed_chunks(chunks: list[str],
                     model: str = "text-embedding-3-large") -> np.ndarray:
        response = client.embeddings.create(
            input=chunks,
            model=model
        )
        return np.array([item.embedding for item in response.data])

    Two practical considerations deserve attention here. First, embed your queries using the same model you used to embed your documents. Mixing embedding models produces meaningless similarity scores because the vector spaces are incompatible. Second, normalise your embedding vectors before storing them if you plan to use cosine similarity for retrieval: most vector databases do this automatically, but it is worth verifying.

    For domain-specific applications, fine-tuning an embedding model on in-domain data consistently outperforms general-purpose embeddings. A legal RAG system fine-tuned on case law retrieves significantly more relevant passages than a general embedding model applied to the same corpus, because the fine-tuned model learns the specific vocabulary and semantic relationships of the domain.

    Stage 3: Vector Storage

    Embeddings must be stored in a system that supports efficient similarity search. The options divide into three categories.

    Dedicated vector databases such as Pinecone, Weaviate, Qdrant, and Milvus are purpose-built for high-dimensional vector search. They support filtering on metadata, approximate nearest-neighbour (ANN) search algorithms such as HNSW (Hierarchical Navigable Small World), and horizontal scaling to billions of vectors. For production systems with large corpora, these are the right choice.

    Hybrid stores such as pgvector (a PostgreSQL extension) allow vector search within an existing relational database. This is useful when you want to combine semantic search with SQL filtering in a single query, which is a common pattern for structured data sources.

    In-memory search using libraries such as FAISS or Annoy is suitable for development, prototyping, and small corpora where latency is critical and persistence is not required.

    import faiss
    import numpy as np
    
    def build_faiss_index(embeddings: np.ndarray) -> faiss.IndexFlatIP:
        dimension = embeddings.shape[1]
        index = faiss.IndexFlatIP(dimension)  # inner product = cosine on normalised vectors
        faiss.normalize_L2(embeddings)
        index.add(embeddings)
        return index

    Stage 4: Retrieval

    At query time, the user’s question is embedded using the same model used to embed the documents, and the vector store is queried for the most similar chunks. This is where several common RAG pipeline failures occur, and where the most productive engineering improvements can be made.

    Naive top-k retrieval returns the k most similar chunks by cosine similarity. It is a good starting point but has two well-known weaknesses. First, it relies entirely on the query embedding capturing the user’s intent accurately, which fails when queries are vague or use different terminology from the source documents. Second, the top-k chunks may all be from the same section of the same document, producing redundant context rather than diverse coverage.

    Hybrid retrieval combines semantic (vector) search with keyword (BM25) search and merges the results using Reciprocal Rank Fusion (RRF). This is the most reliably effective retrieval strategy for general-purpose RAG and is now the default approach in most production systems. Keyword search catches exact term matches that semantic search misses; semantic search catches paraphrases and conceptual matches that keyword search misses.

    Query rewriting passes the user’s query through the LLM before retrieval, expanding it into multiple alternative phrasings or decomposing a complex question into simpler sub-queries. Each sub-query is retrieved separately, and the results are merged before generation. This significantly improves retrieval quality on complex, multi-part questions.

    Re-ranking adds a second-pass model after initial retrieval. A cross-encoder model such as Cohere Rerank or a locally hosted BGE re-ranker scores each retrieved chunk against the original query more accurately than the bi-encoder embedding similarity used in the initial retrieval, and re-orders the results accordingly. Re-ranking consistently improves answer quality at the cost of an additional model call.

    Putting It Together

    A minimal but complete RAG pipeline in production should: chunk documents semantically with metadata, embed with a domain-appropriate model, store in a vector database with metadata filtering, retrieve using hybrid search with re-ranking, and inject the retrieved context into a well-structured prompt with source attribution. Each of these steps is independently tunable, which means RAG pipeline quality can be improved incrementally without retraining the underlying LLM.

    The engineering value of RAG is precisely this modularity. The knowledge base can be updated without touching the model. The retrieval strategy can be improved without reindexing. The generation model can be swapped without rebuilding the index. That separation of concerns is what makes RAG the most widely deployed pattern in production AI engineering today.

  • AI Engineering

    MCP vs API: Why Traditional APIs Are Failing AI Agents

    The Integration Problem Nobody Anticipated

    When developers began building the first generation of LLM-powered applications in 2023, the obvious approach was to reach for the tools already in the toolbox. REST APIs had connected software systems for two decades. They were well-understood, well-documented, and supported by mature tooling. The assumption was that connecting an AI agent to a database, a calendar, or a CRM would work just like connecting any other piece of software to those systems.

    That assumption turned out to be wrong in ways that were not immediately obvious. Industry reports and Microsoft AI Red Team Research from 2025 show that agentic systems failed due to brittle tool integrations, ambiguous context handling, and poorly defined interfaces between models and the external world. The failures were not usually spectacular crashes. More often they were silent: agents producing subtly incorrect behaviour that took days to trace back to a broken integration. Understanding why requires looking at what REST APIs were actually designed for, and why that design is a poor match for how AI agents operate.

    mcpapi

    What REST APIs Were Built to Do

    A REST API is a contract between a developer and a service. The developer writes code that calls a specific endpoint with specific parameters in a specific format, and the service returns a predictable response. The entire model is built around a human programmer who knows in advance what action needs to be taken, which endpoint handles it, and what the response structure means.

    APIs let developers write deterministic code that calls specific endpoints. The distinction reshapes integration architecture for every team deploying AI in production. This is precisely the property that makes APIs unsuitable for agentic systems. An AI agent does not know in advance what actions it will need to take. It discovers the appropriate actions at runtime, based on its reasoning about the current state of a task. Asking an LLM to navigate a traditional REST API is like handing a new employee a 400-page API specification document and asking them to memorise it before making any decisions.

    The second problem is statefulness. REST APIs use stateless HTTP. Each request carries its own authentication, parameters, and context. The server processes the request and forgets the caller. Stateless communication is excellent for web applications where millions of independent clients send independent requests. It is a poor fit for an agent executing a multi-step task over minutes or hours, where context from earlier steps needs to be maintained and referenced throughout.

    The third problem is the integration explosion. Without a standardized protocol, each AI application must integrate directly with every external service, creating N times M separate integrations where N represents the number of tools and M represents the number of clients. This approach quickly becomes impossible to scale. An enterprise deploying five agents across ten internal tools would need fifty bespoke integrations, each hand-coded, each requiring its own maintenance, and each breaking independently when either the agent or the tool changes.

    Enter MCP: The USB-C Port for AI

    The Model Context Protocol (MCP) was introduced by Anthropic on November 25, 2024, and donated to the Linux Foundation’s Agentic AI Foundation (AAIF) in December 2025, co-governed by Anthropic, OpenAI, and Block as a vendor-neutral open standard. Think of MCP as a USB-C port for AI systems: just as USB-C standardises how devices connect to computers, MCP standardises how AI agents access external resources like databases, APIs, file systems, and knowledge bases.

    The architectural difference from REST is fundamental. Rather than hardcoded connections to each external service, AI agents using MCP can dynamically discover available tools, understand their capabilities through structured calls, and invoke them with proper permissions. Instead of requiring the agent to know the endpoint, the parameter schema, and the response format for every possible tool call in advance, an MCP server exposes a machine-readable capability surface that the agent can query at runtime. The agent asks “what can you do?” before deciding what to do, which maps far more naturally onto how LLM reasoning actually works.

    The session model is equally important. MCP maintains stateful JSON-RPC 2.0 sessions, whereas REST APIs are stateless request-response. A stateful session means the agent and the tool can maintain shared context across the entire duration of a multi-step task, with the server able to push progress updates and partial results directly into the agent’s reasoning loop rather than waiting for the agent to poll.

    The N times M integration problem is solved structurally. MCP solves this by requiring each client and each server to implement the protocol just once, reducing total integrations from N times M to N plus M. Build one MCP server for your database, and every MCP-compatible agent can use it immediately, with no additional integration work on either side.

    Tools, Not Endpoints: A Critical Distinction

    One of the most important conceptual shifts MCP introduces is the distinction between a tool and an API endpoint. These sound similar but are architecturally different. Tools are not designed to be an abstraction over API calls but rather an abstraction over functionality. A tool may include multiple API calls in its implementation to achieve the desired outcome.

    This distinction matters because it aligns with how agents reason. An agent does not want to know which HTTP endpoint to call. It wants to know what it can accomplish. A tool called book_flight that internally makes three API calls to a pricing service, an availability checker, and a booking system is far more useful to an agent than three separate REST endpoints that the agent must learn to orchestrate itself. The tool encapsulates the implementation; the agent sees only the capability.

    An agent will review the list of available tools to automatically select the most appropriate tools and determine the appropriate order of execution. This is exactly the kind of dynamic, context-driven decision-making that REST APIs, designed for deterministic developer-written code, cannot support natively.

    Industry Adoption: The Tipping Point Has Passed

    The signal that MCP had won the integration standard debate came in March 2025, when OpenAI officially adopted it. For years, OpenAI had cultivated its own walled garden via the Assistants API. However, the friction of maintaining proprietary integrations against a rapidly expanding open ecosystem became untenable. OpenAI’s adoption was accompanied by the announcement of the deprecation of the Assistants API, scheduled for sunset in mid-2026, compelling the entire developer ecosystem to migrate toward MCP-based architectures. Google followed with its own MCP support shortly after.

    The growth of the ecosystem since then has been rapid. As of February 2026, the official MCP registry has over 6,400 MCP servers already registered. The November 2025 MCP specification update added critical enterprise capabilities: asynchronous operations so agents can initiate long-running tasks and retrieve results later, formal server identity verification, and structured audit trails. These additions directly addressed the governance concerns that slowed enterprise adoption through 2025.

    Salesforce reported 4.5 million MCP calls processed through its Headless 360 platform within weeks of launch. The MCP Dev Summit North America in April 2026 drew approximately 1,200 attendees. The protocol is no longer experimental infrastructure; it is production reality at scale.

    MCP Does Not Replace APIs. It Wraps Them.

    A common misconception is that MCP makes REST APIs obsolete. The more accurate picture is that MCP does not replace APIs. It wraps them into a standardised layer that LLMs can navigate, turning the N times M integration problem into N plus M. The underlying services still expose REST endpoints. MCP sits in front of them as an intelligent, agent-friendly abstraction layer. Atlan

    The practical decision rule is straightforward: use traditional APIs when a human developer is writing deterministic application code that calls a known endpoint. Use MCP when an AI agent needs to discover and invoke tools dynamically at runtime across multiple systems. For teams running three or more AI-connected integrations, the complexity crossover point where MCP reduces total integration cost is typically reached quickly.

    What This Means for AI Engineers

    For practitioners building agent systems today, MCP is no longer optional infrastructure to consider for future projects. It is the current standard. Every new data source requiring its own custom implementation makes truly connected systems difficult to scale. MCP addresses this challenge by providing a universal, open standard for connecting AI systems with data sources, replacing fragmented integrations with a single protocol. Anthropic

    The engineering implication is direct: if you are building an AI agent that needs to connect to more than one external system, build or adopt MCP servers rather than hand-coding REST integrations. The ecosystem already contains over 6,400 servers covering databases, file systems, version control, CRMs, calendars, and hundreds of SaaS platforms. The connective tissue for the agentic web has been standardised. The remaining work is building the agents capable of using it well.

  • AI Engineering

    Building an AI Agent from Scratch: Tools, Memory, and Reasoning Loops

    What Makes Something an Agent?

    There is a meaningful difference between calling an LLM API and building an AI agent. A single API call takes an input, produces an output, and stops. An agent does something more: it perceives a situation, decides what action to take, executes that action, observes the result, and decides what to do next. That loop, repeated until the task is complete, is what makes something an agent rather than a wrapper.

    The concept has deep roots in AI research, but the practical engineering of LLM-based agents has matured enormously in the past two years. Today, a competent Python developer can build a functional agent in an afternoon. Understanding what the agent is actually doing under the surface, and building it in a way that is reliable, observable, and safe, takes considerably more thought. This post walks through the three core components of any agent system: tools, memory, and the reasoning loop.

    The Reasoning Loop: Think, Act, Observe, Repeat

    The architectural heart of an LLM agent is the reasoning loop. The most widely used formulation is ReAct (Reasoning and Acting), introduced in a 2022 paper by Yao et al. at Princeton and Google Brain. The loop works as follows:

    1. The agent receives a task.
    2. It reasons about what to do next (Thought).
    3. It selects and calls a tool (Action).
    4. It receives the tool’s output (Observation).
    5. It reasons again, incorporating the observation.
    6. It repeats until it decides the task is complete and returns a final answer.

    In code, this translates to a loop that sends the current state of the conversation to the LLM, parses its response for a tool call, executes the tool, appends the result to the conversation history, and calls the LLM again. A minimal Python implementation looks like this:

    def run_agent(task: str, tools: dict, max_steps: int = 10) -> str:
        messages = [
            {"role": "system", "content": build_system_prompt(tools)},
            {"role": "user",   "content": task}
        ]
    
        for step in range(max_steps):
            response = call_llm(messages)
            action = parse_action(response)
    
            if action["type"] == "final_answer":
                return action["content"]
    
            observation = tools[action["name"]](**action["args"])
    
            messages.append({"role": "assistant", "content": response})
            messages.append({"role": "user",
                             "content": f"Observation: {observation}"})
    
        return "Max steps reached without a final answer."

    Several things in this skeleton are worth noting. The max_steps guard is not optional: without it, a confused or looping agent will burn tokens indefinitely. The system prompt must describe the available tools clearly, including their names, what they do, and the exact format the model should use to call them. And parse_action needs to be robust: LLMs do not always produce perfectly formatted output, so defensive parsing with fallback handling is essential in production.

    Tools: Giving the Agent Hands

    A tool is any function the agent can call to interact with the world outside the LLM’s context window. Common tools in production agents include web search, code execution, file reading and writing, database queries, REST API calls, calculator functions, and retrieval from a vector store. The principle is simple: if the agent needs information or capabilities that are not already in its context, it needs a tool to get them.

    Defining tools well is one of the most important engineering decisions in agent design. Each tool should do one thing clearly, return results in a consistent and parseable format, handle errors gracefully rather than crashing the loop, and be as fast as possible since every tool call adds latency. A poorly designed tool that returns noisy or ambiguous output will confuse the model, produce bad reasoning, and waste steps.

    In the OpenAI API, tools are defined as JSON schemas that the model uses to structure its calls:

    tools = [
        {
            "type": "function",
            "function": {
                "name": "web_search",
                "description": "Search the web for current information.",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "query": {
                            "type": "string",
                            "description": "The search query."
                        }
                    },
                    "required": ["query"]
                }
            }
        }
    ]

    The Anthropic API uses an equivalent structure. The key discipline is writing the description field carefully: the model reads it to decide when and how to use the tool, so vague descriptions produce inconsistent tool selection.

    Memory: What the Agent Knows and Remembers

    Memory in AI agents divides into four types, each serving a different purpose.

    In-context memory is the simplest: everything in the current conversation history that the model can see. It is immediate and requires no infrastructure, but it is bounded by the context window. For a task that takes many steps or processes large documents, in-context memory alone is insufficient.

    External memory uses a vector database such as Pinecone, Weaviate, or ChromaDB to store and retrieve information by semantic similarity. When the agent needs to recall something from a long earlier conversation, a previous session, or a large document corpus, it queries the vector store and injects the relevant results into the context. This is the retrieval-augmented memory pattern, and it is the most common approach for agents that need persistent knowledge.

    Episodic memory stores summaries of past interactions, allowing the agent to recall what it did in previous sessions without storing every message verbatim. A lightweight implementation writes a structured summary to a database at the end of each session and retrieves relevant episodes at the start of the next one.

    Working memory is an explicit scratchpad the agent maintains during a multi-step task: a structured record of what it has done, what it has found, and what it still needs to do. Externalising this into a structured object rather than relying purely on the conversation history significantly improves performance on complex tasks, because it prevents the model from losing track of earlier steps as the context grows.

    The System Prompt: The Agent’s Constitution

    The system prompt is the most underestimated component of agent design. It defines the agent’s identity, its available tools, the format it must use to call them, its reasoning style, its stopping conditions, and its constraints. A well-written system prompt for an agent typically includes a role definition, a tool registry with descriptions and invocation syntax, an explicit instruction to reason before acting, a rule specifying when to stop, and a safety boundary defining what the agent must not do.

    A common mistake is writing a minimal system prompt and expecting the model to infer the rest. In an agentic loop, ambiguity in the system prompt compounds across steps: a model that is slightly uncertain about when to call a tool versus reason further will make inconsistent decisions, produce unpredictable behaviour, and be difficult to debug.

    Observability and Safety: What Most Tutorials Skip

    Two concerns that most introductory agent tutorials omit are observability and safety, and both matter enormously in production.

    Observability means logging every step of the reasoning loop: what the model decided, which tool it called, what arguments it passed, and what the tool returned. Without structured logs, debugging a failed agent run is nearly impossible. Tools such as LangSmith, Weights and Biases, and Langfuse provide agent tracing infrastructure that makes this practical.

    Safety in agentic systems means imposing hard limits on what the agent can do autonomously. Any action that is irreversible, such as sending an email, deleting a record, or executing a financial transaction, should require a human confirmation step before execution. This is not a limitation on capability; it is a precondition for deploying agents in environments where mistakes have real consequences.

    Conclusion

    An AI agent is a reasoning loop wrapped around an LLM, extended with tools that give it reach and memory systems that give it persistence. Building one from scratch, rather than dropping a framework in place, forces you to understand exactly what is happening at each step, which is the foundation of being able to debug, extend, and trust what you deploy. Start with a simple loop, a handful of clearly defined tools, and in-context memory. Add external memory and episodic summarisation as the complexity of your tasks demands it. Log everything. And never let an agent take an irreversible action without a human in the loop.

  • AI Engineering

    So You Want to Be an AI Engineer: A Subject-by-Subject Study Guide

    A Career That Did Not Exist a Decade Ago

    AI engineering is one of the fastest-growing roles in technology, with job openings increasing by 143% year on year in early 2026. Entry-level roles offer strong compensation, and the field spans virtually every industry, from healthcare and finance to education, manufacturing, and government. Yet the path into AI engineering is still poorly signposted. Many people who want to move into the field are unsure which subjects to prioritise, how deep to go, and where to start.

    This guide cuts through that confusion. It is structured around the subjects and areas of knowledge that actually matter for AI engineering work in 2026: not a list of tools and buzzwords, but the underlying disciplines that give you durable capability regardless of which frameworks rise or fall in the years ahead.

    Programming: The Non-Negotiable Starting Point

    You should learn to code properly before moving on to anything AI-related. Python is a good choice because almost every AI library, framework, and tool is built for it first. The fundamentals to master include variables, functions, loops, data structures such as lists and dictionaries, object-oriented programming with classes and methods, file handling, and error management. This foundation typically takes two to three months of daily practice for complete beginners.

    Beyond Python basics, you will need familiarity with version control via Git and GitHub, working in the terminal, writing clean and testable code, and understanding how to structure a software project. AI engineering is software engineering first; the AI parts sit on top of a solid programming foundation.

    Mathematics: Three Areas That Underpin Everything

    You do not need a mathematics PhD to build AI applications. You do need a working understanding of three areas.

    Linear algebra is the mathematical language of neural networks. Matrices, vectors, dot products, matrix multiplication, eigenvalues, and transformations are the operations that underlie every forward pass through a model. You do not need to derive these from first principles, but you do need to know what they mean and how they behave.

    Calculus and optimisation powers the training of every model you will ever use. Derivatives, the chain rule, gradients, and gradient descent are the mechanisms through which a model learns. A working understanding of why gradient descent converges, and what goes wrong when it does not, is essential for debugging training runs and configuring hyperparameters sensibly.

    Probability and statistics governs how models handle uncertainty, make predictions, and are evaluated. Probability distributions, expectation, variance, Bayes’ theorem, hypothesis testing, and concepts like precision, recall, and AUC are the vocabulary of model evaluation. Understanding statistics is also what separates someone who can read a research paper from someone who cannot.

    Machine Learning Fundamentals

    Before working with large language models and APIs, you need a solid grounding in how machine learning systems actually learn. This means supervised learning (regression, classification), unsupervised learning (clustering, dimensionality reduction), overfitting and regularisation, cross-validation, and evaluation metrics. Understanding these concepts at a practical level gives you the mental model to diagnose model behaviour, choose the right approach for a given problem, and understand what LLMs are actually doing under the surface.

    Successful AI engineers must write clean, efficient Python code, understand how machine learning and deep learning frameworks work in practice, and know how to prepare and handle data well. The two dominant deep learning frameworks are PyTorch and TensorFlow. PyTorch has become the preference for research and most production LLM work; TensorFlow remains widely used in enterprise deployment pipelines.

    Large Language Models and the Modern AI Stack

    This is where AI engineering in 2026 diverges most clearly from traditional machine learning engineering. AI engineers build chatbots, retrieval-augmented generation (RAG) pipelines, autonomous agents, and intelligent workflows that solve real problems. The subjects to study here include how transformer models work (covered in depth in our five-part LLM series on this blog), prompt engineering, the OpenAI and Anthropic APIs, LangChain and LlamaIndex for building LLM applications, vector databases such as Pinecone and Weaviate for semantic retrieval, and RAG architecture for grounding model outputs in specific knowledge bases.

    This layer is evolving quickly, so the most important skill is learning how to learn: reading documentation, following model release notes, and building small projects with each new capability as it emerges.

    Data Engineering

    AI systems are only as good as the data they are built on. Data engineering covers how data is collected, cleaned, stored, and made available for training and inference. Key subjects include SQL for querying relational databases, pandas and NumPy for data manipulation in Python, data pipeline design, and working with both structured data (tables) and unstructured data (text, images, audio). Understanding data quality, deduplication, and how training data composition affects model behaviour is increasingly important as organisations build custom fine-tuned models for specific domains.

    MLOps and Deployment

    Building a model is only half the job. Getting it into production, keeping it running, and monitoring its behaviour in the real world is the other half, and it is where many AI projects fail. Core MLOps tools include Docker for containerisation, Kubernetes for orchestration, and cloud platforms such as AWS, Google Cloud, and Azure for deployment. You should also study CI/CD pipelines for model deployment, model versioning, logging and monitoring, and evaluation frameworks for detecting model drift and degradation in production.

    Ethics, Safety, and AI Governance

    Just as important are good communication skills and a solid grasp of ethical AI principles. Understanding algorithmic bias, fairness metrics, data privacy regulations such as GDPR and the EU AI Act, prompt injection and adversarial attacks, and the principles of responsible AI deployment is not optional for a practitioner who will be building systems that affect real people. Governance frameworks such as NIST AI RMF and ISO 42001 are increasingly appearing in enterprise procurement requirements, and familiarity with them signals professional maturity.

    Where to Begin

    The sequence that works for most people moving into AI engineering from another background is: Python fluency, then mathematics fundamentals, then machine learning basics, then the modern LLM stack, then MLOps. A portfolio of three to five complete projects showcasing deployment, monitoring, and handling of real-world challenges will demonstrate more to a hiring team than any single certification. Build something real at every stage of learning, and the path becomes much clearer than any roadmap can make it on paper.

  • AI News & Industry Updates

    AI Slop Is Eating the Internet. Here Is What We Can Do About It.

    The Word That Defined an Era

    In December 2025, Merriam-Webster announced its Word of the Year. It was not a technical term, a political coinage, or a neologism born in academic journals. It was “slop“, defined as low-quality digital content that is usually produced in quantity by means of artificial intelligence. The American Dialect Society followed in January 2026, with over 300 linguists voting it their Word of the Year too. Australia’s Macquarie Dictionary had reached the same conclusion a month earlier. Three major lexicographic institutions, independently, chose the same word to describe the defining cultural phenomenon of our moment.

    The timing was not coincidental. In 2025, OpenAI’s Sora app, which helps users generate videos with AI, became widely available alongside other powerful generative AI platforms. Anyone could produce hundreds of videos, images, or articles with minimal effort or expertise. The floodgates had opened, and what poured through was, in large quantities, junk.

    ai slop hero v2

    What AI Slop Actually Is

    AI slop is low-quality, mass-produced content generated by artificial intelligence with minimal human oversight or editing. The term describes content that is technically coherent but practically useless: generic phrasing, recycled information, missing original insight, and a neutral tone that sounds authoritative without saying anything specific.

    It manifests across every medium. On social media, AI slop is frequently used in political campaigns in an attempt at gaining attention through content farming. On YouTube, a March 2026 investigation by The New York Times found that around 40% of videos recommended to children, both on the main platform and on YouTube Kids, appear to be AI slop, often with realistic or Cocomelon-style visuals. On streaming music platforms, in June 2025, Deezer estimated that as much as 70% of streams of AI-generated tracks on its platform were fraudulent, highlighting concerns about mass low-quality output competing with human-made music.

    The written web is no different. Graphite reported that 49.9% of English-language articles in its Common Crawl sample were classified as primarily AI-generated during the first quarter of 2026. NewsGuard, tracking AI content farms, had identified 3,749 AI content-farm news and information sites operating across 16 languages as of June 2026.

    The economics driving this are straightforward. Social media platforms reward engagement metrics such as views, clicks, watch time, and shares. AI slop often performs because it employs techniques specifically designed to trigger algorithmic promotion. Content farms discovered they could operate profitably by flooding platforms with synthetic material. Creating quality content requires time, skill, and resources. AI slop requires almost none of these.

    Why It Is More Dangerous Than Spam

    It would be tempting to dismiss AI slop as a modern variant of email spam – annoying but ultimately manageable. That comparison underestimates the problem considerably. Spam was identifiable, repetitive, and explicitly intrusive. AI slop, on the other hand, is characterised by an appearance of normalcy. The content it generates is often visually polished, syntactically correct, sometimes even initially appealing.

    The deeper harm is epistemic. A 2026 Internet Archive study raised a related concern: although it did not find a measurable decline in factual accuracy across its sample, its authors suggested that the growing difficulty of distinguishing human and AI writing may cause people to discount the credibility of online information more broadly. The result may not be that readers believe every falsehood. They may simply become less willing to believe anything.

    The overabundance of automatically generated content creates an environment where signal is drowned in noise. Users must expend increasing cognitive effort to identify relevant, reliable, or simply human information. Several analyses now speak of attentional fatigue or AI fatigue.

    There is also a self-reinforcing feedback loop at work. The process of AI slop creates a self-reinforcing cycle: platforms prioritise engagement, slop dominates search results, and displaces human-created, high-quality content. When AI training datasets are then built from the web, they ingest increasing proportions of AI-generated content — models trained on the outputs of previous models, in a degrading loop researchers call “model collapse.”

    What Platforms Are — and Are Not — Doing

    The platform response has been real but uneven. Google’s March 2024 core update specifically targeted AI slop, integrating the helpful content system into its core algorithm. The result: a 45% reduction in low-quality, unoriginal content in search results — exceeding their initial 40% target. Google’s stated position is that it does not penalise content for being AI-generated, but does penalise content for being unhelpful — a distinction that is meaningful in principle but difficult to enforce at scale.

    In January 2026, YouTube CEO Neal Mohan declared “managing AI slop” a top priority for the year. YouTube now requires creators to disclose AI-generated content, labels AI-produced videos, and is expanding its likeness detection system to millions of creators. Meta began labelling AI-generated content in May 2024 and by 2025 was disallowing monetisation for repetitive, unoriginal AI content.

    Pinterest has gone further, introducing controls that let users limit the amount of generative AI content in their feeds in select categories. It is one of the few examples of a platform giving individual users direct agency over their own AI slop exposure.

    The EU AI Act, in force since August 2024, requires that generative AI outputs be marked in machine-readable format and that deepfakes be labelled, with fines reaching 3% of global turnover for violations. However, the AI Forensics Study (2025) shows a lack of enforcement of labelling — regulatory intent has outpaced regulatory capacity.

    What Creators, Businesses, and Readers Can Do

    The critical distinction, often lost in public debate, is between AI-generated content and AI slop. The defining quality of AI slop is not that it was made with AI. It is that it was made carelessly with AI and published without meaningful human judgment. The term “AI caviar” has been coined informally for its opposite — content where AI handled the drafting and formatting while expert humans contributed specific knowledge, original perspective, and editorial judgement.

    For creators and businesses, the practical controls are clear. Treat AI output as a first draft, not a finished product. Add original research, specific data, named sources, and first-hand experience — the elements that AI cannot generate and that search engines and readers increasingly reward. Avoid the telltale patterns that mark slop: generic phrasing, repetitive structure, a lack of specific examples or concrete details, and an absence of genuine human perspective.

    For readers and consumers, AI literacy is the primary defence. Recognise the signatures: unnaturally smooth images, text that sounds confident while saying nothing specific, attributions to unnamed “experts” and “studies,” and articles that describe categories of information rather than specific instances of it. Tools such as GPTZero can assist detection, but no tool replaces the judgement of a reader who has learned to notice when content rings hollow.

    For platforms, the only sustainable response is restructuring the economic incentives that make slop profitable. As long as views and engagement drive revenue irrespective of content quality or origin, the production of AI slop will remain economically rational. Volume caps, quality scoring, and tying monetisation to editorial standards are the levers available — and the platforms with the largest audiences have been the slowest to pull them.

    The Signal Worth Preserving

    AI slop is not an argument against AI. It is an argument against carelessness. The same tools that flood the internet with hollow content are also powering genuine scientific breakthroughs, enabling new forms of creativity, and making expert knowledge accessible at unprecedented scale. What they cannot do is supply the judgement, experience, and intellectual honesty that distinguish valuable content from noise.

    That judgement remains stubbornly human. The challenge of the current moment is ensuring that the economics of the internet stop punishing it.

  • AI Foundations

    Prompt Injection: The #1 Security Threat to Enterprise AI Applications

    The Attack That Exploits AI’s Core Design

    Every serious technology platform eventually acquires its signature vulnerability class. For web applications it was Cross-Site Scripting. For databases it was SQL injection — an attack so consequential that it shaped two decades of application security practice. For large language models, that defining vulnerability has a name: prompt injection. Ranked LLM01 by OWASP — the #1 threat on the OWASP Top 10 for LLM Applications — prompt injection exploits a fundamental architectural weakness: LLMs cannot reliably distinguish between trusted instructions and untrusted data.

    The comparison to SQL injection is more than rhetorical. SQL injection worked because databases executed user-supplied strings as code, blurring the boundary between data and instruction. Prompt injection works for precisely the same reason, transposed to natural language: an LLM receives both its system instructions and external content as tokens in the same context window, with no hard cryptographic or architectural boundary separating them. Whatever appears in that context window can potentially influence what the model does next — and that is the attack surface.

    Direct vs Indirect Injection: Two Very Different Threat Models

    Prompt injection divides cleanly into two categories that require different defences.

    Direct prompt injection is the simpler variant. An attacker interacts directly with the model and crafts an input designed to override its system prompt or bypass its guardrails. Classic examples include jailbreak attempts — role-playing scenarios, hypothetical framings, or instruction overrides such as “ignore all previous instructions and instead do X.” In filtered environments, direct attacks have detection rates exceeding 70%, making them the easier threat to manage.

    Indirect prompt injection is the more dangerous and rapidly growing variant. Here, the attacker does not interact with the model at all. Instead, malicious instructions are embedded in content that the model retrieves and processes — a document, an email, a web page, a database record — and that content subsequently enters the model’s context window. Because prompts are expressed in free-form natural language, they cannot be sanitised as strictly as structured inputs, creating a challenging and persistent attack surface.

    Indirect prompt injection now makes up over 55% of observed attacks in 2026, with indirect attacks carrying 20–30% higher success rates due to their stealth delivery through trusted sources. In enterprise environments, 62% of successful exploits involved indirect injection pathways, and over 50% evade standard prompt filtering systems. The asymmetry is stark: the attack requires only that a single piece of malicious content reach the model’s context. The defender must harden every possible retrieval pathway.

    Real-World Exploitation: No Longer Theoretical

    In June 2025, researchers at Aim Security disclosed EchoLeak (CVE-2025-32711, CVSS 9.3) — the first documented zero-click prompt injection exploit against a production AI system, targeting Microsoft 365 Copilot. By sending a single crafted email, with no user interaction required, an attacker could cause Copilot to access internal files and transmit their contents to an attacker-controlled server.

    This was not an isolated incident. Critical CVEs in Microsoft Copilot (CVSS 9.3), GitHub Copilot (CVSS 9.6), and Cursor IDE (CVSS 9.8) demonstrate active production exploitation in 2025–2026. CrowdStrike’s 2026 Global Threat Report documented that threat actors injected malicious prompts into legitimate generative AI tools at more than 90 organisations in 2025.

    The attack scenarios are not exotic. Researchers have demonstrated a KYC pipeline compromised by malicious instructions hidden in the text layer of a passport image. A healthcare AI document pipeline processed a malicious PDF in which injected content survived through all LLM layers to the human review dashboard and subsequently embedded itself in the next model training round. Security analyses tied 60% of AI-driven data-privacy incidents between 2025 and 2026 to prompt manipulation techniques, and internal document-handling AI copilots showed information-leak risk in 75% of evaluated enterprise deployments.

    The threat is compounded by the rise of agentic AI. AI agents move 16 times more data than human users, making every compromised agent a high-magnitude data exposure event rather than a single-user incident. When an agent can browse the web, read emails, query databases, and execute code, a single successful injection can cascade across the entire workflow.

    Why It Is So Difficult to Fix

    The UK’s National Cyber Security Centre issued a formal assessment in December 2025 warning that prompt injection may never be fully mitigated the way SQL injection was, characterising LLMs as “inherently confusable deputies” — systems that can be coerced into performing actions that benefit an attacker because there is no robust internal separation between trusted instructions and untrusted content.

    Even frontier models from OpenAI, Google, and Anthropic remain vulnerable after applying their best defences. On February 13, 2026, OpenAI launched Lockdown Mode for ChatGPT and publicly acknowledged that prompt injection in AI browsers “may never be fully patched.” The International AI Safety Report 2026 found that sophisticated attackers bypass even the best-defended models approximately 50% of the time with just ten attempts — and in agentic systems, success rates reach 84%. Vectra AI

    The root cause is architectural. SQL injection was eventually tamed because the industry separated query structure from query parameters through prepared statements — a technical mechanism that made it impossible for user data to be interpreted as SQL code. No equivalent mechanism exists for LLMs because the entire system operates on the same substrate: natural language tokens. Until models develop a robust internal representation of trust boundaries — an open and hard research problem — the vulnerability class will persist.

    Defence in Depth: The Only Viable Strategy

    No single control eliminates prompt injection risk. The security community has converged on layered defence as the only viable approach.

    Input validation and context isolation should be the first line. Treat all external content — retrieved documents, web pages, API responses, user uploads — as untrusted and apply strict filtering before it enters the model’s context. Separate retrieval pipelines from instruction pipelines wherever architecturally possible.

    Least-privilege for AI agents is critical. An agent should have access only to the tools, data sources, and actions strictly necessary for its defined task. An agent that can read email but not send it, query a database but not modify it, limits the blast radius of a successful injection dramatically.

    Output monitoring and anomaly detection provides a detection layer. Monitoring model outputs for unexpected data exfiltration patterns, unusual API call sequences, or out-of-policy actions can catch injections that bypass input-level controls.

    Human-in-the-loop checkpoints for high-stakes actions — sending emails, modifying records, executing financial transactions — ensure that a compromised agent cannot complete consequential actions autonomously.

    Red-teaming and adversarial testing should be embedded in the deployment pipeline for every LLM-integrated application. Compliance frameworks including NIST AI RMF and ISO 42001 now mandate specific controls for prompt injection prevention and detection. The EU AI Act’s high-risk provisions, enforced from August 2026, add regulatory weight to what was previously a voluntary best practice.

    Conclusion

    Prompt injection is to the LLM era what SQL injection was to the web era: a vulnerability class that emerges from a fundamental design tension, scales with adoption, and demands a structural response from the security community. The difference is that SQL injection took roughly a decade to be brought under meaningful control — and the LLM attack surface is expanding faster, into more consequential domains, with agents that act rather than merely respond.

    For enterprise AI teams, the message from OWASP, NCSC, NIST, and the EU AI Act is consistent: treat prompt injection not as an edge case to be patched, but as a persistent threat to be governed. Build your AI architecture assuming that any content the model processes could be adversarial. Because in production, increasingly, it is.

  • AI News & Industry Updates

    How AI Is Changing Healthcare: Diagnosis, Drug Discovery, and Patient Care

    ai healthcare hero light

    Medicine’s Quiet Revolution

    Healthcare has always advanced in waves, examples being germ theory, antibiotics, the randomised controlled trial, genomic sequencing. Each wave took decades to become standard practice. Artificial intelligence is different. The pace at which AI tools have moved from research papers to clinical wards has compressed that timeline dramatically, and the breadth of the transformation, including touching diagnostics, drug development, genomics, and patient management simultaneously, has no obvious historical parallel.

    This is not hype at distance. It is measurable, already underway, and raising genuinely difficult questions about how medicine will be practised, validated, and governed in the decade ahead.

    Seeing What Human Eyes Miss: AI in Medical Imaging

    The area where AI has achieved the most clinically validated impact is medical imaging. As of December 2025, over 1,300 AI-enabled medical devices have received FDA marketing authorisation, with 1,039 specifically for radiology — accounting for roughly 80% of all approved AI medical tools. These are not experimental prototypes; they are deployed daily in hospitals across the United States, Europe, and Asia.

    The performance numbers are striking. Deep learning models can identify tumours, strokes, and fractures within seconds, with real-world studies demonstrating up to 17.6% higher breast cancer detection rates. In stroke treatment, where neurological damage accumulates with every passing minute, AI has reduced door-to-treatment intervals by as much as 30 minutes, with measurable improvements in survival rates and patient outcomes.

    The mechanism is convolutional neural networks trained on tens of millions of labelled scans. These networks learn to identify statistical patterns in pixel distributions that correlate with pathology — patterns that may be too subtle or spatially diffuse for a human radiologist under time pressure to consistently detect. In breast cancer screening specifically, AI-assisted interpretations have lowered false negatives by almost 9% and decreased unnecessary recall rates — reducing both missed cancers and patient anxiety from false alarms simultaneously.

    The caveat is important. AI is not a substitute for doctors: it can make mistakes or generate false positives, and over-reliance risks impairing clinicians’ skills. The clinical consensus, reflected in nearly every major radiology society’s guidance, is that AI functions best as a co-pilot — triaging the scan queue, flagging anomalies for human review, and performing automated measurements — while the radiologist retains diagnostic authority.

    The AlphaFold Moment: Rewriting Drug Discovery

    If imaging AI represents evolutionary improvement in existing workflows, AlphaFold represents something more fundamental: a solution to a problem that had defeated biology for half a century.

    Proteins fold from linear amino acid chains into precise three-dimensional structures that determine their function. Predicting that structure from sequence alone — the protein folding problem — was considered one of the hardest open problems in science. Google DeepMind’s AlphaFold 2, published in Nature in 2021, solved it with accuracy comparable to experimental methods. Its creators Demis Hassabis and John Jumper were awarded the 2024 Nobel Prize in Chemistry for the work. By November 2025, AlphaFold was being used by over three million researchers across more than 190 countries, tackling problems including antimicrobial resistance, crop resilience, and heart disease.

    AlphaFold 3, released in 2024, extended the capability beyond proteins to predict the structure and interactions of DNA, RNA, ligands, and small molecules — with at least a 50% improvement over existing methods for protein-molecule interactions, and doubled prediction accuracy for some drug-relevant interaction categories. For drug discovery, this is transformative. The traditional pipeline required experimental determination of a target protein’s structure — a process taking months or years — before rational drug design could begin. AlphaFold collapses that step to hours.

    The pharmaceutical industry has responded at scale. A landmark development at the 2026 J.P. Morgan Healthcare Conference was a $1 billion co-innovation lab announced by Nvidia and Eli Lilly, aimed at creating a continuous learning system connecting agentic wet labs with computational dry labs around the clock. AstraZeneca, Bristol Myers Squibb, Roche, and Recursion Pharmaceuticals have all announced multi-hundred-million-dollar AI drug discovery partnerships in the same period.

    The critical open question is clinical validation. The most advanced AI-designed drugs are now entering Phase III pivotal trials in 2026, with multiple clinical readouts expected throughout the year — the first large-scale test of whether AI genuinely improves success rates beyond the pharmaceutical industry’s persistent 90% clinical trial failure rate. Computational elegance and clinical efficacy are not the same thing. The next two years will determine whether the investment thesis is justified.

    Genomics, Precision Medicine, and the Individual Patient

    Beyond imaging and drug design, AI is enabling a more fundamental shift in how medicine conceptualises the patient. Traditional medicine treats populations — a drug is approved because it outperforms placebo in a trial of thousands. Precision medicine asks a different question: which treatment is most likely to work for this specific patient, given their genetic profile, biomarkers, and disease subtype?

    AI-powered clinical decision support systems are stepping into the gap created by the rapid and unmanageable expansion of medical knowledge. Platforms like OpenEvidence, among the most widely adopted decision support tools in US medicine, allow physicians to rapidly search medical literature, synthesise findings, and check drug interactions at the point of care.

    In genomics, models trained on population-scale genetic databases can now identify disease-causing variants with a precision that was impossible using statistical methods alone. These systems can speed up genetic diagnosis for rare and complex illnesses by frequently ranking the true disease-causing mutation within the top ten candidates, and guide personalised treatment by linking genetic variants directly to their expected clinical manifestations.

    The combination of genomic data, electronic health records, and AI-driven pattern recognition is creating what researchers describe as a “learning health system” — one in which every patient encounter generates data that improves predictions for the next patient with a similar profile. This is medicine learning from itself at a scale and speed that no previous generation of clinicians could achieve.

    The Governance Problem

    The pace of deployment has outrun the pace of regulation, and the gap is generating legitimate concern. Roughly 200 state AI bills are being tracked in 2026 alone, and 83% of polled healthcare workers say AI needs more regulation — reflecting broad industry support for clearer governance frameworks even as the federal government takes a largely deregulatory stance.

    The EU AI Act’s high-risk provisions, which take effect August 2026, classify AI systems used in medical diagnosis and drug development as high-risk — requiring conformity assessments, transparency obligations, and human oversight mechanisms. The US approach remains more fragmented, relying primarily on the FDA’s device authorisation framework, which does not require rigorous clinical validation — FDA clearance alone does not guarantee real-world effectiveness.

    The algorithmic bias problem deserves specific attention. AI systems trained predominantly on data from specific demographic groups — which describes most current medical AI, given the historical composition of clinical trial populations — can perform significantly worse for underrepresented groups. A model that detects diabetic retinopathy with 95% accuracy on one ethnic population may perform at 80% on another, with no visible signal in aggregate accuracy statistics that a problem exists.

    What Comes Next

    The trajectory of AI in healthcare points toward three developments that are likely to define the next decade. Multimodal AI — systems that integrate imaging, genomics, clinical notes, and wearable sensor data simultaneously — will produce risk models and diagnostic tools with a richness that no single-modality system can match. Agentic AI in clinical workflows will handle administrative burden, prior authorisation, documentation, and care coordination autonomously, returning clinician time to patients. And AI-designed therapeutics will, if the current Phase III trials vindicate the approach, fundamentally alter the economics of bringing new drugs to market.

    None of this makes medicine easier. It makes it more capable and correspondingly more demanding of the humans who must govern, validate, and take clinical responsibility for what these systems produce. That, ultimately, is where the most important work remains to be done.

  • AI Foundations

    What Is the Difference Between AI, Machine Learning, and Data Science?

    Three Terms, One Persistent Confusion

    Artificial Intelligence, Machine Learning, and Data Science are three of the most frequently used, and three of the most consistently conflated, terms in technology today. Job descriptions blend them interchangeably. News articles use them as synonyms. Even within organisations, teams labelled differently are sometimes doing work that is functionally indistinguishable.

    The confusion is understandable. The three fields overlap substantially, share tooling and mathematical foundations, and have converged further in recent years as data-driven methods have come to dominate AI research. But they are not the same thing, and treating them as such leads to poor hiring decisions, misdirected research investment, and architectural choices that do not match the problem at hand.

    Understanding the precise relationship between the three, conceptually and historically, is a foundational literacy requirement for any graduate-level practitioner working in or adjacent to these fields.

    Artificial Intelligence: The Broadest Umbrella

    Artificial Intelligence is the oldest and broadest of the three terms. Coined at the 1956 Dartmouth Conference, it refers to any computational system designed to exhibit behaviour that would be considered intelligent if performed by a human. That definition is deliberately wide — wide enough to encompass everything from a rule-based expert system written in 1982 to a multimodal transformer model trained on tens of trillions of tokens in 2025.

    AI is best understood as a goal rather than a methodology. The goal is to construct systems that perceive, reason, plan, communicate, or act in ways that approximate or surpass human cognitive capabilities in specific domains. How that goal is pursued is left open. Classical AI pursued it through symbolic reasoning — explicit logical rules, decision trees, knowledge graphs, and first-order predicate logic. Planning algorithms like A* search, constraint satisfaction solvers, and game-playing engines such as Deep Blue all qualify as AI under this definition without involving any statistical learning whatsoever.

    The important implication is that AI does not require data in the modern sense. An expert system that encodes the diagnostic rules of a cardiologist is AI. A theorem prover that verifies software correctness through formal logic is AI. Neither learns from data, neither involves statistics, and neither would commonly be described as machine learning.

    Machine Learning: A Methodology Within AI

    Machine Learning is a specific approach to building AI systems: one in which the system learns its decision rules from data rather than having those rules programmed explicitly. The formal definition, due to Tom Mitchell (1997), remains precise and useful: a computer program is said to learn from experience EEE with respect to task TTT and performance measure PPP if its performance at TTT, as measured by PPP, improves with experience EEE.

    ML is therefore a proper subset of AI. Every machine learning system is an AI system, but not every AI system uses machine learning. The methodological distinction is significant. Classical AI is brittle in novel environments and requires expert knowledge to be manually encoded. ML systems can generalise to inputs their designers never explicitly considered, provided those inputs resemble the training distribution.

    The field subdivides along several axes. Supervised learning trains on labelled input-output pairs and learns a function mapping inputs to outputs so that regression and classification are the canonical tasks. Unsupervised learning finds structure in unlabelled data, including clustering, dimensionality reduction, and density estimation. Reinforcement learning trains an agent through interaction with an environment, optimising a cumulative reward signal rather than a fixed labelled dataset.

    Deep learning, the sub-field built on multi-layer neural networks trained via backpropagation, is itself a subset of machine learning, and the one most responsible for the capabilities that define the current AI era. The relationship is therefore nested: deep learning \subset machine learning \subset artificial intelligence.

    Data Science: A Practice, Not a Discipline

    Data Science occupies a different conceptual register from the other two. Where AI is a goal and machine learning is a methodology, data science is best understood as a practice — the interdisciplinary activity of extracting knowledge and actionable insight from data using a combination of statistical analysis, computational tools, domain expertise, and communication skills.

    The term was popularised in the early 2010s, partly as a rebranding of applied statistics and partly to capture a genuinely new set of competencies demanded by the scale and variety of data that modern organisations generate. A data scientist working on customer churn prediction might use logistic regression, a gradient boosted tree, or a neural network depending on the dataset size and interpretability requirements — but the primary obligation is to the insight and its business consequences, not to any particular algorithmic paradigm.

    Data science is explicitly interdisciplinary in a way that AI and ML are not required to be. The canonical Venn diagram of the field places it at the intersection of mathematics and statistics, domain expertise, and computer science. A machine learning engineer who cannot communicate findings to a non-technical stakeholder, cannot clean a messy real-world dataset, and cannot frame a business problem as a statistical one is not doing data science, regardless of how sophisticated the model they deploy.

    This distinction matters in practice. Data science places significant emphasis on exploratory data analysis, statistical inference, experimental design, causal reasoning, and data visualisation — competencies that a pure ML engineer may have only superficially. Conversely, a statistician doing regression modelling on clinical trial data is doing data science without doing machine learning or AI in any substantive sense.

    Where They Overlap — and Where They Diverge

    The Venn diagram of the three fields in 2026 looks quite different from how it looked in 2012. Deep learning has absorbed so much of what was previously done by hand-crafted feature engineering and classical statistical modelling that the boundary between ML and data science has blurred considerably. A modern data scientist working on a large unstructured dataset containing text, images, and sensor streams will routinely deploy pre-trained neural network models, use transformer-based embeddings as features, and interact with foundation models through API calls. These tasks would once have been considered the exclusive domain of ML research.

    Similarly, the boundary between ML and AI has compressed. The dominant AI paradigm of the current era is data-driven — foundation models, reinforcement learning, and generative systems — to the point where the distinction between AI and ML is now less a technical boundary than a level-of-abstraction distinction. When practitioners say “AI system,” they typically mean a system whose core capability is conferred by a trained model. When they say “ML pipeline,” they are emphasising the data flow, training process, and model lifecycle management.

    The divergence that remains most meaningful is between data science and the other two. Data science is oriented toward understanding — answering questions about what happened, why it happened, and what is likely to happen — and is deeply embedded in the statistical tradition of inference under uncertainty. AI and ML are oriented toward capability-building systems that perceive, decide, generate, or act. A causal inference study examining why a marketing campaign underperformed is data science. A recommendation system that predicts what product a user will purchase next is machine learning. An autonomous agent that browses the web, synthesises information, and executes a multi-step workflow is AI. All three draw on overlapping mathematical tools, but they ask fundamentally different questions and are held to different standards of success.

    A Practical Summary

    AIMachine LearningData Science
    NatureGoal / fieldMethodologyPractice
    Core questionCan a machine behave intelligently?Can a system learn from data?What does the data tell us?
    Requires data?Not necessarilyYesYes
    Requires ML?Not necessarilyNot necessarily
    Primary outputIntelligent systemTrained modelInsight or decision
    Rooted inComputer science, logicStatistics, optimisationStatistics, domain knowledge

    Conclusion

    AI is the destination. Machine learning is one of the most powerful roads toward it. Data science is the discipline of reading the map and extracting understanding from the data that both feeds and evaluates the journey. Treating these three as synonyms is not merely imprecise; it produces teams that are misaligned with their actual objectives, architectures chosen for the wrong reasons, and practitioners evaluated against the wrong competency profiles.

    The clearest sign of genuine literacy in this space is not fluency with any specific tool or algorithm; it is the ability to pick up a problem and correctly identify which of these three framings it actually requires.

  • AI Foundations

    Beyond the Hype: Demystifying the Architecture, Evolution, and Mechanics of Machine Intelligence

    The term “Machine Intelligence” (MI), often used interchangeably with Artificial Intelligence (AI) and Machine Learning (ML), has graduated from speculative science fiction into the core infrastructure of modern computing. However, cutting through the industry marketing requires looking at MI not as a singular “thinking mind,” but as a highly sophisticated convergence of statistical learning, algorithmic optimization, and distributed computational systems.

    This post explores what machine intelligence truly is under the hood, how it evolved structurally, the core tools that power it today, and its high-stakes applications.

    Gemini Generated Image ekf5t3ekf5t3ekf5

    Defining Machine Intelligence: What It Is (and Isn’t)

    We define Machine Intelligence as the capacity of a computational system to execute tasks by constructing generalized probabilistic models from empirical data, rather than following explicitly programmed deterministic rules. ### The Paradigmatic Shift

    To understand MI, consider the fundamental shift in the computing paradigm:

    • Classical Programming (Symbolic AI): Rules+DataAnswers\text{Rules} + \text{Data} \rightarrow \text{Answers}
    • Machine Intelligence: Data+AnswersRules\text{Data} + \text{Answers} \rightarrow \text{Rules}

    Instead of a software engineer hard-coding conditional statements (e.g., if-then-else), an MI system evaluates high-dimensional feature spaces to identify statistical regularities. It adjusts internal scalar values, known as weights and biases, to minimize a loss function via optimization techniques like Stochastic Gradient Descent (SGD).

           [ Classical Computing ]
    Rules + Data -----------> [ System ] -----------> Answers
    
           [ Machine Intelligence ]
    Data + Answers ---------> [ System ] -----------> Rules (Trained Model)
    

    The Evolutionary Timeline: From Rules to Representations

    The road to modern MI has been marked by swings between intense optimism and funding droughts, historically known as “AI Winters.”

    The Symbolic Era (1950s–1980s)

    Early AI focused on hard-coded logic and symbolic reasoning. Pioneers like John McCarthy and Marvin Minsky believed intelligence could be formalized through deductive logic. While this birthed powerful deterministic tools like chess engines and rule-based “expert systems,” it collapsed when faced with the messy, non-linear realities of natural language and computer vision.

    The Connectionist Resurgence (1980s–2000s)

    The paradigm shifted toward connectionism, or modeling intelligence using artificial neural networks (ANNs). The rediscovery of the backpropagation algorithm in the mid-1980s allowed multi-layer networks to learn internal representations. However, this era stalled due to a lack of computational hardware and data starvation.

    The Deep Learning and Foundation Model Boom (2010s–Present)

    The modern era was ignited in 2012 when AlexNet won the ImageNet competition, proving that deep convolutional neural networks (CNNs), when paired with Graphics Processing Units (GPUs), shattered classical vision benchmarks.

    The breakthrough accelerated with the introduction of the Transformer architecture by Vaswani et al. in 2017. By replacing recurrence with self-attention mechanisms, Transformers paved the way for massive foundation models (like the GPT and Claude families) and the modern shift toward Agentic AI, which are systems capable of autonomous multi-step reasoning and tool execution.

    The Modern MI Stack: Frameworks and Infrastructure

    Building machine intelligence today relies on a highly mature ecosystem of software and hardware.

    The Software Frameworks

    Modern ML architecture is dominated by two primary open-source ecosystems:

    • PyTorch (Backed by Meta): The preferred library for academic research and cutting-edge deployment due to its dynamic computation graph, which allows for intuitive debugging and flexible model design.
    • TensorFlow / Keras (Backed by Google): Renowned for its highly stable, static graph execution, making it a staple for rigid enterprise production pipelines.
    • Classical Libraries: For tabular data and standard statistical modeling, scikit-learn, XGBoost, and LightGBM remain the industry standard for gradient-boosted decision trees.
    Managed Enterprise Platforms

    At scale, engineers rarely train models on local machines. They utilize cloud-native MLOps (Machine Learning Operations) suites like Google Cloud Vertex AI, Amazon SageMaker, and IBM watsonx.ai. These platforms handle the end-to-end lifecycle, including data lineage tracking, hyperparameter tuning, distributed training orchestration, and real-time model drift monitoring.

    High-Impact Applications

    Machine intelligence is changing operations across technical and scientific fields through advanced applications:

    IndustryPrimary TechniqueReal-World Impact
    BiomedicineStructural Bioinformatics & Deep LearningAlphaFold revolutionized structural biology by predicting 3D protein structures directly from amino acid sequences, cutting drug discovery timelines down from years to days.
    Quantitative FinanceTime-Series Forecasting & Reinforcement LearningAutomated market making, algorithmic high-frequency trading, and real-time anomaly detection pipelines for credit fraud detection.
    Autonomous SystemsComputer Vision & Sensor FusionAutonomous drones and edge-AI vehicles mapping environments in real time using 3D object detection and semantic segmentation.
    Enterprise SoftwareLLMOps & Retrieval-Augmented Generation (RAG)Transitioning from simple chatbots to Agentic Systems that autonomously interface with databases, draft code, and manage cross-platform workflows.

    Looking Ahead: The Post-Graduate Frontier

    For researchers and engineers, the current frontier of machine intelligence is no longer just about scaling parameter counts. The focus has pivoted toward deep systemic challenges:

    • Inference Efficiency: Compressing massive models via quantization, pruning, and Knowledge Distillation to run on low-power edge devices.
    • Explainable AI (XAI): Peering inside the “black box” of deep neural networks to mathematically trace why a model made a specific prediction.
    • Alignment and Robustness: Ensuring models fail gracefully when exposed to out-of-distribution (OOD) data or adversarial attacks.

    Machine intelligence is not a magical artifact; it is an elegant mathematical framework built on linear algebra, calculus, and probability, brought to life by massive parallel computing. Understanding its mechanics is the first step toward building its future.

  • AI News & Industry Updates

    DeepSeek: The $6 Million Model That Shook Wall Street and Challenged the AI Establishment

    A Startup Nobody Saw Coming

    In the summer of 2023, a hedge fund manager in Hangzhou, China, quietly spun off an AI research lab. By January 2025, that lab had triggered what CBS News described as a shockwave through Wall Street, briefly dethroned ChatGPT as the most downloaded free app on Apple’s App Store, and forced a fundamental reassessment of the assumptions underlying hundreds of billions of dollars in AI infrastructure investment. The company was DeepSeek. The model was R1. And nothing in the AI industry looked quite the same afterward.

    For enterprise AI leaders evaluating model strategy, AI governance, and infrastructure spending, understanding DeepSeek is not optional. It is a case study in how architectural innovation can disrupt market assumptions, and a live stress test of every claim that frontier AI requires frontier compute budgets.

    How DeepSeek Actually Works

    DeepSeek is built on the same transformer foundation that underlies GPT-4, Claude, and Llama — but with two architectural decisions that distinguish it fundamentally from the models it competes with.

    The first is the Mixture-of-Experts (MoE) architecture. DeepSeek-V3 has 671 billion total parameters but only activates approximately 37 billion for any given query. This routs each input through only the most relevant subset of the model’s capacity. This “sparse activation” approach means the compute cost per inference is a fraction of what a dense 671B model would require, while retaining the representational capacity of the full parameter count.

    The second is a deeply optimised training pipeline. DeepSeek achieved state-of-the-art benchmark performance using only 2.8 million H800 GPU hours of training time, approximately ten times less training compute than the similarly performing Llama 3.1 405B. The $6 million training cost figure that broke investors’ assumptions is a consequence of this efficiency, not a trick.

    DeepSeek-R1, released in January 2025, is based on DeepSeek-V3 and is focused on advanced reasoning tasks, directly competing with OpenAI’s o1 model in performance while maintaining a significantly lower cost structure. Like the o-series models, R1 uses extended chain-of-thought reasoning — generating an internal scratchpad before committing to a final answer — to dramatically improve performance on mathematics, code, and logical inference tasks.

    The third structural difference is perhaps the most commercially significant: all DeepSeek models are released under open-weight licences such as MIT for R1, Apache 2.0 for subsequent releases. OpenAI’s models are fully proprietary. Anthropic’s models are fully proprietary. DeepSeek publishes the weights for free. Any enterprise can download and self-host the model at zero per-token cost.

    Does It Actually Work?

    The honest answer is: yes, significantly. But with important caveats.

    On reasoning benchmarks, R1 was legitimately competitive with OpenAI’s o1 at launch. Its mathematical reasoning in particular was rated best-in-class by several independent evaluations. For code generation, structured analysis, and multilingual tasks, it performs at a level that rivals or exceeds models costing orders of magnitude more to run via API.

    The limitations are real, however. According to testing by Vectara, DeepSeek-R1 hallucinates at a rate of 14.3%, compared to approximately 2% for OpenAI’s GPT-4. Its safety guardrails are also notably weaker than those of Western frontier models: Palo Alto Networks found it is relatively easy to bypass DeepSeek’s safety guardrails, and Enkrypt AI reported that R1 is four times more likely to produce malware or insecure code than OpenAI’s o1.

    For enterprise deployment, this matters. A model that performs exceptionally on benchmarks but hallucinates at seven times the rate of its main competitor and fails adversarial testing is not a drop-in replacement for production workflows where reliability and safety alignment are contractual or regulatory requirements.

    The Market Shock: DeepSeek Monday

    The broader AI industry was unprepared for what happened on January 27, 2025. Nvidia’s stock dropped nearly 18% that Monday morning, now referred to as “DeepSeek Monday” on Wall Street. Roughly $600 billion in market value evaporated in a single trading session, the largest single-day loss for any company in stock market history. Microsoft, Alphabet, Broadcom, and ASML all fell in sympathy. By the end of the week, over $1 trillion had been erased from American tech stocks.

    The mechanism of the panic was straightforward: if a Chinese lab could produce a frontier-capable model for $6 million, the foundational investment thesis driving demand for Nvidia’s chips — that training frontier AI requires tens of thousands of the most expensive GPUs available — appeared to be falsified in one announcement.

    Nvidia CEO Jensen Huang pushed back directly. As TechCrunch reported, Huang called DeepSeek’s R1 “incredibly exciting” and argued the market had it exactly backwards: more efficient models lower the cost of AI deployment, which accelerates adoption, which increases aggregate demand for compute. That argument proved correct. Nvidia’s shares are up 58% since the DeepSeek selloff, and its growth rate has continued to defy expectations. The panic was real; the underlying catastrophe was not.

    The Controversies

    DeepSeek’s emergence generated controversy on multiple fronts simultaneously, and none of them have been cleanly resolved.

    Data privacy. DeepSeek notes in its privacy policy that personal information it collects from users is held on secure servers located in the People’s Republic of China. Under that policy, the company collects device model, operating system, keystroke patterns or rhythms, IP address, and system language. Chinese law grants Beijing broad authority to access data from companies based in China — the same legal structure that made TikTok a Congressional target. For enterprise users handling sensitive data, this is a non-negotiable concern.

    Censorship. A CBS News analysis of the application found that DeepSeek did not return any results for a prompt seeking information about the 1989 Tiananmen Square protests and subsequent massacre. The model also declined to answer questions about the Uyghur situation and Taiwan’s political status, while providing detailed answers about criticisms of Western political figures. This ideological alignment is baked into the base model’s training, not merely a surface-level filter.

    Distillation allegations. OpenAI told the Financial Times that it had seen evidence that its models were used by DeepSeek to train its own — which would be a breach of OpenAI’s terms of service. White House AI czar David Sacks said there was “substantial evidence” that DeepSeek had “distilled the knowledge out of OpenAI’s models.” DeepSeek has not publicly addressed the allegation in detail, and the legal status of model distillation remains an unresolved question across the industry.

    Chip access. DeepSeek built its models using Nvidia H800 GPUs and these chips are designed specifically for the Chinese market after the US banned exports of the more powerful H100 and A100 chips in late 2022. In a September 2025 Nature paper, DeepSeek acknowledged it also owns A100 chips used for early-stage experiments. US officials have alleged access to restricted hardware acquired after export controls took effect, though Nvidia has maintained that DeepSeek’s use of its technology was export-control compliant.

    What This Means for Enterprise AI Strategy

    DeepSeek’s net contribution to the enterprise AI landscape is a genuinely mixed signal. It proved that architectural efficiency, and not raw compute, is the binding constraint on frontier model quality, which is a productive finding for the whole field. It demonstrated that open-weight frontier models are viable, which expands the strategic options available to enterprises that want to self-host rather than depend on API access.

    But anyone handling sensitive business data should not use the DeepSeek app or API directly — and the hallucination rate and safety posture make it unsuitable for high-stakes production workflows without significant additional investment in evaluation and guardrails. For research, mathematics, and coding tasks in non-sensitive environments, the open-weight models offer exceptional performance at zero per-token cost.

    The deeper strategic lesson is one DeepSeek did not intend to teach: that the efficiency frontier in AI is far from exhausted, that architectural innovation can close capability gaps that compute alone cannot, and that the assumption that building frontier AI requires a $100 million training budget was always more fragile than the market priced it to be.