First: What Are We Actually Talking About?

Before the steps, it helps to see the range of what a sovereign agent can actually do, because “AI agent” means very different things depending on the setting.

Private sector examples: - A customer service agent for a telecom or insurance company, grounded in the organization’s own contracts, policy documents, and billing systems — able to resolve a disputed charge, explain a coverage clause, or update an account, while escalating anything genuinely ambiguous to a human. - A contract and compliance review agent for an in-house legal team, checking incoming vendor contracts against the organization’s own playbook and flagging deviations, exactly as covered in the earlier post on the institutional moat. - An HR case-handling agent, resolving routine leave, payroll, and benefits questions — one of the highest-ROI starting points precisely because it’s rule-heavy, high-volume, and entirely digital, as covered in the earlier governance post. - An agricultural advisory agent for a farming cooperative, grounded in historical logbooks, current EU organic legislation, and real-time weather data — the orchard example covered in an earlier post. - A clinical documentation assistant for a hospital, reconstructing the implicit reasoning behind treatment decisions from ICU notes.

Public sector examples — often the most interesting, and the most demanding: - A policy implementation assistant that helps caseworkers apply a new subsidy scheme or environmental regulation consistently across hundreds of cases, reducing the variation that inevitably creeps in when dozens of humans interpret the same rule slightly differently. - A citizen-facing navigation agent that helps someone apply for housing benefit, understand a tax notice, or figure out which permit they need — walking them through the actual bureaucratic process, citing the correct forms and deadlines, and clearly flagging the point at which they need a human caseworker or a lawyer rather than continuing to rely on the agent. - A benefits-eligibility assessment agent that pre-screens simple, unambiguous applications (freeing caseworkers for the genuinely hard cases) while routing anything borderline to a human — never issuing a final rejection on its own. - A municipal permitting assistant helping a small business owner understand which environmental, zoning, or safety regulations apply to their specific situation, before they ever need to speak to an official.

What all of these share is the same underlying requirement: they need to be grounded in your rules, your procedures, and your exceptions — not the generic training data of a foundation model — and, especially in the public-sector cases, they need to be auditable, fair, and clearly bounded in what they’re allowed to decide versus merely suggest.

Step 1: Build (or Rent) a Sovereign Hardware Platform

Before any of the interesting work happens, you need somewhere to run it that you actually control. As covered in the earlier post on AI sovereignty, this doesn’t mean you need to out-build Google — it means being deliberate about where inference actually happens, not just where training happens.

Realistic options, roughly in order of how much control they give you:

• Fully on-premise, on hardware your own IT department manages — the most control, the most upfront investment, and the right choice when data sensitivity (medical, law-enforcement, or classified data) makes anything else a non-starter.
• A trusted local or national cloud — a government-run community cloud (increasingly common across the EU, exactly the kind of infrastructure sovereignty investment covered in the earlier sovereignty post), or a regional/EU-based sovereign cloud provider bound by domestic data-protection law rather than foreign jurisdiction.
• Your corporate IT department’s private cloud, if your organization already runs one — often the pragmatic middle ground for companies that don’t want to manage physical hardware but still need contractual and jurisdictional control over where data lives and who can access it.

The sizing question matters here too: a fine-tuning workload needs meaningfully more compute than a pure inference deployment. If you’re following the compression path described in a later step, a specialized, distilled, quantized model can often run comfortably on a single consumer-grade or mid-range enterprise GPU for inference — which is precisely what makes on-premise or small-scale sovereign cloud deployment realistic rather than aspirational.

Step 2: Collect and Structure Your Organization’s Own Data

This is the step every organization already has the raw material for, usually without realizing it. As covered in the earlier posts on agricultural logbooks and text-to-agentic-workflows, the core insight is that documentation written for the next human on the job — SOPs, case files, policy handbooks, historical decisions, exception logs — is exactly the raw material needed to train the next agent on the job.

Concretely, for the examples above, this means gathering: - Procedures and protocols: SOPs, policy handbooks, case-handling guidelines — the equivalent of the orchard’s cultivation manuals or a caseworker’s internal procedure guide. - Exceptions and edge cases: the messy, real-world deviations from the clean procedure — a citizen’s situation that doesn’t fit the standard eligibility category, a customer complaint that doesn’t match any FAQ entry. - Regulatory and legal texts: the actual legislation, regulation, or compliance framework the agent’s advice must never contradict — for a public-sector agent, this is non-negotiable ground truth, not optional context. - Historical case outcomes: past decisions and their justifications, ideally captured as observation-action-result triples — a citizen’s situation observed, a decision made, an outcome recorded — the same structural pattern covered in both the agricultural and text-to-action posts.

Once collected, this text needs to be converted from documents built for human reading into a structured, machine-usable format — typically JSON, mapping procedures into workflow graphs with explicit steps, conditions, and branches, exactly as described in the earlier post on converting text into agentic training data.

Step 3: Extract the Agentic Training Data

With structured data in hand, the next step is extracting the specific training artifacts each component of the architecture needs — this is where a generic “chatbot” becomes a genuine, specialized agent:

• RAG chunks and reranker pairs: semantically coherent sections of legislation, policy, or procedure, embedded for retrieval, paired with real query examples so a neural reranker learns to surface the correct clause, not just a similar-sounding one.
• Supervised fine-tuning (SFT) pairs: instruction-response examples that teach the model your organization’s specific terminology, tone, and reasoning style — a citizen’s question paired with the response a well-trained caseworker would actually give.
• Preference data (DPO): paired examples where one response is correct but not preferred — for instance, a technically accurate answer that fails to flag a citizen’s right to appeal, versus one that does — teaching the agent to favor the response that meets your organization’s standards, not just any factually correct one.
• Planning and task-decomposition data: multi-step processes (how a benefits application actually moves from intake to decision) encoded so the agent can decompose a broad goal into an ordered sequence of sub-tasks with real dependencies.
• Multi-hop reasoning chains: the “why” behind a decision — an eligibility conclusion linked back through the specific regulation, the specific facts of the case, and the specific prior precedent that justified it.
• Guardrail and critic training data: contrastive pairs of rejected and corrected outputs, each with an explicit rationale (“rejected: violates Article 9 of Regulation X”), which is exactly what trains a critic agent to catch violations rather than merely check spelling.

Step 4: Fine-Tune the Components — Not One Giant Model

The architecture that emerges from this data isn’t a single fine-tuned model — it’s several specialized components working together, each trained on the data extracted above: a memory layer (RAG plus reranker), a reasoning/planning layer, a fine-tuned critic enforcing your specific standards, and a set of guardrails. As demonstrated in the earlier post comparing a small, specialized model against a frontier foundation model, this approach can get remarkably close to frontier-level performance on your specific task, precisely because the small model isn’t being asked to be good at everything — only at the one thing your organization actually needs it to do.

Step 5: Deploy on Your Sovereign Infrastructure

With the components trained, deployment means wiring them together — typically using an orchestration framework built for cyclical, multi-step agent behavior (LangGraph and CrewAI are common choices) — and running the whole system on the infrastructure chosen in step one. The foundation model underneath (whether a large commercial model used only for the heaviest reasoning steps, or a fully local open-weight model) becomes a replaceable, commodity component, exactly as argued in the earlier post on the institutional moat — the actual value, and the actual sovereignty, lives in the architecture and data around it, not in which base model happens to sit at the center.

Step 6 (Optional but Recommended): Distill, Prune, and Quantize

If energy efficiency, battery-powered deployment, or running on genuinely modest hardware matters, the compression techniques covered in an earlier post become directly relevant here: distilling the fine-tuned components into smaller student models, quantizing weights down to 8-bit or even more aggressive precision (BitNet-style quantization at the extreme end), and pruning unnecessary connections. Beyond the obvious cost and energy savings, this step has a genuine security side benefit, also covered in that post: a compressed model tends to have a smoother, less exploitable decision boundary, making certain classes of adversarial manipulation measurably harder.

Step 7: Validate Against Bias and Other Failure Modes

This step is not optional, and it deserves to be treated as seriously as the model architecture itself — especially for anything citizen-facing or decision-adjacent. Concretely, this means:

• Fairness and bias testing across demographic groups. For a benefits-eligibility or policy-implementation agent, this is a legal requirement in most jurisdictions, not just good practice — the agent must be tested to confirm it doesn’t systematically disadvantage applicants by name, address, language, or any other proxy for a protected characteristic. This requires deliberately constructed test sets, not just spot-checking outputs that happen to come up.
• Factuality and hallucination checks, comparing every generated claim against the retrieved source of truth and flagging anything unsupported.
• Adversarial robustness testing, following the attack patterns covered in an earlier post — deliberately probing the system with edge cases, ambiguous phrasing, and adversarially crafted inputs to see where it breaks, before a real citizen or customer finds the same weakness by accident.
• Human-in-the-loop audit sampling. No validation suite catches everything; ongoing, randomized human review of a percentage of the agent’s actual decisions remains necessary, precisely to catch the “the model was right 49 times, will anyone still check the 50th” complacency risk covered in the earlier post on governance.

Step 8: Test Against Ethical and Moral Requirements

The final step, and arguably the one public-sector deployments can least afford to skip, is holding the system to an explicit ethical and legal standard rather than an implicit one:

• Write an explicit constitution. As covered in the earlier post on the institutional moat, this means a documented set of immutable principles — “never deny a benefit without citing the specific regulation and offering an appeal path,” “never make a final decision on X without human sign-off” — checked by a dedicated node in the pipeline before any output reaches an end user, decoupled from the task-execution logic itself.
• Build in the four-phase governance framework covered in the earlier governance post: validated training, tested robustness, a reflection mechanism that catches the system’s own mistakes, and genuine explainability — the ability to show, for any given decision, exactly which regulation, which fact, and which prior case justified it.
• Guarantee a human appeal path. For any public-sector or legally consequential use, the agent should support, never replace, the human decision-maker of record — flagging omissions a caseworker might have missed, proposing a first-pass recommendation for review, but never issuing a final, unappealable decision autonomously.
• Be transparent with the end user. A citizen or customer interacting with the system should know they’re talking to an AI agent, understand what it can and cannot decide, and know exactly how to reach a human when they need one.

The Takeaway

None of these eight steps is optional if the goal is genuine sovereignty rather than a thin wrapper around someone else’s model. But none of them requires a national government’s budget either — the same techniques covered across this site’s earlier posts on agriculture, institutional strategy, sovereignty, and model compression apply directly, whether the organization building the agent is a cherry orchard cooperative, a hospital, a corporate legal department, or a municipal benefits office. The common thread across every genuinely sovereign deployment is the same: your own data, your own infrastructure, your own explicit standards for what the system is and isn’t allowed to decide — with the foundation model reduced to exactly what it should be, a replaceable component at the center of an architecture your organization actually owns.

Key papers

• Wang et al. (2023), A Survey on Large Language Model Based Autonomous Agents — arXiv:2308.11432
• Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — arXiv:2005.11401
• European Commission (2024), EU Artificial Intelligence Act — eur-lex.europa.eu

Further reading on this site

• What Is an AI Agent? — the seven components this eight-step guide is assembling
• From Text to Action: Turning NLP Pipelines into Agentic Workflows — the deep dive on steps 2 and 3
• The Institutional Moat — why the architecture matters more than the model
• Why Europe Needs Its Own AI Models — the geopolitical case for step 1
• Governing the Autonomous Organization — the governance framework behind steps 7 and 8
• Bias Analysis in Practice — the step-by-step technical guide to carrying out the bias analysis required in step 7
• David vs. Goliath: Small Model vs. Frontier — the evidence behind step 4

Why This Matters Beyond Cost

Three separate motivations converge on the same set of techniques:

1. Energy and battery life. On-device and edge deployment — implanted medical devices, mobile phones, military hardware — simply cannot run a full-size frontier model. Every watt matters.
2. Local, sovereign deployment. As covered in an earlier post on this site, a specialized model fine-tuned on your own data and run on your own hardware can get remarkably close to frontier-model performance on a specific task — but only if that specialized model is actually small enough to run locally at reasonable cost. Compression is what makes that “David vs. Goliath” story physically possible.
3. Security. Somewhat counter-intuitively, a smaller, distilled or pruned model can be more robust to certain attacks, because compression removes exactly the kind of redundant, unused connections that adversarial attacks tend to exploit.

Let’s go through the three main techniques in turn.

1. Knowledge Distillation: Teaching a Small Model to Mimic a Big One

Knowledge distillation compresses and transfers what a large, expensive “teacher” model has learned into a smaller “student” model, while trying to preserve as much of the teacher’s competence as possible. The canonical example is DistilBERT (Sanh, Debut, Chaumond & Wolf, 2019): by distilling BERT-base during pre-training, the authors reduced model size by 40%, ran it 60% faster, while retaining 97% of BERT’s language understanding performance on the GLUE benchmark.

The training mechanics are worth understanding, because they explain why this works as well as it does. Rather than training the student purely to predict the correct label, distillation trains it on a combined loss:

total loss = (alpha * student loss) + ((1 - alpha) * distillation loss)

The distillation loss is computed using KL-divergence between the teacher’s and student’s softmax outputs — but crucially, using a softened temperature (t > 1) so the outputs aren’t just hard 0/1 labels but “soft labels” carrying the teacher’s relative confidence across all classes, not just the winning one. That soft signal is where most of the useful information actually lives: it tells the student not just what the right answer is, but how the teacher weighs the alternatives — an implicit signal a hard label alone can never carry. DistilBERT specifically combines this distillation loss with a masked-language-modeling loss and a cosine-embedding loss (to align the direction of the teacher’s and student’s hidden vectors), and initializes the student by taking every other layer directly from the teacher rather than starting from scratch.

Other notable examples in the same family include TinyBERT, which pushes distillation further for even smaller footprints. This is very much an active area of research — see the Nature coverage of ongoing model-compression work for a sense of how fast the field is still moving.

2. Quantization: Fewer Bits, Almost the Same Answer

A standard neural network stores its weights as 32-bit floating point numbers. Quantization asks a simple question: do you actually need all 32 bits? Often, no. Representing weights with 8-bit integers instead of 32-bit floats gives you a model that’s a quarter the memory footprint, with four times faster data transfer, and computation that’s often dramatically faster too, especially for integer arithmetic. Depending on the hardware, this alone can produce speedups in the range of 25% to 250%.

There are two main flavors:

• Post-Training Quantization (PTQ) — take an already-trained model and quantize its weights afterward.
• Quantization-Aware Training (QAT) — train the model while simulating the effects of quantization, so it learns to be robust to the reduced precision from the start.

The most striking recent example of QAT taken to its logical extreme is BitNet b1.58 (a 2024 paper), which restricts every weight to just three possible values: {-1, 0, 1} — technically 1.58 bits per weight (since log₂(3) ≈ 1.585). Remarkably, this achieves performance comparable to a standard LLaMA model of equivalent size, while requiring dramatically less energy — the paper reports roughly 1/71.4 of the energy consumption of the equivalent full-precision model, and it’s efficient enough to run on ordinary CPUs rather than requiring specialized accelerator hardware at all.

The obvious challenges are quantization error and numerical overflow/underflow, which is exactly why QAT exists — training the model to be well-behaved under low precision from the start, rather than hoping a post-hoc rounding step doesn’t break anything important.

3. Pruning: Deleting What the Model Doesn’t Need

Where quantization makes every number smaller, pruning removes numbers entirely — introducing zeros (sparsity) into weight matrices so that whole connections, or in some cases entire neurons, simply stop existing. Sparse matrices are cheaper to store and, on the right hardware, cheaper to compute with, since operations on zero-valued weights can often be skipped outright.

Pruning and quantization aren’t mutually exclusive — a real deployment pipeline typically combines several of these techniques at once: distill a smaller student model, quantize its weights, and prune unnecessary connections, often alongside Low-Rank Adaptation (LoRA) for efficient fine-tuning (and its quantized cousin, QLoRA) so that adapting the compressed model to a specific domain remains cheap too.

Why Smaller Models Can Be More Secure, Not Less

Here’s the connection that doesn’t get made often enough: compression isn’t just a cost-and-energy story — it’s also a security story. Adversarial attacks on neural networks frequently exploit unused or redundant capacity: connections and parameters that don’t contribute meaningfully to the model’s actual task, but that an attacker can manipulate to produce a targeted misclassification or an unwanted output while leaving the input looking normal to a human.

Pruning removes exactly this kind of slack. If a connection has been eliminated because it wasn’t contributing to genuine task performance, it’s no longer available as an attack surface either — the model’s decision boundary gets forced to depend only on the features that actually matter, rather than on incidental correlations an attacker can nudge. Knowledge distillation does something related: because the student model is trained to reproduce the teacher’s softened, generalized decision boundary rather than to memorize the training data directly, it tends to inherit a smoother, less exploitable response surface. This specific idea has its own name in the adversarial-robustness literature — defensive distillation, and a closely related family of techniques called feature squeezing, which reduces the input’s degrees of freedom in a way that echoes exactly what pruning does to a model’s internal degrees of freedom.

None of this means a compressed model is immune to adversarial attacks — it isn’t, and dedicated adversarial training remains necessary on top of compression, not instead of it. But it does mean that shrinking a model for cost and energy reasons is not a security trade-off you’re forced to accept. Done well, it can move in the opposite direction.

For a full treatment of what adversarial attacks on images and text actually look like — and the concrete defenses, beyond compression, that address each type — see the earlier post in this series: Adversarial Attacks: How a Few Invisible Pixels or One Swapped Word Can Fool an AI.

Bringing It Together

Distillation, quantization, and pruning are usually presented as separate cost-optimization tricks, but they share a common thread: each one asks the model to do more with less — fewer parameters, fewer bits, fewer connections — while preserving the behavior that actually matters. That’s precisely the mechanism behind the “small specialized model nearly matching a frontier model” result covered in an earlier post on this site: a model doesn’t need the full capacity of a generalist if it’s only being asked to do one thing well, and compression is the concrete engineering path that turns “doesn’t need the capacity” into “doesn’t have the capacity, and runs on a laptop instead of a data center.” Lower training and inference cost, genuine on-device and edge deployment, and — as a welcome side effect — a smaller, harder-to-exploit attack surface. That combination is why this remains one of the most active areas of research in the field.

Key papers

• Hinton et al. (2015), Distilling the Knowledge in a Neural Network — arXiv:1503.02531
• Sanh et al. (2019), DistilBERT, a Distilled Version of BERT — arXiv:1910.01108
• Ma et al. (2024), The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (BitNet b1.58) — arXiv:2402.17764
• Hu et al. (2022), LoRA: Low-Rank Adaptation of Large Language Models — arXiv:2106.09685

Further reading on this site

• Adversarial Attacks: Invisible Pixels and Swapped Words — the full treatment of attack types and defenses that compression partially addresses
• Why You Can’t Fine-Tune or Distill Without XAI — why distillation specifically makes explainability a requirement, not a nice-to-have
• David vs. Goliath: Small Model vs. Frontier — compression and specialization together: what a small, task-focused model can actually do
• Why Europe Needs Its Own AI Models — why edge and on-device deployment matters geopolitically, not just for cost
• Why NLP Finally Works — the transformer architecture whose weights are being compressed here

The Knowledge Was Always Meant to Be Passed On

Modern agriculture — fruit farming especially — is losing something it can’t easily get back: an aging workforce is retiring, and with it goes decades of empirical knowledge about micro-climates, pest pressure, and crop cycles that was never formalized anywhere except in handwritten logbooks and oral tradition. That knowledge was always meant to transfer to a successor. The logbook was the transfer mechanism.

At the same time, farmers now face a genuinely hard balancing act: mounting pressure toward organic methods and reduced chemical use, set against the economic reality of needing to protect yield and quality. A general-purpose chatbot can’t responsibly help here — a wrong pesticide recommendation can mean crop failure, environmental damage, or the loss of organic certification, and generic models trained on the open internet have no access to what actually happened, and worked, in this orchard, under these conditions.

The knowledge that would actually help already exists. It’s just sitting in a form built for a human reader, not a machine.

The Same Data, Read a Different Way

Here’s the reframe that makes this possible: every entry in a farming logbook is already, structurally, an observation → action → consequence record. Cloudy, no wind, post-bloom (observation) → applied ureum, borium, zinc (action) → maximal leaf absorption without evaporation (intended consequence). That’s not a coincidence — it’s how anyone documents a decision they want a successor to be able to trust and repeat. And an observation-action-consequence triple is, not coincidentally, exactly the data structure reinforcement learning is built around: state, action, reward, next state. The format a farmer used to teach a human apprentice turns out to be the format needed to teach an AI agent.

The practical pipeline looks like this:

1. Digitize. OCR and multimodal models convert handwritten pages — dates, dosages, weather, outcomes, all in a farmer’s own shorthand — into machine-readable text.
2. Structure into JSON. Free-form, tabular handwriting gets serialized into a consistent schema: date, weather conditions, substance applied, quantity, timing, and (where recorded) the outcome. This structured layer is what every downstream training pipeline builds on.
3. Extract component-specific training data. This is the part worth walking through in detail, because a single logbook entry can be turned into several different kinds of training signal, each feeding a different part of an agentic architecture:

One Logbook Entry, Five Different Kinds of Training Data

Take that same April entry and see what it becomes for each component of an agent:

For retrieval (RAG): the entry becomes a semantically chunked fact the agent can look up later — “on this orchard, copper oxychloride at this dose was applied under these specific weather conditions” — embedded and stored so a future query can retrieve it directly, cited to its source.

For reranking: paired with the question a farmer might actually ask (“how do I treat fruit rot risk during a wet cold spring”), the same entry becomes a query-answer pair used to train a reranker to recognize this historical intervention as the relevant one, rather than a generically similar but wrong result — solving exactly the “lost in the middle” problem where an agent drowns in twenty broadly similar retrieved passages and misses the one that actually matters.

For instruction-following (SFT): the raw entry gets rewritten as a structured instruction-response pair — “It’s late May, light rain, and the orchard is at risk of fruit rot. What’s the historical intervention?” → “Historically, Signum (0.25 kg) combined with a leaf mix was applied to prevent fruit rot during light rain.” This is what teaches a model to adopt the terminology, tone, and reasoning style of an actual agronomist rather than a generic assistant.

For preference learning (DPO): the same scenario, paired with a rejected alternative — a heavy chemical spray that would technically work but violates current organic certification — teaches the model that not every factually correct answer is the preferred one. This is where an agent learns to default toward the sustainable, compliant option rather than whatever seems most immediately effective.

For planning and multi-hop reasoning: a multi-week sequence from the logbook — evaluate blossom readiness, order bees, place hives on a specific date, remove hives before spraying three weeks later — becomes a task-decomposition example: how a single seasonal goal (“manage pollination”) breaks down into an ordered sequence of dependent sub-tasks, each with its own timing constraint.

One page of handwriting, five different downstream capabilities — and none of it required the original farmer to think about AI at all. They were just doing what farmers have always done: writing down what happened so someone else could learn from it.

Why the Oldest Text Might Be the Most Valuable

Here’s a genuinely counter-intuitive point buried in this approach: the most useful historical documents for building a modern, organic-focused agricultural agent might be agricultural texts from 1850 to 1930 — before synthetic pesticides existed at all. These archives, digitized in collections like Cornell’s Core Historical Literature of Agriculture and the USDA’s National Agricultural Library, describe ecological pest management and soil techniques that were simply forgotten once cheap synthetic chemicals took over for most of the 20th century. As farming now transitions back toward organic methods, that “obsolete” pre-chemical knowledge is suddenly directly relevant again — and it exists in exactly the same form as the personal logbook: text written by one generation of practitioners for the benefit of the next, that happened to skip a generation or two before finding its intended reader.

The Guardrails Have to Come From the Same Place

None of this works safely without an equally serious investment in what not to do, and — fittingly — that data comes from the same kind of source. A “critic agent” checking a draft recommendation needs concrete negative examples: a 2020 logbook entry mentioning a chemical (say, Calypso) that was legal then but is now restricted, paired with the corrected, currently compliant alternative, and — crucially — a stated rationale (“rejected: violates EU Organic Regulation 2018/848, Article 9”). That rationale is what teaches a critic model not just that something is wrong, but why, in a form it can generalize from. Historical logbooks and current legislation, cross-referenced against each other, become the raw material for the safety layer just as much as they do for the advice-giving layer.

The Bigger Pattern

Step back from the orchard specifically and the pattern generalizes cleanly, and it connects directly to something this site has covered before: any organization’s internal manuals, standard operating procedures, incident logs, and handover notes were always written with an implicit reader in mind — the next person doing the job. That framing has quietly been the whole point of workplace documentation for as long as workplaces have existed. What’s changed is who that “next person” can now be. The same discipline that made a good logbook entry useful to a human successor — being concrete about conditions, actions, and outcomes, rather than vague — is precisely what makes it usable as training data for an agent. Good documentation was never really about compliance or record-keeping for its own sake. It was about making tacit expertise transferable. It just turns out the recipient doesn’t have to be human anymore.

The Takeaway

A logbook entry from 2013, or a farming manual from 1890, was never written with a language model in mind — and that’s exactly why it works so well as training data. It’s honest, concrete, and grounded in what actually happened, because it was written for someone who’d need to trust and repeat it. Turning that text into structured retrieval data, reranker pairs, instruction examples, preference pairs, and planning templates isn’t a reinterpretation of what the knowledge was for — it’s the same transfer of expertise from one practitioner to the next, just with a new kind of successor on the receiving end.

Key papers

• Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — arXiv:2005.11401
• Rafailov et al. (2023), Direct Preference Optimization — arXiv:2305.18290
• Yao et al. (2022), ReAct: Synergizing Reasoning and Acting in Language Models — arXiv:2210.03629

Further reading on this site

• From Text to Action — the general pipeline this agricultural example instantiates
• What Is an AI Agent? — what the trained agent this data is meant to power actually is
• David vs. Goliath: Small Model vs. Frontier — what a domain-specialized model built on this kind of data can achieve
• The Institutional Moat — why proprietary training data like this is the real competitive advantage

The Problem That Never Went Away: Language Doesn’t Follow Rules Cleanly

Natural language is ambiguous at every level. “I saw the man on the hill with a telescope” has at least five valid readings depending on who has the telescope and who’s on the hill. Words like “run” carry a dozen related senses (polysemy); words like “bank” carry two entirely unrelated ones (homonymy). Negation scope, pronoun resolution, irony, sarcasm, discourse structure — each adds another dimension of ambiguity that has to be resolved using context, not just grammar.

Underneath all of this sits an even harder mathematical fact: language follows Zipf’s Law. A small number of words and constructions are extremely common, and then there’s an enormous, unpredictable “long tail” — idioms, rare domain terms, novel phrasings — that occurs too rarely to ever be fully enumerated. Any system built on a fixed set of rules will inevitably run into cases the rules don’t cover, no matter how many rules you write.

That single fact — the long tail is unsolvable by fixed rules — explains almost the entire fifty-year history of the field.

Generation 1: Grammars and Logic (the Rule-Based Era)

The earliest computational approach to language, tracing back through Chomsky’s generative grammars to Aristotle’s original grammatical categories, tried to formalize language as a system of rules: part-of-speech categories, phrase structure, syntactic derivations. This gives you precision — a grammar either accepts a sentence or it doesn’t — but it’s fundamentally brittle. Hand-written rules can’t account for the long tail, don’t adapt to new vocabulary (a neologism like “Zoom” as a verb didn’t exist yesterday and does today), and require a linguist to manually extend the system every time real-world text breaks it.

Generation 2: Probabilistic and Corpus-Based Models

The next generation added likelihoods on top of structure: instead of a rule either firing or not, assign it a probability learned from a corpus of real text. This was a genuine step forward — it let systems handle ambiguity gracefully instead of failing outright, and it learned patterns directly from annotated data rather than requiring them to be hand-coded. But probabilistic models still largely operated over fixed, engineered features (n-grams, hand-built lexical resources like WordNet encoding synonymy and hyponymy by hand). They were more robust than pure grammars, but they still weren’t learning the deep structure of language — they were learning statistics about a structure someone else had defined.

Generation 3: Neural Networks Learn Sequential Dependencies (RNN, LSTM)

Early artificial neural networks, and then recurrent neural networks and LSTMs, represented a real architectural leap: instead of hand-built features, the network learns sequential dependencies directly from data. An RNN passes a hidden state forward through a sentence, meaning it can, in principle, use “what came before” to interpret “what comes now” — exactly the kind of context resolution that ambiguity requires. LSTMs, introduced in 1997, added a gated memory mechanism specifically to help the network retain information across longer gaps (“I grew up in France … I speak fluent French” requires connecting words that are far apart).

Here’s where the wall appeared again, though, and it was a serious one. RNNs and LSTMs process a sentence strictly sequentially — token by token, each step depending on the previous one’s output. That has two consequences that turned out to be fatal at scale: training cannot be parallelized (you can’t compute step 10 before step 9 finishes), and the sequential chain of multiplications causes gradients to either vanish or explode as sentences get longer, making genuinely long-range dependencies hard to learn no matter how clever the gating mechanism. On top of that, the entire meaning of an input sequence had to be compressed into a single final hidden-state vector — often just a handful of dimensions — before being handed to a decoder. Too many relations, too small a bottleneck.

By the mid-2010s, the field had a real diagnosis: RNNs and LSTMs were data-hungry, unstable, computationally expensive to train, and structurally incapable of running in parallel. There was room — need, really — for something fundamentally different.

Generation 4: Transformers — Solving the Bottleneck and the Sequential Constraint at Once

The 2017 paper “Attention Is All You Need” replaced recurrence entirely with self-attention: instead of passing information step-by-step through a chain, every token in a sequence is compared directly against every other token, all at once, via matrix multiplication. That single change fixes both of the RNN/LSTM problems simultaneously. Because there’s no sequential dependency between tokens during training, the entire computation parallelizes — and matrix multiplication happens to be exactly what modern GPU and TPU hardware is built to do at enormous speed. And because attention is computed between every pair of tokens directly, there’s no more single bottleneck vector trying to hold an entire sentence’s meaning; the model can look directly at any earlier or later token when it needs to.

But the more interesting part, for anyone who cares about why this actually works for language specifically, is what happens when you stack multiple layers of self-attention on top of each other. Each layer takes the output of the layer before it and refines it further — and empirically, the layers specialize into something that looks remarkably like the classical linguistic pipeline computational linguists had been hand-building for decades: the lower layers pick up on relatively obvious relations like punctuation and morphology, the middle layers start capturing syntax and basic semantic relationships, and the upper layers handle the genuinely hard stuff — resolving pronouns and co-references, tracking long-distance logical relationships, the kind of discourse-level structure that used to require dedicated hand-built modules.

This is the point worth sitting with: the classical NLP pipeline — tokenization, morphology, syntax, semantics, discourse — doesn’t disappear in a Transformer. It re-emerges automatically, layer by layer, purely as a consequence of training on enough text, without anyone hand-coding a single grammatical rule. Fifty years of linguistics tried to specify that pipeline explicitly, one layer at a time. Transformers derive the same structure implicitly, from data, and — critically — they don’t get stuck the way every previous generation did on the long tail, because the model isn’t relying on any single fixed rule; it’s building a probabilistic representation flexible enough to generalize to inputs it’s never seen.

Was It Luck, or Did They Know Exactly What They Were Doing?

Here’s a genuinely interesting historical question that gets asked about the eight authors of “Attention Is All You Need”: was this breakthrough a lucky architectural accident, or did the people who built it actually understand the linguistics they were encoding?

The honest answer leans toward the latter. Several of the paper’s authors had real, substantial backgrounds in computational linguistics rather than pure machine learning — most notably Jakob Uszkoreit, whose father, Hans Uszkoreit, is a well-known computational linguist. Growing up around that field isn’t a minor biographical detail; it’s a plausible reason the architecture was built with an intuition for how meaning actually layers in language — morphology, then syntax, then semantics, then discourse — rather than being purely an engineering exercise in scaling matrix multiplications. It’s a rare case where the “chance versus deliberate plan” question doesn’t have an obvious answer, and where the deliberate-plan interpretation is genuinely well supported by who was in the room.

Why “Finally Works” Is the Right Phrase

Put the whole history side by side and the pattern is unmistakable. Grammars gave precision but no robustness to the long tail. Statistical models gave robustness but were still built on fixed, hand-engineered features. RNNs and LSTMs finally learned sequential structure directly from data, but couldn’t scale — sequential processing capped both their training speed and how far back they could actually “remember.” Transformers are the first architecture to combine genuine data-driven learning of linguistic structure at every level — from morphology up through discourse — with a computation that parallelizes cleanly enough to be trained at a scale no earlier architecture could reach. That combination — not any single trick — is why, after fifty years of partial solutions, natural language processing finally works.

Course material referenced: Advanced NLP lecture series, Department of Advanced Computing Sciences.

Key papers

• Vaswani et al. (2017), Attention Is All You Need — arXiv:1706.03762
• Hochreiter & Schmidhuber (1997), Long Short-Term Memory — Neural Computation
• Devlin et al. (2018), BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — arXiv:1810.04805

Further reading on this site

• What Is an AI Agent? — the transformer-based language model as the missing engine that makes agentic systems work
• David vs. Goliath: Small Model vs. Frontier — what happens when you specialize one of these architectures on a single task
• Shrinking the Giant: Pruning, Quantization, and Distillation — how to compress what transformers learned into something that runs on modest hardware
• Adversarial Attacks: Invisible Pixels and Swapped Words — the security vulnerabilities that emerge from how these models draw their decision boundaries

The Problem: AI Agents Are Bad at Following Instructions

It’s tempting to assume that if a language model can write an essay or debug code, it can obviously follow a company’s standard operating procedure. Recent benchmarking says otherwise. One 2025 study deploying AI agents to execute real company SOPs found the best frontier model achieved only a 30.3% success rate, with open-source models falling under 8%. Separate work testing frontier models on multi-hop reasoning tasks in a clinical records setting found accuracies below 11%. The common thread: unstructured procedural text — full of conditional branches, loops, and ambiguous phrasing (“if the name is already in the database… otherwise, start a registration process and execute steps 1 to 10”) — is exactly where general-purpose models stumble, because they’re reasoning over free-form text rather than a structured notion of where they are in a process and what’s actually still valid to do next.

The Approach: Turn Manuals Into Training Signal, Not Just Context

The core idea is to stop treating a procedure as text to reason over at inference time, and instead treat it as a source of structured, step-level training data the model can actually learn from. The pipeline runs in five stages:

1. Extract the workflow as a graph. An LLM-based extraction loop reads a procedural document and recovers it as a directed graph — actions, decision gateways, branch conditions — with a self-checking refinement loop (a structural checker verifying every node is reachable and a path exists from start to end, plus a semantic checker comparing the extracted graph back against the original text for missed steps).
2. Turn the graph into a notion of progress. Every valid path through the graph, enumerated deterministically (via breadth- and depth-first search), becomes a sequence of states: what’s been done so far, which conditions held, and what’s validly available next.
3. Generate both positive and near-miss training signals. Alongside the correct next action at each state, the pipeline deliberately generates wrong next actions — a skipped step, two steps done in the wrong order, a repeated action, a wrong starting point — each labeled with exactly why it’s wrong.
4. Train a small, specialized model on this data — a Process Reward Model that scores candidate next actions given the current state, layered as a LoRA adapter on top of a small open-weight base model (Llama 3.1 8B).
5. Deploy an agent that combines the small trained model with the extracted graph itself as a real-time filter: when the model is uncertain, it narrows its candidate actions to only what the graph says is actually valid at that point — at no extra training cost.

The Results: Each Layer Earns Its Keep

The step-by-step improvement is the clearest evidence that this isn’t one clever trick but a genuinely compounding architecture:

• A bare, ungrounded small model invents actions that don’t exist in the procedure roughly 35% of the time, completes only 10.2% of procedures end-to-end, and produces valid next-steps just 56% of the time.
• Simply grounding it in the extracted action list (no training required) already lifts completion to 36.7% and valid steps to 81%, and — critically — eliminates hallucinated actions entirely, since the model can now only pick from what’s actually real.
• Adding the trained Process Reward Model, blended with the base model’s own judgment, pushes completion to 51% and valid steps to 87%.
• Adding the graph as a live safety net for uncertain moments brings completion to 55.1% and valid steps to 88.7% — with almost no additional cost, since the graph was already extracted in step one.

And the punchline: benchmarked against a frontier foundation model (also given the extracted action list, so it’s not competing at a disadvantage) — the frontier model scored 89.9% on valid steps and 61.2% on completion, versus the small system’s 88.7% and 55.1%. A gap of roughly one and six points, against a model estimated to be 625 times larger.

One more finding worth flagging: when the training data itself was deduplicated and rebalanced — cutting the dataset by 43% but removing redundant, near-identical examples and correcting a skew toward “correct” labels — the resulting model performed better on some configurations than the model trained on the larger, messier dataset. Quality beat volume, not just for the base architecture, but for the training data feeding it.

Why This Result Makes Complete Sense

It’s tempting to read a result like this as “small models are secretly as good as huge ones” — that’s not the right lesson, and it’s worth being precise about why.

A frontier foundation model is a generalist. It has to be simultaneously good at writing poetry, debugging Rust, explaining quantum mechanics, translating Portuguese, and — only as one of thousands of competencies — following a company’s specific loan-approval procedure. Its capacity is spread across an enormous space of possible tasks it might be asked to do. That’s precisely why it’s a 5-trillion-parameter-class system in the first place: breadth is expensive.

A specialized model trained specifically on step-level, state-aware procedural data doesn’t need any of that breadth. It needs to answer exactly one narrower question extremely well: given where we are in this specific process, what’s the valid next step? All of its (comparatively tiny) capacity gets spent on that one competency, reinforced by thousands of concrete examples of what’s correct and — just as importantly — labeled examples of what’s almost correct but wrong in a specific, named way. A generalist model has never seen that kind of dense, structured, negative-example-rich training signal for your specific procedures, because it doesn’t exist publicly — it has to be built from an organization’s own manuals.

This is the same logic that shows up whenever a narrow specialist competes with a generalist: a chess engine that only plays chess beats a human generalist at chess, not because it’s more intelligent overall, but because it isn’t spending any capacity on anything else. The 625x parameter gap doesn’t disappear — it’s just mostly irrelevant to this specific, narrow task, because the small model was never asked to be good at everything else the large model also has to be good at.

Why This Matters Practically

The practical implications extend past the raw numbers:

• It runs locally, on a single consumer-grade GPU — no data leaves the building, which matters directly for the kind of institutional and data sovereignty concerns covered earlier on this site.
• No proprietary procedural data has to be sent to an external API provider to get this level of performance — the specialization happens entirely on infrastructure the organization controls.
• The cost and energy footprint drop by orders of magnitude relative to querying a frontier model repeatedly, without giving up much of the task-specific performance that matters.
• The bottleneck that remains is extraction quality, not model size — when the underlying workflow graph is perfect (gold-standard), the same small model completes 98% of procedures with 99.6% valid steps. The gap to frontier-model performance in real deployment comes almost entirely from imperfect automatic extraction of the workflow graph from messy source text, not from the small model’s reasoning capacity itself.

The Takeaway

A handful of percentage points separating an 8-billion-parameter specialist from a multi-trillion-parameter generalist, on a task that actually matters to an organization’s operations, is not a coincidence and not a trick — it’s the direct, predictable result of trading breadth for depth. Foundation models are built to be good at almost everything; a fine-tuned small model built on an organization’s own procedural data only has to be good at one thing. When that one thing is narrow enough and the training data is good enough, the size gap stops being the deciding factor. That’s the entire argument for building your own specialized agentic components on top of a commodity foundation model, rather than renting the whole capability from someone else’s much bigger, much more expensive, much less controllable system.

Key papers

• Hu et al. (2022), LoRA: Low-Rank Adaptation of Large Language Models — arXiv:2106.09685
• Sanh et al. (2019), DistilBERT, a Distilled Version of BERT — arXiv:1910.01108
• Ouyang et al. (2022), Training Language Models to Follow Instructions with Human Feedback — arXiv:2203.02155

Further reading on this site

• From Text to Action — how to build the training data pipeline that produces a model like this
• From Logbooks to Agent Training Data — a concrete domain example of building exactly this kind of specialized training signal
• Shrinking the Giant: Pruning, Quantization, and Distillation — the complementary compression techniques that make small models even smaller and cheaper
• Why Europe Needs Its Own AI Models — why running this model locally on sovereign infrastructure matters
• The Institutional Moat — the strategic case for building specialized rather than renting general

The Kissinger Warning, Applied to the Classroom

Before getting to the exciting use cases, it’s worth sitting with an uncomfortable observation from Henry Kissinger, writing about AI and chess: “AI is likely to win any game assigned to it. But for our purposes as humans, the games are not only about winning; they are about thinking… By treating a mathematical process as if it were a thought process, and either trying to mimic that process ourselves or merely accepting the results, we are in danger of losing the capacity that has been the essence of human cognition.”

Swap “games” for “assignments” and the warning applies directly to education. The purpose of education was never just passing exams — it’s learning to think. Yet assessment, historically, has often rewarded memorization over reasoning. With models like GPT now able to produce a polished essay in seconds, that mismatch has become impossible to ignore. AI can revolutionize education by acting as a scalable, personalized teaching assistant — but only if institutions redesign how they teach and assess around it, rather than bolting AI onto an assessment model built for a pre-AI world.

What Generative AI Actually Does Well in the Classroom

Set aside the risks for a moment — the concrete use cases are genuinely compelling, and most have already been tested in real courses.

Socratic dialogues, generated on demand. The Socratic method — probing questions that challenge assumptions and stimulate reasoning rather than handing over answers — is exactly the kind of interaction AI can produce at scale. Ask a model to generate five Socratic dialogues on a topic (what makes an approach powerful, where it breaks down, what the societal risks are) and you get ready-to-use classroom material that shifts the room from lecture to dialogue, in minutes rather than hours of prep.

A teaching assistant that works 24/7. Students hit conceptual walls at 2 a.m. the night before an exam, not during office hours. An AI tutor with infinite patience — willing to re-explain a difficult concept as many times and in as many different ways as a student needs — fills a gap no human teaching staff can realistically cover.

Exercises, readers, and lecture material, generated at scale. One concrete example from this material: a full 240-page reader — chapters, exercises, answers, and coding assignments — generated from 1,300 lecture slides in a few evenings, chapter by chapter. That’s not a marginal efficiency gain; it’s work that would otherwise take weeks, freeing an educator’s time for the parts of teaching that actually require a human — deeper interaction, mentoring, judgment calls about where a specific student is stuck.

Personalized explanation and adaptive content. The same underlying material can be re-explained at different levels of complexity depending on what a specific student needs — something a single lecture, delivered once to a room of 200, structurally cannot do.

Immediate feedback on your own teaching. Educators can use the same tools to review their own lecture material and get concrete feedback on clarity and structure — a second pair of eyes available on demand.

Automated conversational assessment. Early research (including ongoing work at Maastricht University’s DACS group) is exploring conversational agents that combine a teacher’s rubric, the content document, and a dialogue with the student to produce assessment recommendations — not to replace the teacher’s judgment, but to structure and speed up a process that’s currently manual and slow.

The Arms Race Nobody Wanted

Here’s where the honest picture gets more complicated. The moment AI got good enough to write a passable essay, an entire counter-industry appeared. Tools now exist specifically to “humanize” AI-generated text so it evades AI-detection software — and in a strange twist, students who write their own original work are now running it through the same detectors first, worried their authentic writing might get flagged as AI-generated. Teachers use AI-detection tools to catch AI-written submissions; students use humanizing tools to defeat those detectors; some students run their own real writing through detectors defensively. Everyone is now writing partly for the detector, not for the reader.

This isn’t entirely new — “homework machines” predate ChatGPT by decades, and every generation of technology has triggered similar panic (the same fear was raised about Google two decades ago: “Is Google Making Us Stupid?”). But generative AI’s fluency makes the cat-and-mouse dynamic sharper and faster-moving than anything before it, and it’s producing genuinely unhappy students and frustrated educators on both sides of the detection arms race.

Why the Underlying Technology Still Needs Guardrails

It’s worth being precise about what these models are — and aren’t — actually doing, because it explains both their usefulness and their failure modes in an educational setting. A model like GPT is fundamentally a next-token predictor: given a sequence of words, it estimates the probability of what comes next, generating text autoregressively, one token at a time. It has no built-in factuality check, no persistent memory across sessions unless explicitly engineered, and no real-world grounding for the words it produces — critiques often summarized as “stochastic parrot”: fluent language production without genuine understanding.

That has direct educational consequences: the longer the generated text, the higher the odds of factual drift or outright hallucination — a serious problem when the content is a formula, a citation, a historical date, or a piece of code a student will trust. The mitigation techniques that matter here for an education setting are the same ones used in any serious deployment: retrieval-augmented generation to ground answers in actual course material rather than the model’s general training data, knowledge-graph-based context injection for structured facts, and simply instructing the system to say “I don’t know” rather than confidently inventing an answer when confidence is low. None of this is optional polish — it’s the difference between a tutor that’s occasionally wrong in a way students can catch, and one that’s confidently wrong in a way they can’t.

The Real Redesign Question

The most productive framing isn’t “should we ban this” or “should we embrace this uncritically” — it’s “what does assessment need to become now that this exists.” A few concrete directions worth taking seriously:

• Redesign assignments around understanding, not output. Have students use GenAI to critique and improve their own work, understanding why something can be improved — turning the AI into a teammate rather than a shortcut.
• Consider flipping the classroom. If lecturing information is a solved problem AI can do reasonably well, live class time can shift toward Socratic dialogue and applied reasoning — though this requires a real shift in what’s expected of students, not just of instructors.
• Keep genuinely AI-free assessment where it matters. Some final assessments should remain “do it on your own,” without any AI tooling, specifically to verify the reasoning skill the AI-assisted work was meant to build toward.
• Avoid “AI-friendly assignments” — tasks that don’t require critical analysis in the first place are exactly the ones AI will complete indistinguishably from a human, telling you nothing about what the student actually learned.

There’s also a quieter risk worth naming: if an AI grading or tutoring assistant is right 49 times in a row, will an educator still carefully check the 50th? Automation complacency is a real failure mode, not a hypothetical one, and it argues for keeping a human genuinely in the loop rather than rubber-stamping AI output because it’s usually been reliable.

Weighing It Honestly

The case for: 24/7 access to explanation, genuinely personalized pacing, scaffolding for difficult topics, efficient tutoring support at a scale no institution could staff with humans alone, reduced barriers for students in under-resourced situations, and language support for non-native speakers.

The case for caution: over-reliance that erodes the exact skill education is meant to build, the risk of confidently wrong information reaching students who can’t yet tell the difference, unequal access if tools are costly, and the very real possibility that assignments get outsourced rather than understood.

The Takeaway

The technology genuinely works now, after decades of research — that part isn’t in question. What’s still being worked out, in real time, in real classrooms, is how educators shift from being information providers to facilitators of critical thinking, and how assessment shifts from rewarding memorization to rewarding reasoning and application. Used well, AI in education looks like a tireless, patient, always-available teaching assistant that frees human educators for the parts of teaching machines still can’t do. Used carelessly, it looks like an arms race between detection tools and humanizing tools that leaves everyone writing for the algorithm instead of for understanding. The technology doesn’t decide which of those futures we get — the redesign of assessment and pedagogy around it does.

Based on “Exploring the Potential of AI Tutors in Higher Education” (Prof. dr. ir. Jan Scholtes, UM Education Day, June 2025).

Key papers

• Kasneci et al. (2023), ChatGPT for Good? On Opportunities and Challenges of LLMs for Education — Learning and Individual Differences
• Bender et al. (2021), On the Dangers of Stochastic Parrots — ACM FAccT 2021
• Mollick & Mollick (2023), Assigning AI: Seven Approaches for Students, With Prompts — SSRN

Further reading on this site

• What Is an AI Agent? — the architecture behind the AI tutors and assessment agents discussed here
• Why NLP Finally Works — the transformer architecture that made generative AI in education possible
• Adversarial Attacks: Invisible Pixels and Swapped Words — why AI systems need guardrails, including in educational settings
• From Text to Action — how course materials and corrections can become training data for a specialized educational agent

The Governance Framework Nobody Wrote Down (Until Now)

A useful way to think about governing an AI agent is to map its lifecycle onto four phases, each with its own governance question — and each with a corresponding evaluation standard, such as Stanford’s HELM benchmark suite, that gives you something concrete to measure rather than a vague sense of trust.

1. Training — is the foundation sound? This is the pre-training and instruction-tuning phase where the model learns patterns, language, and baseline tasks. The governance question here is simple but foundational: does this system have adequate baseline accuracy and knowledge quality across the scenarios it will actually face? If the foundation is weak, nothing built on top of it will be trustworthy, no matter how sophisticated the oversight layers are.

2. Validation — does it behave consistently under pressure? Testing an agent in simulated environments — can it reliably call APIs and execute code without breaking? — surfaces two governance-critical properties: robustness (does small variation in input cause large variation in behavior?) and calibration (does the system actually know how confident it should be?). A model that’s simultaneously wrong and confident is far more dangerous to govern than one that’s wrong and says so.

3. Reflection — does it catch its own mistakes? Agentic frameworks like ReAct and Reflexion give a system the ability to look at the outcome of its own action — an error message from a tool, an unexpected result — think about it, and correct its own plan. From a governance standpoint, this is where you assess adaptivity: does the agent keep following a logical, defensible chain of steps, or does it derail under uncertainty? This is the layer that catches problems before a human ever has to.

4. Explanation — can it show its work? This is Explainable AI (XAI): why did the agent choose action A instead of action B? A governed agent needs to expose its internal decision tree, the weight it gave to competing considerations, or at minimum a clear log — to the human supervisor who remains accountable for the outcome. This is also where fairness and bias reduction get enforced in practice, because you cannot audit what you cannot see. Without this layer, “the AI decided” is not an answer — it’s an evasion.

Together, these four phases aren’t just an engineering pipeline. They’re a governance framework: each phase answers a specific question a board, a regulator, or an auditor will eventually ask, and each has a measurable standard behind it rather than a hand-wave.

What This Looks Like in Practice: The US Navy

Abstract frameworks are easier to trust when you can point to them working under real operational pressure. The US Navy’s recent agentic AI deployments are instructive precisely because the stakes — safety, cost, mission readiness — leave no room for governance to be an afterthought.

MyNavy HR: eliminating bureaucracy, not just automating it. Internal administration was one of the biggest sources of wasted time and frustration for sailors, contributing directly to lower retention. The Navy’s answer was to place AI agents as an invisible translator between the sailor and the Navy’s clunky ERP systems — a sailor submits an HR, pay, or leave change through a radically simple interface, and the agent autonomously updates the backend systems. The result: hundreds of thousands of saved labor-hours annually, a reduced HR headcount, and operational crews freed from paperwork. The governance lesson here is that the sailor never has to understand or trust the AI directly — they trust that the outcome is correct, because the governance sits one layer back, in how the agent’s actions are validated against HR policy.

Condition-Based Maintenance Plus (CBM+): sovereignty as a governance requirement, not a preference. The Navy is moving away from reactive or purely calendar-based maintenance on ships and submarines, using what they call Sovereign Edge AI — models that run physically and locally on the ship, independent of a satellite connection. Sensors feed a local model; when it detects an abnormal pattern, the agent doesn’t just generate a report — it autonomously orders the needed spare part and schedules the repair for the next port call. The result: massive savings from minimizing “Awaiting Parts” downtime — a billion-dollar ship no longer sits idle because a hundred-dollar valve is missing. Note what’s doing the governance work here: the AI is authorized to act (order parts, schedule repairs) precisely because it runs on infrastructure the Navy fully controls, with a bounded, well-tested action space. Autonomy and sovereignty aren’t separate concerns — the first depends on the second.

NAVSUP: predictive logistics that anticipate need. The Naval Supply Systems Command manages global depots and must resupply fleets at sea, where bureaucratic delay can be genuinely dangerous. The AI links a mission profile to logistics: if a fleet operates in heavy weather, the system proactively calculates increased wear and positions the needed parts forward in the nearest depot — before the crew even asks. The result is a near-elimination of expensive emergency logistics flights and a sharp reduction in excess “dead” inventory sitting in ports.

Across all three, the pattern is the same: AI is given real decision-making and action authority, but only within a tightly governed envelope — bounded actions, local/sovereign infrastructure where the stakes require it, and a clear chain back to a human accountable for the outcome.

Digital Sovereignty Is Part of Governance, Not a Separate Topic

It’s tempting to treat “who governs the AI’s decisions” and “who controls the infrastructure the AI runs on” as two different conversations. The Navy’s CBM+ case shows why they’re the same conversation. Governing an autonomous system and owning its infrastructure are the same question — as the five-dimension sovereignty framework covered separately on this site makes clear. Governing an autonomous decision-making system requires control over:

• Training infrastructure — who shapes what the model learns
• Inference infrastructure, including edge deployment — where and how the model actually runs when it makes a decision
• Reliable, often smaller models you can fully validate, rather than an opaque general-purpose model you can only prompt
• Fine-tuning, RAG, RLHF, and reflection — the mechanisms that let you actually shape behavior rather than just hope for it
• Compression, pruning, and distillation — the techniques that make it feasible to run a governed model locally, at the edge, instead of depending on a remote API you don’t control

An organization that governs only the prompt layer, while every actual inference call goes out to infrastructure it doesn’t control, hasn’t really solved the governance problem — it’s outsourced part of it to a party outside the accountability chain.

The Open Question: Centralized or Decentralized Governance?

This is where the framework runs out of easy answers, and honestly should. Should AI governance in a large organization be centralized — one body setting standards, validation thresholds, and explainability requirements for every AI system in the organization — or decentralized, with each department or ship, in the Navy’s case, governing its own agents against a shared baseline?

There’s a reasonable case for both. Centralized governance gives you consistency, a single point of audit, and economies of scale in validation and XAI tooling. Decentralized governance gives you speed and domain-specific judgment — the team running shipboard maintenance AI understands failure modes that a central AI ethics board never will in the same depth. Most organizations that have gone furthest with agentic AI seem to converge on a hybrid: centrally defined standards (the four-phase framework above, minimum XAI requirements, escalation thresholds for human sign-off) paired with decentralized implementation and monitoring close to where the decisions actually happen. Sovereignty and accountability get set centrally; day-to-day judgment stays local.

Where to Start: HR, Legal, and Finance

If governing agentic AI across an entire organization feels daunting, there’s a well-established place to build the muscle before extending it to higher-stakes domains: HR, in-house legal, and finance. These functions operate almost entirely in a digital and paper reality — no unpredictable physical world of weather, broken pumps, or physical distance to contend with. They deal in text and numbers in office documents, follow strict rules and protocols, and typically represent a slow, expensive bottleneck for the rest of the organization. That combination — high rule-density, high volume, low physical unpredictability — makes them the ideal low-risk proving ground for the governance framework above: you can validate the four phases, test your explainability tooling, and establish your escalation thresholds somewhere the cost of an early mistake is a delayed invoice, not a stranded ship.

The Takeaway

The technology question — does agentic AI actually work? — is behind us. What’s left is the governance question, and it doesn’t have a single settled answer yet, but it does have a workable shape: train and validate against measurable standards, build in reflection so the system catches its own errors, require genuine explainability so a human can actually audit the decision, control the infrastructure the decisions run on, and decide deliberately — not by default — how much of that sits centrally versus close to where the work happens. Organizations that treat this as an engineering afterthought will find out the hard way that “the AI decided” is not an acceptable answer to a board, a regulator, or a sailor whose ship is stuck in port.

Based on “AI en Bestuurlijke Besluitvorming” (Prof. dr. ir. J.C. Scholtes, roundtable session, August 2026).

Key papers

• Jobin et al. (2019), The Global Landscape of AI Ethics Guidelines — Nature Machine Intelligence
• Doshi-Velez & Kim (2017), Towards a Rigorous Science of Interpretable Machine Learning — arXiv:1702.08608
• Cath (2018), Governing Artificial Intelligence: Ethical, Legal and Technical Opportunities and Challenges — Phil. Trans. R. Soc. A

Further reading on this site

• The Institutional Moat — the architecture layer that governance sits on top of
• Why Europe Needs Its Own AI Models — sovereignty as the infrastructure prerequisite for real governance
• What Is an AI Agent? — what the system being governed actually is
• What It Takes to Build a Sovereign AI Agent — the end-to-end practical guide that applies this governance framework
• Bias Analysis in Practice — how to carry out the required bias analysis, justification test, and ethical desirability test

The Sandbox You’re Actually Building In

Foundation model providers aren’t just shipping smarter chatbots anymore — they’re building fully integrated, closed-loop agentic ecosystems: multi-step reasoning chains, native tool integration, proprietary knowledge graphs, neural rerankers trained on billions of queries, and increasingly recursive self-improvement. What they hand external developers, by contrast, is a controlled sandbox: system prompts (shallow control), function calling to trigger tools, standard RAG (basic vector search), and pre-built templates. You do not get access to their RLHF layers, their model weights, or their internal reranking. You get to build on the platform — not into it.

That gap matters more than it looks like at first glance. A system prompt, however carefully engineered, is treating the model as a black box. When your entire “agent” is a Markdown file layered over a general-purpose LLM, you haven’t built a proprietary system — you’ve built a configuration of someone else’s product. That’s the trap: you end up building a dependency, not a company.

Why “The Next Model Version Will Fix It” Is a Fatal Strategy

It’s tempting to defer. Base models improve every few months, so why invest in custom architecture now? Because relying solely on the ever-improving base model turns your business into a temporary feature on someone else’s roadmap. Customer loyalty built on “good enough” is a mirage, particularly in conservative markets — the moment a foundation model provider ships a superior integrated capability in your exact niche, that loyalty disappears overnight. Without your own memory layer, your own reasoning logic, and your own verification loops, you are not a company with an AI product. You are a wrapper waiting to be absorbed.

The uncomfortable question every CTO should be asking isn’t whether the platforms will expand into your niche — it’s whether your organization can out-specialize them before they do.

Where the Real Moat Actually Lives

Value-add has shifted. It’s no longer in which foundation model you call — that’s increasingly a replaceable, commodity choice between Claude, GPT, Gemini, or an open-weight model like Llama. The moat is in the proprietary architecture and data sovereignty you build around it. Four components do the heavy lifting.

The Compounding Advantage: Reasoning Triples

Put these four together and something compounding starts to happen. Every real interaction generates a reasoning triple: the observation the system saw, the action or tool call it chose, and the consequence — success or failure. That data refines your reranker, your critic, and your planner, which produces better outcomes, which drives more usage, which produces more data. This is the flywheel that turns a one-off integration project into a genuinely defensible asset — and, notably, it’s exactly the kind of proprietary dataset that later lets you fine-tune around each new generation of foundation model rather than being replaced by it.

What to Do Monday Morning

If you’re a CTO or CIO reading this and recognizing your own stack, the fix doesn’t require a moonshot:

1. Audit your AI stack. If it’s prompts plus API calls and nothing else, you currently have no moat.
2. Start collecting reasoning triples today, from every expert interaction — this data only exists going forward, not retroactively.
3. Implement a neural reranker on your domain data. This is usually the highest-ROI first step.
4. Build your first Critic agent. Fifty curated examples is enough for a working prototype.
5. Decouple your architecture into modular, independently upgradable components — reranker, critic, planner, and guardrails should each be replaceable without rebuilding the whole system.

The EU Angle: Why This Isn’t Optional Here

Everything above applies globally. But European CTOs and CIOs have an additional, very concrete reason not to treat this as a nice-to-have: almost all of this architecture currently sits on top of American infrastructure and American foundation models — and that dependency has already proven fragile once. This is the dimension covered in depth in the companion post on AI sovereignty on this site.

In mid-2026, Anthropic suspended access to its newly released Fable and Mythos model tiers for several weeks to comply with U.S. Department of Commerce export controls, before the Department lifted those restrictions and access was restored. Whatever the specifics of that episode, it’s a concrete, recent, real-world demonstration of the exact risk this post is about: if your entire operation runs on a thin layer of prompts over a foreign-hosted model, an export control decision, a pricing change, or a policy shift made in Washington — not in Brussels, and not by you — can suspend or reprice your core capability with no notice and no recourse.

This is precisely why Stanford’s 2026 AI Index frames AI sovereignty across five dimensions — infrastructure, data, model, application, and talent — and why the report shows Europe investing heavily but still trailing in raw compute and model production. An organization that has built nothing but a prompt layer on a U.S. model has zero independent leverage in that scenario. An organization that has built its own RAG-and-reranking memory, its own fine-tuned critics, its own planning logic, and its own guardrails — all trained on proprietary European data — can survive a model swap. The foundation model becomes what it should always have been: a replaceable component, not the whole business.

For European institutions specifically — hospitals, legal firms, financial services, public sector bodies bound by GDPR and sector-specific regulation — this isn’t abstract risk management. It’s the difference between an AI capability you control and one you merely rent, at a price and on terms someone else can change at any time.

The Takeaway

The question every CTO and CIO needs an honest answer to isn’t whether to use Claude, GPT, or Gemini — it’s what you’ve built around it. If the answer is “a system prompt and some function calling,” you have a sandbox, not a moat, and a single point of foreign dependency sitting underneath your entire operation. Build the architecture — memory, tools, planning, critics, guardrails — and the foundation model underneath becomes exactly what it should be: swappable, commoditized, and no longer the thing your business depends on for survival.

This post draws on “The Institutional Moat: Beyond the Big Tech AI Trap” (Prof. dr. ir. J.C. Scholtes, Endeit Capital, March 2026).

Key papers

• Bommasani et al. (2021), On the Opportunities and Risks of Foundation Models — arXiv:2108.07258
• Brown et al. (2020), Language Models are Few-Shot Learners (GPT-3) — arXiv:2005.14165
• Ouyang et al. (2022), Training Language Models to Follow Instructions with Human Feedback — arXiv:2203.02155

Further reading on this site

• What Is an AI Agent? — what the seven agentic components you are building actually are
• From Text to Action — the concrete pipeline for building the proprietary architecture described here
• Why Europe Needs Its Own AI Models — the geopolitical dimension of why this architecture cannot rest on foreign infrastructure
• Governing the Autonomous Organization — the governance layer that sits on top
• What It Takes to Build a Sovereign AI Agent — the step-by-step guide putting everything in this post into practice

What “AI Sovereignty” Actually Means

The Index defines AI sovereignty as a state’s — or, by extension, an organization’s — “capacity to act deliberately and make independent decisions over the development, deployment, and governance of AI systems” rather than depending on external actors for critical capability. Crucially, the report breaks this down into five distinct layers, and each one maps directly onto a decision your own organization has to make, not just a government:

• Infrastructure sovereignty — who controls the compute your models run on
• Data sovereignty — where your data lives and who can access it
• Model sovereignty — whether you can build, adapt, and control the models themselves
• Application sovereignty — how much control you have over the systems built on top
• Talent sovereignty — whether you have the people to do all of the above

Let’s walk through what the data says about each — and where edge computing fits in.

Infrastructure Sovereignty: Europe Is Investing, But the Gap Is Real

Domestic compute capacity is “increasingly used as an indicator of compute sovereignty” — a way to reduce reliance on foreign providers and maintain continuity of access during export controls or geopolitical disruptions. The numbers show Europe moving fast, but from behind: state-backed AI supercomputing clusters in Europe and Central Asia grew from 3 to 44 between 2018 and 2025 — the sharpest acceleration of any region, driven largely by coordinated efforts like the European High Performance Computing Joint Undertaking (EuroHPC JU). That’s real momentum. But China still leads with 85 clusters, and North America grew nearly sevenfold to 41 over the same period.

Here’s the direct link to edge computing: infrastructure sovereignty isn’t only about who owns the largest training supercomputer. It’s equally about where inference happens — the day-to-day running of models once they’re built. An organization that trains a model on sovereign European infrastructure but then has to run every single query through a foreign cloud API has only solved half the problem. Edge and on-premise deployment — running models locally, close to where the data is generated and used — is how infrastructure sovereignty actually reaches daily operations rather than remaining a policy talking point.

Data Sovereignty: Europe Already Set the Global Standard

This is where Europe’s position is genuinely strong, and the Index confirms it with hard numbers. Data localization measures — legal requirements that data be stored or processed within a country’s borders — have risen sharply worldwide since 2016, “coinciding with the implementation of GDPR in Europe and the subsequent Brussels Effect, whereby other nations adopted similar frameworks.” Europe and Central Asia now have 66 such measures, part of what the report calls a high-localization cluster alongside East Asia (77) and sub-Saharan Africa (71). North America, by contrast, sits at just 3 measures, reflecting what the report describes as a long-standing “flow-first” policy orientation.

That regulatory framework is only meaningful in practice if the infrastructure exists to back it. GDPR-grade data governance and edge computing are natural partners: keeping sensitive data — medical records, legal documents, agricultural sensor data — physically within a domestic or even on-premise environment is far easier when the model doing the processing runs locally rather than shipping every query to infrastructure outside the jurisdiction.

Model Sovereignty: The Concentration Problem

Model production remains heavily concentrated. The report’s data on publicly released models shows the United States reaching 1,618 cumulative model releases by 2025, China 849, and Europe and Central Asia 666 — with the UK (229) and France (141) as the leading European contributors. Europe is a distant third globally, though its trajectory is steady rather than stagnant.

This matters because model sovereignty is what determines whether an organization can actually adapt a model to its own domain — fine-tuning on proprietary legal, medical, or agricultural data, for instance — rather than being limited to whatever a foreign general-purpose model happens to support. Open-source frameworks have lowered the barrier to entry, which is precisely why smaller, specialized models — the kind that can realistically run at the edge, on modest hardware, close to the data — are becoming a viable European strategy rather than an afterthought.

Application Sovereignty: Where Europe Has Real Room to Compete

Interestingly, the report notes that the application layer — how AI is actually deployed within specific sectors like healthcare, finance, or agriculture — is “less concentrated than the model or compute layer,” giving countries “more space… to develop niche specializations.” Germany’s strength is in industrial and manufacturing applications; Estonia’s is in education technology. This is exactly the layer where a smaller, sovereignty-conscious European organization can compete on domain depth rather than on raw model scale — which again favors an edge-capable, domain-specific deployment model over a one-size-fits-all cloud API.

Talent Sovereignty: A Quiet Warning Sign

The fifth dimension — the ability to develop and retain AI talent — shows a less visible but structurally important trend: cross-border AI talent circulation has slowed globally, with both inflows and outflows declining, meaning talent increasingly stays within national or regional systems. For Europe, this cuts both ways — talent retained is talent not lost to the US, but it also means the region cannot simply assume it can import its way out of a capability gap. Building the domestic expertise to design, deploy, and maintain sovereign AI infrastructure is itself part of the sovereignty equation.

Why This Adds Up to “Run It Yourself, Close to Home”

Put these five dimensions together and a clear strategic picture emerges for European organizations, not just European governments:

1. You already operate under some of the strongest data protection requirements in the world (data sovereignty) — but that framework only bites if your infrastructure choices actually keep processing local.
2. The compute is being built (infrastructure sovereignty) — European supercomputing capacity grew faster than any other region between 2018 and 2025 — but training capacity alone doesn’t help if every inference call still leaves the region.
3. Open-source models have lowered the barrier to model sovereignty — meaning smaller, specialized, fine-tunable models are realistic even for organizations well below nation-state scale.
4. The application layer is where Europe can actually win — domain-specific deployment, not general-purpose scale, is the competitive opening the data points to.

Edge computing is the practical mechanism that ties all four together. It’s what lets a compliance-sensitive sector — healthcare, legal, agriculture — actually benefit from Europe’s regulatory strength instead of being undermined by it, because the data and the model doing the reasoning never have to leave the building, let alone the jurisdiction.

The Takeaway

Stanford’s AI Index doesn’t use the phrase “edge computing,” but its own sovereignty framework makes the case for it implicitly and repeatedly: sovereignty is only real when it reaches all the way down to where inference actually happens. For Europe specifically — with genuinely strong data protection law, fast-growing but still second-tier compute capacity, a widening base of open, adaptable models, and a real opening at the application layer — running your own models on your own infrastructure, as close to the edge as the use case demands, isn’t a defensive or nostalgic choice. It’s the version of the strategy the data actually supports.

Source: Stanford HAI, 2026 AI Index Report, Chapter 8.3, “AI Sovereignty.”

Key papers

• Stanford HAI (2026), AI Index Report — hai.stanford.edu/ai-index
• Bommasani et al. (2021), On the Opportunities and Risks of Foundation Models — arXiv:2108.07258
• European Commission (2024), EU Artificial Intelligence Act — eur-lex.europa.eu

Further reading on this site

• The Institutional Moat — how to build architecture that survives a model swap or export-control decision
• Governing the Autonomous Organization — sovereignty and governance as one question, not two
• David vs. Goliath: Small Model vs. Frontier — the evidence that small, local models can nearly match frontier performance on specific tasks
• What It Takes to Build a Sovereign AI Agent — the practical eight-step guide to building a system your organization actually owns

Why the Model Isn’t Your Moat

It’s tempting to think that wiring a system prompt onto a strong LLM is building an agent. It isn’t. Relying purely on system prompts and Markdown “skill files” gives you only shallow control — you’re treating the model as a black box and, more importantly, building a dependency on someone else’s roadmap rather than a company. Many teams that claim to have “built an agent” have really just layered instructional text over a general-purpose model. That’s the God-Prompt trap, and it’s brittle: rigid if-then rules can’t account for the infinite long tail of exceptions that real-world text mining and operations actually contain.

The real competitive advantage — the “moat” — isn’t the underlying LLM at all. Big Tech models are generalists: they own multi-step reasoning chains, native tool integration, and recursive self-improvement, but what you get through their APIs is largely a sandbox — system prompts, function calling, and standard vector search. If your business’s value is purely tied to whichever model you’re calling, you’re standing in the path of a tidal wave: the next model update might simply absorb what you built. The moat is in the architecture around the model — and that architecture has to be trained on your text.

Step One: From Raw Text to Structured JSON

Raw text — especially from PDFs — is built for human eyes, not machine reasoning. An LLM reading a flat text dump can’t reliably tell a chapter heading from a technical definition, a warning label from a procedural step. That ambiguity is exactly what breaks downstream agentic components, so the first real engineering step is converting unstructured text into a hierarchical, machine-readable JSON schema — a “foundational ontology” that explicitly maps chapters to sub-topics to definitions to procedures.

This step matters more than it looks like it should, because errors cascade. If the JSON conversion flattens the hierarchy or merges unrelated paragraphs:

• RAG chunks lose their parent context — a retrieved procedure like “terminate the malicious process” might silently lose the fact that it only applies to one specific scenario, and the reranker downstream won’t be able to surface the right answer.
• Sequential workflows get taught out of order — if Phase 1/2/3 structure isn’t preserved, an agent’s fine-tuned planning layer can learn to execute step 3 before step 1.
• Knowledge graphs lose their edges — entity relationships that aren’t captured hierarchically can’t later support multi-hop reasoning.
• RLHF preference pairs lose their justification — without the surrounding context, a Critic agent can’t judge whether an action actually violated a rule.

Get the structure right once, early, and every downstream training pipeline benefits. Get it wrong, and every downstream component inherits the flaw.

Step Two: Memory — RAG and Neural Reranking as Your Corporate Memory

Once your text is structured, it becomes the backbone of the agent’s memory. The mechanics are familiar but worth being precise about:

1. Chunking — break long documents into semantically coherent sections (roughly 500 words is a common target), not arbitrary fixed-length slices.
2. Indexing — convert chunks into vectors and store them in a vector database (Pinecone, pgvector, Azure AI Search).
3. Retrieval — given a query, pull the most relevant chunks.
4. Augmentation — instruct the model to answer using only those retrieved sources.
5. Generation — produce a cited, factual answer.

A plain vector search finds things that are similar; it doesn’t reliably find the thing that’s correct. That’s the job of a neural reranker — a fine-tuned cross-encoder trained on your own query/best-answer pairs, which re-examines the top 50–100 candidates and pushes the actual “smoking gun” answer into position one, dramatically cutting hallucinations and solving the “lost in the middle” problem where a model buried in twenty similar-looking chunks misses the one that matters. Building this well requires real infrastructure choices: a strong embedding model (don’t skimp here — a bad translator produces bad search), an orchestration framework such as LlamaIndex or LangChain to wire files to database to model, and a reliable document parser for the messy PDFs you’re starting from.

Step Three: Tools — Turning Text into Operational Limbs

Skills and tools are the “limbs” of an agentic architecture — structured definitions (API schemas, JSON functions, SQL interfaces) that let a model act instead of only talk. Proprietary internal APIs are where this becomes a genuine business moat: a general-purpose model from a big provider is legally and technically blocked from touching your internal systems, however smart it is. An agent wired into your own electronic patient database, CRM, or monitoring tools can pull real historical values, cross-reference them, and act on live context instead of guessing from pre-trained weights.

This is also where multi-hop orchestration becomes necessary rather than optional. Complex professional tasks rarely resolve in one lookup — an agent chaining an HR database (who’s on call), a monitoring tool (current load), and an execution tool (restart the service) is running an OODA loop: observe, orient, decide, act, and repeat until the objective is met, adjusting its plan as each tool call returns new information.

Step Four: Reasoning and a Digital Peer Reviewer

Planning frameworks like Chain-of-Thought, Tree-of-Thought, and Skeleton-of-Thought give a large model room to decompose a task into sub-goals with dependencies rather than answering in one pass. But a large, general model generating a first draft still benefits from a second, much smaller and more specialized model reviewing it — a Critic — fine-tuned specifically on your organization’s own corrections.

The training data for that critic is concrete and collectible: the rejected or edited raw outputs, the expert-corrected version, the rationale for the correction (“violates Section 4.2”), and any hard negative constraints or compliance red lines. As a rough scale guide: 50–100 examples gets you a prototype, 500–1,000 a production-grade critic, and 2,000+ starts to approach genuine domain expertise. This is precisely the kind of proprietary, real-world correction data that a general-purpose LLM provider will never have access to — which is exactly why it’s a durable advantage rather than a temporary one.

Step Five: Guardrails — Where Institutional Trust Actually Comes From

Hallucination and compliance risk is the single biggest barrier to real institutional AI adoption, so the architecture needs independent verification layers surrounding the core model, not baked into it:

• Fact-checking — compare every draft against the RAG source of truth; every claim needs to be citable.
• Safety filters — enforce legal, ethical, and compliance constraints, blocking outputs that cross defined red lines.
• Self-consistency checks — run the same query through multiple reasoning paths and discard answers that don’t agree with each other.

None of these should be delegated entirely to another instance of the same probabilistic model policing itself. Deterministic, hard-coded checks — input guarding before a prompt even reaches the main agent, strict schema enforcement on outputs — provide the backstop that a purely LLM-based critic cannot guarantee.

The Flywheel

Put these pieces together and something compounding happens. Every real interaction generates a reasoning triple: the observation the system saw, the action or tool call it chose, and the consequence — success or failure. That data doesn’t just sit there; it refines your rerankers, your critics, and your planners, which produces better outcomes, which drives more usage, which produces more data. This is the flywheel that converts a one-time text-mining project into a compounding, defensible asset — and it’s precisely the dataset that later lets you fine-tune around each new generation of foundation model, rather than being replaced by it.

The Takeaway

Converting text into agentic training data isn’t a single technical trick — it’s a pipeline: structure the text into a faithful JSON hierarchy, use that hierarchy to build a real memory layer (RAG plus reranking), extract the API definitions that give the agent hands, collect the correction data that trains a critic, and wrap all of it in deterministic guardrails. Do this well with your own proprietary text, and you’re not renting intelligence from a foundation-model provider — you’re building an architecture, and a dataset, that gets more valuable every time it’s used.

For a concrete domain example of this exact pipeline applied to agricultural logbooks and institutional knowledge, see From the Next Farmhand to the Next AI Agent.

Key papers

• Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — arXiv:2005.11401
• Ouyang et al. (2022), Training Language Models to Follow Instructions with Human Feedback (InstructGPT) — arXiv:2203.02155
• Rafailov et al. (2023), Direct Preference Optimization — arXiv:2305.18290

Further reading on this site

• What Is an AI Agent? — what the seven components being trained here actually are
• From Logbooks to Agent Training Data — the same pipeline applied to a real agricultural knowledge base
• David vs. Goliath: Small Model vs. Frontier — what a well-trained specialized model built this way can achieve
• The Institutional Moat — why building this proprietary architecture matters strategically

Agents Are Older Than You Think

Long before ChatGPT, Artificial Intelligence: A Modern Approach (Russell & Norvig, 1995) defined an agent as anything that:

1. senses its environment,
2. perceives inputs from that environment,
3. reasons about those inputs, and
4. acts upon the environment through its actions.

That definition doesn’t mention language models at all — it includes thermostats, robots, web crawlers, and self-driving cars. What separates a mere “computer program” from an agent is that an agent operates autonomously, persists over time, adapts to change, and pursues goals rather than simply executing one instruction and stopping.

In the late 1990s, “agent” was everywhere — intelligent agents, software agents, personal digital assistants. They were built on symbolic AI, expert systems, and rule-based reasoning. And they mostly failed to live up to the hype. APIs were primitive, the web was barely accessible programmatically, memory was static, and language understanding amounted to keyword spotting. The concepts — memory, goals, planning, an interface to the world — were sound. The technology to make them work simply wasn’t there yet.

What Changed: The LLM Is the Missing Engine

A useful mental model: think of a stand-alone LLM as a brain in a jar. It can reason in language, but it has no body, no senses beyond the text you feed it, and no memory once the conversation ends. An agent gives that brain:

• a body — tools it can call to act on the world,
• senses — retrieval systems and APIs that bring in fresh information,
• memory — context windows and external databases, and
• goals — a planner that decides what to do next.

That’s the moment a language model stops being a very articulate autocomplete and starts being something that can act.

This matters because stand-alone LLMs have real, well-documented limitations: they hallucinate confidently, their knowledge has a cutoff date, their context window is finite, they cannot browse the web or call an API on their own, they reason poorly in a single pass on multi-step problems, and they have no built-in way to notice or correct their own mistakes. Agentic architecture doesn’t eliminate these limitations — but it gives you the scaffolding to manage them.

The Seven Things an Agent Architecture Needs

Whether you’re looking at a 1990s symbolic system or a 2026 LLM-based one, a working agent architecture requires the same seven ingredients:

• Memory (short-term and long-term)
• Knowledge access
• Tool use / action execution
• Planning & control loop
• Goal management
• Verification & guardrails
• A user/system interface

What’s changed isn’t the list — it’s the horsepower behind each item. Let’s walk through the ones that matter most in practice.

Why This Matters Now

None of this is achievable with simple retrieval alone. Consider four examples of what agentic architectures make possible that plain RAG cannot:

• Autonomous research and synthesis — planning multi-hop queries across medical, legal, and financial domains, identifying conflicting studies, and producing a structured report without a human directing each step.
• End-to-end troubleshooting — diagnosing an issue through conversation, calling diagnostic APIs to check system logs, and autonomously executing a fix.
• Strategic decision support — decomposing a broad question like “analyze our market expansion risk” into sub-queries, pulling live competitive data, running calculations, and adapting strategy as findings come in.
• Forensics and compliance monitoring — continuously watching logs or communications for patterns of concerning behavior and triggering escalation when thresholds are crossed.

None of these are single-shot text generation tasks. They require memory across steps, tools that reach into the real world, a plan that can be revised, and a mechanism to catch the system’s own mistakes before they become someone else’s problem.

The Takeaway

An AI agent isn’t a chatbot with a longer memory, and it isn’t a rebrand of a 1990s rule-based system either. It’s what happens when a language model — genuinely capable of open-ended reasoning for the first time — gets wrapped in the same seven components AI researchers identified thirty years ago: memory, knowledge access, tool use, planning, goal management, verification, and an interface to act through. The concepts haven’t changed. What changed is that we finally have an engine powerful enough to make them work.

This post is a high-level overview. Later posts in this series go deeper into specific components — RAG architectures and neural reranking, multi-agent orchestration and governance, and the training data needed to specialize these components for a given domain.

Key papers

• Russell & Norvig (2020), Artificial Intelligence: A Modern Approach (4th ed.) — norvig.com/aima.html
• Yao et al. (2022), ReAct: Synergizing Reasoning and Acting in Language Models — arXiv:2210.03629
• Wang et al. (2023), A Survey on Large Language Model Based Autonomous Agents — arXiv:2308.11432

Further reading on this site

• From Text to Action: Turning NLP Pipelines into Agentic Workflows — how to build the training data behind each of these components
• Governing the Autonomous Organization — who is accountable when the agent acts
• Why NLP Finally Works — the transformer architecture that made agents possible
• The Institutional Moat — why the architecture around the model matters more than the model itself

Why This Is Possible at All

Every trained model draws a decision boundary through an extremely high-dimensional space, separating “panda” from “gibbon,” or “positive review” from “negative review.” That boundary is learned from data, and it’s shaped by whatever correlations happen to exist in the training set — not necessarily by the same features a human would use to make the same distinction. An adversarial attack works by finding the shortest possible path across that boundary: a tiny, deliberately calculated perturbation to the input that pushes it just barely to the other side, without changing anything a human would notice. The vulnerability isn’t a bug in one particular model — it’s a structural consequence of how these decision boundaries get learned in the first place, which is why the problem shows up across model types and domains.

Adversarial Attacks on Images

Gradient-based perturbation attacks. The classic approach — the Fast Gradient Sign Method and its more iterative successor, Projected Gradient Descent — computes the gradient of the model’s loss with respect to the input pixels rather than the model’s weights, and then nudges every pixel by a tiny amount in whichever direction increases the loss (i.e., makes the model more wrong) the fastest. Because the perturbation is spread across the entire image and kept within a small bound, it’s imperceptible to a human viewer while being precisely calibrated to cross the model’s decision boundary.

Patch attacks. Rather than perturbing an entire image subtly, a patch attack places one small, highly visible, often strange-looking sticker or region somewhere in the frame — on a stop sign, on a piece of clothing, in the corner of a photo — that reliably causes a targeted misclassification regardless of everything else in the image. Unlike gradient-based perturbations, these are often robust to being photographed at different angles and distances, which is exactly what makes them a practical physical-world concern (a modified stop sign that a self-driving system misreads as a speed limit sign, for instance) rather than just a digital curiosity.

Black-box and transfer attacks. Both attack families above assume the attacker has access to the model’s internals to compute gradients. In practice, many real attacks don’t need that: adversarial examples crafted against one model frequently transfer and fool a different model trained on similar data, even without access to that second model’s weights at all. This is what makes the threat practical rather than purely academic — an attacker doesn’t need to steal your exact model to attack it.

Adversarial Attacks on Text

Text is discrete — you can’t nudge a word by 0.1% the way you can nudge a pixel’s brightness — so text-based adversarial attacks look structurally different, but the underlying goal is identical: change the input as little and as unnoticeably as possible while flipping the model’s output.

Character-level perturbations. Swapping visually similar characters, inserting or deleting a character, or introducing common typos can flip a classifier’s output while remaining perfectly readable to a human — “excellent” becomes “excel1ent” or “exceellent,” and a sentiment classifier’s confidence collapses even though no reasonable reader would misread the word.

Word-level substitution attacks. A more sophisticated family of attacks replaces individual words with close synonyms — chosen specifically to preserve the sentence’s actual meaning to a human reader while shifting it just enough in the model’s embedding space to cross a decision boundary. Because the substitution is semantically faithful, these are harder to catch with a simple typo filter and harder to dismiss as “obviously broken input.”

Universal adversarial triggers. Rather than crafting a bespoke perturbation for each individual input, some attacks discover a short sequence of words or tokens that, when prepended or appended to almost any input, reliably pushes the model toward a specific wrong output — a kind of reusable “skeleton key” for a given model, rather than a one-off attack tailored to a single sentence.

Prompt injection (the LLM-specific version). In agentic systems specifically, an increasingly relevant variant is prompt injection: text embedded in a document, webpage, or tool output that a language model reads as part of its context — not something the actual user typed — but that’s crafted to look like an instruction the model should follow. This is structurally the same underlying vulnerability as classic adversarial text attacks (a carefully chosen input pushes the model somewhere it shouldn’t go), applied to the specific weakness of a system that can’t always distinguish “data I’m reading” from “instructions I should obey.”

What Actually Works as a Defense

No single technique fully solves this, which is why real deployments layer several together:

Adversarial training. The most direct defense: deliberately generate adversarial examples during training and include them in the training set, so the model learns a decision boundary that isn’t so easily crossed by small perturbations. This can be extended with broader data augmentation — random noise, hand-crafted negations, synonym substitution, and general training-set diversity — so the model doesn’t overfit to a narrow, brittle notion of what “normal” input looks like.

Defensive distillation and feature squeezing. Training a smaller student model on a teacher’s softened output distribution tends to produce a smoother, less sharply-creased decision boundary that’s harder to exploit with a tiny, precisely-calculated nudge. Feature squeezing works in a related but more direct way: it reduces the input’s degrees of freedom before the model ever sees it — for images, this might mean reducing color depth or applying mild smoothing; for text, normalizing spelling variants and character substitutions before classification — specifically to collapse the fine-grained space an attacker needs to operate in.

Input transformations and randomization. Techniques like slight image compression, cropping, or resizing before classification can disrupt the very precise pixel-level calculations an adversarial perturbation depends on, without meaningfully affecting a genuine image. On the text side, spelling normalization and canonicalization serve a similar purpose against character-level attacks.

Ensembles. Running the same input through multiple independently trained models and requiring agreement (or flagging disagreement) raises the bar significantly for an attacker, since a perturbation crafted to fool one specific model’s particular decision boundary often doesn’t transfer cleanly to a differently-trained model with a different boundary.

Detection rather than pure prevention. Rather than trying to make a model unconditionally robust to every possible perturbation, a separate detection layer can flag inputs that look statistically unusual — oddly high-frequency noise patterns in an image, unusual character sequences in text — and route them for additional scrutiny before they ever reach a decision.

For agentic systems specifically: input guarding and deterministic guardrails. Because prompt injection exploits the boundary between “data” and “instructions,” the most effective defense isn’t asking the language model to police itself — it’s routing untrusted input (documents, web content, tool outputs) through a separate, deterministic check before it reaches the main reasoning model, and treating anything that looks like an embedded instruction as suspicious by default rather than as a legitimate command.

The Common Thread

Every one of these defenses is really doing the same underlying thing: making the model’s decision boundary smoother, less sensitive to tiny or narrowly-targeted changes, and less trusting of any single unverified signal. None of them make a model perfectly unbreakable — this remains a genuinely active arms race between attack and defense research, on both images and text — but layering several of these together (adversarial training plus distillation plus input normalization plus detection) meaningfully raises the cost and difficulty of a successful attack, which in most real deployments is exactly the practical goal.

Key papers

• Goodfellow et al. (2014), Explaining and Harnessing Adversarial Examples (FGSM) — arXiv:1412.6572
• Madry et al. (2017), Towards Deep Learning Models Resistant to Adversarial Attacks (PGD) — arXiv:1706.06083
• Wallace et al. (2019), Universal Adversarial Triggers for Attacking and Analyzing NLP — arXiv:1908.07125

Further reading on this site

• Shrinking the Giant: Pruning, Quantization, and Distillation — how model compression closes off a whole class of adversarial attacks as a side effect
• Why You Can’t Fine-Tune or Distill Without XAI — how LIME-based before/after testing detects adversarial brittleness introduced by fine-tuning
• Governing the Autonomous Organization — the governance layer that handles security threats in institutional AI deployments
• The Institutional Moat — why deterministic guardrails around a model matter more than trusting a model to police itself
• What Is an AI Agent? — the agentic architecture where prompt injection specifically becomes a threat

Three Words That Aren’t Synonyms

Practitioners often use “transparency,” “interpretability,” and “explainability” interchangeably, but they mean genuinely different things:

• Transparency exists when the process that extracts model parameters from training data, and generates labels from test data, can be described and justified by the person who designed the approach.
• Interpretability is the ability to comprehend the model and present the basis for its decision-making in a way a human can actually understand.
• Explainability — a term without full field-wide consensus yet — is best understood as the set of features from an interpretable domain that contributed to a specific decision for a specific example.

The distinction matters practically: you can have a transparent process (you know exactly what algorithm you used and why) without having an interpretable model (you still can’t explain any individual decision it makes).

What Actually Counts as a Valid Explanation?

This is worth taking seriously as a genuinely human, social question, not just a technical one. A valid explanation is part of a social interaction — it has to be sound, cogent, convincing, and it has to build trust in the receiving party, delivered as a step-by-step transfer of knowledge they can actually follow. Beyond that, a good explanation needs to show the difference between possible outcomes (why this answer rather than that one), needs to be relevant to what the person actually wants to know, and — importantly — is allowed to be incomplete. Highlighting a few salient examples is often sufficient; an explanation doesn’t need to be exhaustively complete to be useful, and in fact often shouldn’t be.

The Fundamental Trade-off: Accuracy vs. Explainability

There’s a well-known, rough hierarchy of model families running from highly interpretable but often less powerful (decision trees, linear/logistic models, simple statistical models) to highly accurate but much harder to interpret (deep learning, ensemble methods, large neural networks). Three broad strategies exist to deal with this trade-off:

1. Interpretable models — techniques that learn inherently structured, causal, interpretable models in the first place.
2. Deep explanation — modified deep learning techniques specifically designed to learn explainable features as part of training.
3. Model induction — techniques that infer an explainable model from any other model, treated purely as a black box.

Most of the practical tools covered below fall into the third category — they don’t require you to change your architecture, only to interrogate it after the fact.

The Practical XAI Toolbox

Why This Stops Being Optional When You Train the Model Yourself

Everything above is useful for understanding any model, including one you’re only calling through an API. It becomes a genuine requirement — not a nice-to-have — the moment you’re fine-tuning, distilling, or training a component yourself, for several concrete reasons.

Fine-tuning and distillation can change how a model decides without changing whether it gets the right answer. As covered in the earlier post on pruning, quantization, and distillation, a distilled student model is trained to match a teacher’s softened output distribution — but matching the output distribution doesn’t guarantee the student arrived there via the same reasoning path. A validation accuracy score that looks identical before and after distillation can hide a genuinely different — and possibly less trustworthy — decision process underneath. XAI tools are how you actually check this: run SHAP or saliency analysis on the same set of examples before and after distillation, and compare not just the predictions but which tokens or features drove them.

This connects directly to the “faithfulness versus plausibility” trap. A valid human explanation, as covered above, needs to be convincing and build trust — but a method like LIME is specifically built to produce a simplified, human-palatable local approximation. That raises an uncomfortable question worth sitting with honestly: when you check your fine-tuned model’s explanations and they look reasonable, are you verifying that the model actually reasoned that way, or are you just generating a comforting post-hoc rationalization that satisfies human psychology without reflecting the actual high-dimensional computation? This is exactly why a serious verification process for a self-trained model needs more than one XAI method — a saliency map, a SHAP explanation, and an attention visualization that all tell a consistent story is meaningfully stronger evidence than any single one alone.

Fine-tuning can introduce new adversarial brittleness that a clean validation set won’t reveal. As covered in the earlier post on adversarial attacks, a model fine-tuned on a narrower dataset can become more sensitive to small, meaning-preserving perturbations — a synonym swap or a minor typo. There’s a concrete detection method worth adopting as a standard step: run LIME on an example before an adversarial perturbation, apply the perturbation, then run LIME again. If the explanation shifts dramatically, that’s a direct signal of adversarial sensitivity your fine-tuning process introduced or failed to remove.

Regulatory and governance frameworks explicitly require this. As covered in the earlier posts on AI governance and bias analysis under the Dutch Algorithm Framework, being able to show why a specific decision was made — not just that the system performs well in aggregate — is often a legal requirement, not a courtesy. A fine-tuned or distilled model deployed in a public-sector, medical, or financial context needs to support exactly this kind of per-decision explanation, and that capability has to be built in and verified during training, not retrofitted after a complaint arrives.

A Practical Checklist for Anyone Fine-Tuning or Distilling Their Own Model

1. Run attention visualization (BERTViz) before and after fine-tuning on a fixed set of representative examples, and look for attention patterns shifting toward unexpected tokens.
2. Run SHAP or LIME on the same validation examples before and after distillation, and compare not just the predicted label but which features drove it — a matching label with a different explanation is a warning sign, not a pass.
3. Use saliency or Integrated Gradients to spot brittleness, especially single-word substitutions that flip a prediction — these are exactly the failure modes a plain accuracy metric will never surface.
4. Test explicitly for adversarial sensitivity introduced by training, using the before/after LIME comparison method — a stable model should show stable local explanations under small, meaning-preserving perturbations.
5. Look past attention at neuron activations and hidden-state evolution, since two-thirds of a Transformer’s capacity sits in the feed-forward layers that most standard tooling ignores.
6. Document all of this, the same way the earlier posts on governance and bias analysis argue for documenting fairness testing.

The Takeaway

Borrowing a foundation model through an API means borrowing someone else’s explainability obligations along with its capability. The moment you fine-tune it, distill it, or train a specialized component yourself — exactly the path this site has argued for repeatedly as the route to genuine sovereignty and specialization — that obligation transfers to you in full. A model that scores well on a held-out test set has told you what it does. Only XAI tells you how — and without that, you’re deploying a system you’ve built but genuinely can’t account for.

Based on “XAI for NLP” (Advanced Natural Language Processing course, Department of Advanced Computing Sciences).

Key papers

• Ribeiro et al. (2016), “Why Should I Trust You?”: Explaining the Predictions of Any Classifier (LIME) — arXiv:1602.04938
• Lundberg & Lee (2017), A Unified Approach to Interpreting Model Predictions (SHAP) — arXiv:1705.07874
• Sundararajan et al. (2017), Axiomatic Attribution for Deep Networks (Integrated Gradients) — arXiv:1703.01365
• Vig (2019), A Multiscale Visualization of Attention in the Transformer Model (BERTViz) — arXiv:1906.05714

Further reading on this site

• Why NLP Finally Works — the transformer architecture whose internals XAI tools are designed to open up
• Shrinking the Giant: Pruning, Quantization, and Distillation — where the need for XAI during distillation originates
• Adversarial Attacks: Invisible Pixels and Swapped Words — the brittleness that LIME-based adversarial testing detects
• Governing the Autonomous Organization — the governance framework that makes explainability a legal requirement
• Bias Analysis in Practice — the complementary fairness testing that XAI enables
• What It Takes to Build a Sovereign AI Agent — step 7 (validation) is where the XAI checklist in this post belongs

Why This Is More Than a Compliance Checkbox

The Algorithm Framework frames this as fulfilling a requirement from the Dutch Data Protection Authority (Autoriteit Persoonsgegevens) around automated decision-making: the risk of discriminatory processing has to be investigated and mitigated. Two recent Dutch cases show what goes wrong when this doesn’t happen:

• DUO / student housing grant control: between 2010 and 2023, the Dutch education executive agency (DUO) used a risk profile to detect misuse of the “living-away-from-home” student grant. An investigation by Algorithm Audit found insufficient statistical association between several of the selection criteria used — the profile was discontinued.
• Ministry of Foreign Affairs / visa applications: the government’s Rijks ICT Gilde ran a quantitative bias test on the assessment of short-stay visa applications and found a significant difference based on nationality — recommending that the use of profile scores and risk groups be discontinued entirely.

Both cases illustrate the same lesson: bias is rarely sitting in an explicit “race” or “nationality” variable. It’s in the patterns a model distills from data.

Step 1: Analyze Whether Bias Is Present

Step 2: Carry Out a Justification Test

An observed difference isn’t automatically prohibited — but it always requires an explicit test against four sub-questions:

1. Does the algorithm pursue a legitimate aim?
2. Is the algorithm suitable for achieving that aim?
3. Is it necessary — are there no reasonable, less intrusive alternatives?
4. Is it, all things considered, proportionate?

For direct discrimination, there’s only room for an exception if the law itself provides one. For indirect discrimination, an objective justification can also suffice — but only if all four questions above are answered affirmatively. Without a valid justification, an observed difference is by definition prohibited discrimination, and the algorithm may not be used. The Netherlands Institute for Human Rights has published a detailed Risk Profiling Assessment Framework specifically addressing discrimination on grounds of race and nationality.

Step 3: Carry Out an Ethical Desirability Test

This is the step that goes beyond the legal question. Even with a valid, objective justification, an observed difference can still be ethically undesirable. Deliberately involve a broad group of stakeholders here — not just lawyers and data scientists, but also the people the algorithm actually affects — and discuss explicitly: what are the possible adverse effects, do we consider this fair, and are there alternatives? As the Algorithm Framework itself notes, this weighing can ultimately become a political-administrative question — make sure that responsibility is then explicitly assigned there.

A Concrete Example: Bias Is Already Baked Into the Building Blocks (Word2Vec)

Everything above is about how you measure bias in an algorithm’s output. But bias often creeps in much earlier — in the language representations a model is built on. A tangible, hands-on illustration of this comes from a tutorial on document representation, and it’s exactly why “measurement bias” and “historical bias” (mentioned in Step 1) aren’t abstract — they’re literally visible in the numbers of a word vector.

Word2Vec and GloVe learn dense word vectors based on the distributional hypothesis: a word is characterized by the company it keeps (Firth, 1957). Semantically similar words end up close together in vector space — which works remarkably well for analogies like king - man + woman ≈ queen. But the same mechanism also picks up societal stereotypes from the training corpus, with concrete and recognizable consequences:

man : programmer :: woman : ?

Query a GloVe model (trained on Wikipedia plus news articles) with this, and the terms that surface clearly reflect a gender stereotype rather than a neutral translation of the profession. The same happens with man:doctor :: woman:? and father:doctor :: mother:? — the model hasn’t learned what a doctor is; it has learned which words tend to co-occur in text that’s already saturated with societal assumptions.

Why this doesn’t just sneak in but actually stays in: these embeddings are trained on huge amounts of web text, news articles, and books — sources that reflect historical and societal prejudice. When such embeddings then end up in a downstream application — a CV-screening tool, a chatbot, a risk-assessment instrument — they can pass on and amplify that discrimination. A résumé screener running on GloVe embeddings that associates “programmer” more strongly with “man” than with “woman” will systematically rank male candidates higher, without “sex” ever being used as an explicit input variable. This is exactly the type of indirect discrimination described in Step 1 above — only arising at the representation level, deeper in the pipeline than most bias audits typically look.

How to Operationalize This in an AI/Agent Project

For anyone looking to operationalize this within a larger agentic AI project — as covered in the earlier post on building sovereign agents and the governance framework post on this site:

1. Document the three steps as a standing part of your validation phase — not as a one-time check at delivery, but as a recurring process, since both data and populations change over time.
2. Choose your fairness metrics before you see the results, together with stakeholders — otherwise the temptation to pick, after the fact, whichever metric happens to look favorable becomes hard to resist.
3. Build the bias test in as a separate, repeatable step in your pipeline — using one of the open-source tools mentioned above — the same way you’d build a fact-checking or guardrail module.
4. Document every judgment call explicitly, including why a given difference was or wasn’t found to be justified. This is precisely what a regulator — or a court, if it ever comes to that — will want to see later.
5. Keep monitoring after go-live — through the complaints and objections process, and through periodic repetition of the analysis, especially whenever the underlying population or the model changes.

The Takeaway

Bias analysis isn’t a one-time checklist — it’s a repeated process of measuring, explaining, justifying, and, where necessary, correcting course. The Dutch cases covered here show this isn’t a theoretical exercise: profiles that ran in production for years turned out, on closer inspection, to be insufficiently substantiated and were discontinued. Build these three steps in from the design phase of your algorithm — not as a reaction to an incident after the fact.

Source: Algorithm Framework (Algoritmekader), Dutch Ministry of the Interior and Kingdom Relations, measure 5-VER-03.

Key papers

• Bolukbasi et al. (2016), Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings — arXiv:1607.06520
• Gonen & Goldberg (2019), Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases — arXiv:1903.03862
• Hardt et al. (2016), Equality of Opportunity in Supervised Learning — arXiv:1610.02413
• Barocas, Hardt & Narayanan, Fairness and Machine Learning — fairmlbook.org

Further reading on this site

• Governing the Autonomous Organization — the four-phase governance framework that bias testing is part of
• What It Takes to Build a Sovereign AI Agent — step 7 of the guide deals with bias validation in practice
• Adversarial Attacks: Invisible Pixels and Swapped Words — the related category of security vulnerabilities in model inputs
• The Institutional Moat — why guardrails and compliance architecture matter for institutions
• Why NLP Finally Works — the transformer architecture whose embeddings carry the representational bias discussed here