Why You Can’t Fine-Tune, Train, or Distill a Model Without XAI

fundamentals
governance
The moment you start fine-tuning, training, or distilling your own models, you inherit full responsibility for how they reason — not just what they output. Explainable AI is what lets you actually verify that.
Author

Jan Scholtes

Published

August 22, 2026

When you call a foundation model’s API, you’re borrowing someone else’s reasoning and someone else’s accountability. The moment you fine-tune it on your own data, distill it into a smaller student model, or train a specialized component from scratch — as covered across the earlier posts on this site about building sovereign agents and compressing models for local deployment — that borrowed accountability disappears. The model’s behavior is now your behavior. And a model that performs well on your validation set can still be reasoning in ways you’d never approve of if you could actually see them. This post covers what XAI (eXplainable AI) actually is, the concrete toolbox for looking inside a model, and why this stops being optional the moment you’re the one training it.

Three Words That Aren’t Synonyms

Practitioners often use “transparency,” “interpretability,” and “explainability” interchangeably, but they mean genuinely different things:

  • Transparency exists when the process that extracts model parameters from training data, and generates labels from test data, can be described and justified by the person who designed the approach.
  • Interpretability is the ability to comprehend the model and present the basis for its decision-making in a way a human can actually understand.
  • Explainability — a term without full field-wide consensus yet — is best understood as the set of features from an interpretable domain that contributed to a specific decision for a specific example.

The distinction matters practically: you can have a transparent process (you know exactly what algorithm you used and why) without having an interpretable model (you still can’t explain any individual decision it makes).

What Actually Counts as a Valid Explanation?

This is worth taking seriously as a genuinely human, social question, not just a technical one. A valid explanation is part of a social interaction — it has to be sound, cogent, convincing, and it has to build trust in the receiving party, delivered as a step-by-step transfer of knowledge they can actually follow. Beyond that, a good explanation needs to show the difference between possible outcomes (why this answer rather than that one), needs to be relevant to what the person actually wants to know, and — importantly — is allowed to be incomplete. Highlighting a few salient examples is often sufficient; an explanation doesn’t need to be exhaustively complete to be useful, and in fact often shouldn’t be.

The Fundamental Trade-off: Accuracy vs. Explainability

There’s a well-known, rough hierarchy of model families running from highly interpretable but often less powerful (decision trees, linear/logistic models, simple statistical models) to highly accurate but much harder to interpret (deep learning, ensemble methods, large neural networks). Three broad strategies exist to deal with this trade-off:

  1. Interpretable models — techniques that learn inherently structured, causal, interpretable models in the first place.
  2. Deep explanation — modified deep learning techniques specifically designed to learn explainable features as part of training.
  3. Model induction — techniques that infer an explainable model from any other model, treated purely as a black box.

Most of the practical tools covered below fall into the third category — they don’t require you to change your architecture, only to interrogate it after the fact.

The Practical XAI Toolbox

Attention visualization: BERTViz and exBERT

BERTViz provides an interactive way to visualize attention weights inside Transformer models (BERT, GPT-2, T5, and most Huggingface models). The head view shows attention from one token to another within a single layer, with line weight showing attention strength and color identifying which head. The model view gives a bird’s-eye view across the entire model — every layer and every head at once, letting you click into any cell for detail. exBERT goes further, combining attention visualization with contextual embedding search — letting you find other contexts in a corpus that produced similar embeddings for a given token, which is genuinely useful for exploratory analysis beyond just “which token attended to which.”

Feature attribution: gradient-based saliency and Integrated Gradients

Saliency methods score how important each input token was to a specific output. The mechanism: the model’s final hidden state gets projected onto its vocabulary, producing a score per possible next token; after selecting the generated token, you calculate the gradient of that selected logit with respect to every input token, back-propagating all the way to the input. The intuition is that the input token whose smallest change would produce the largest change in output is the token that mattered most.

In practice, using a library like Inseq with Integrated Gradients, you can literally watch this work: asking GPT-2 to complete “Heathrow airport is located in the city of” and seeing which input tokens the model actually leaned on to produce “London” — or, more usefully, feeding a movie review into a sentiment classifier and seeing precisely which words drove the “positive” classification. One striking demonstration: swapping a single word can flip the model’s sentiment prediction entirely, which is exactly the kind of brittleness saliency analysis exposes that a simple accuracy number never would.

Perturbation-based methods: LIME and SHAP

LIME (Local Interpretable Model-agnostic Explanations, Ribeiro, Singh & Guestrin, 2016) treats the model as a black box and asks: what happens to the prediction as I perturb the input? It generates a new dataset of perturbed samples plus the black-box model’s predictions on them, then trains a simple, interpretable local model (often linear) weighted by proximity to the original instance. This works across tabular data, text, and images alike, and its strength is producing short, human-friendly, contrastive explanations. Its real weaknesses are worth naming honestly: defining the right “neighborhood” for perturbation is an unsolved problem, explanations can be unstable (rerun the sampling and you can get a different-looking explanation for the same instance), and — most concerning for anything compliance-adjacent — LIME explanations can be deliberately manipulated to hide bias. This is why LIME suits lay-audience or debugging use well, but is not sufficient on its own anywhere you’re legally required to fully justify a specific decision.

SHAP (SHapley Additive exPlanations) takes a more mathematically principled route, borrowing Shapley values from cooperative game theory: each feature is treated as a “player,” and its contribution is its average marginal effect across every possible combination (coalition) of features. A SHAP plot typically shows the model’s average prediction across a dataset as a baseline, then shows exactly how much each individual feature pushed the prediction up or down from that baseline for one specific instance. The honest trade-off against LIME: SHAP is computationally far more expensive (it has to consider a combinatorial number of feature coalitions), but it comes with a much stronger theoretical guarantee of consistency.

Looking past attention: neuron activations and the streetlight fallacy

Here’s a genuinely important and under-appreciated point. The feed-forward network (FFN) sitting on top of the self-attention block in a Transformer holds roughly two-thirds of all the model’s parameters — it’s the primary store of the model’s learned capacity. Yet the overwhelming majority of popular NLP explainability tools (BERTViz, self-attention heatmaps) focus almost exclusively on the attention mechanism, largely because attention weights are easy to visualize as lines connecting words. That’s worth naming as what it actually is: a version of the streetlight fallacy — searching where the light is easy to shine, rather than where the computation actually lives. Techniques like Non-negative Matrix Factorization applied to neuron activations, and tracking how the hidden state evolves layer by layer, are the tools that actually look at where most of the model’s capacity sits — and they matter precisely because attention-only analysis can give you a comfortable but incomplete picture of what’s really driving a decision.

Why This Stops Being Optional When You Train the Model Yourself

Everything above is useful for understanding any model, including one you’re only calling through an API. It becomes a genuine requirement — not a nice-to-have — the moment you’re fine-tuning, distilling, or training a component yourself, for several concrete reasons.

Fine-tuning and distillation can change how a model decides without changing whether it gets the right answer. As covered in the earlier post on pruning, quantization, and distillation, a distilled student model is trained to match a teacher’s softened output distribution — but matching the output distribution doesn’t guarantee the student arrived there via the same reasoning path. A validation accuracy score that looks identical before and after distillation can hide a genuinely different — and possibly less trustworthy — decision process underneath. XAI tools are how you actually check this: run SHAP or saliency analysis on the same set of examples before and after distillation, and compare not just the predictions but which tokens or features drove them.

This connects directly to the “faithfulness versus plausibility” trap. A valid human explanation, as covered above, needs to be convincing and build trust — but a method like LIME is specifically built to produce a simplified, human-palatable local approximation. That raises an uncomfortable question worth sitting with honestly: when you check your fine-tuned model’s explanations and they look reasonable, are you verifying that the model actually reasoned that way, or are you just generating a comforting post-hoc rationalization that satisfies human psychology without reflecting the actual high-dimensional computation? This is exactly why a serious verification process for a self-trained model needs more than one XAI method — a saliency map, a SHAP explanation, and an attention visualization that all tell a consistent story is meaningfully stronger evidence than any single one alone.

Fine-tuning can introduce new adversarial brittleness that a clean validation set won’t reveal. As covered in the earlier post on adversarial attacks, a model fine-tuned on a narrower dataset can become more sensitive to small, meaning-preserving perturbations — a synonym swap or a minor typo. There’s a concrete detection method worth adopting as a standard step: run LIME on an example before an adversarial perturbation, apply the perturbation, then run LIME again. If the explanation shifts dramatically, that’s a direct signal of adversarial sensitivity your fine-tuning process introduced or failed to remove.

Regulatory and governance frameworks explicitly require this. As covered in the earlier posts on AI governance and bias analysis under the Dutch Algorithm Framework, being able to show why a specific decision was made — not just that the system performs well in aggregate — is often a legal requirement, not a courtesy. A fine-tuned or distilled model deployed in a public-sector, medical, or financial context needs to support exactly this kind of per-decision explanation, and that capability has to be built in and verified during training, not retrofitted after a complaint arrives.

A Practical Checklist for Anyone Fine-Tuning or Distilling Their Own Model

  1. Run attention visualization (BERTViz) before and after fine-tuning on a fixed set of representative examples, and look for attention patterns shifting toward unexpected tokens.
  2. Run SHAP or LIME on the same validation examples before and after distillation, and compare not just the predicted label but which features drove it — a matching label with a different explanation is a warning sign, not a pass.
  3. Use saliency or Integrated Gradients to spot brittleness, especially single-word substitutions that flip a prediction — these are exactly the failure modes a plain accuracy metric will never surface.
  4. Test explicitly for adversarial sensitivity introduced by training, using the before/after LIME comparison method — a stable model should show stable local explanations under small, meaning-preserving perturbations.
  5. Look past attention at neuron activations and hidden-state evolution, since two-thirds of a Transformer’s capacity sits in the feed-forward layers that most standard tooling ignores.
  6. Document all of this, the same way the earlier posts on governance and bias analysis argue for documenting fairness testing.

The Takeaway

Borrowing a foundation model through an API means borrowing someone else’s explainability obligations along with its capability. The moment you fine-tune it, distill it, or train a specialized component yourself — exactly the path this site has argued for repeatedly as the route to genuine sovereignty and specialization — that obligation transfers to you in full. A model that scores well on a held-out test set has told you what it does. Only XAI tells you how — and without that, you’re deploying a system you’ve built but genuinely can’t account for.

Based on “XAI for NLP” (Advanced Natural Language Processing course, Department of Advanced Computing Sciences).

Key papers

  • Ribeiro et al. (2016), “Why Should I Trust You?”: Explaining the Predictions of Any Classifier (LIME) — arXiv:1602.04938
  • Lundberg & Lee (2017), A Unified Approach to Interpreting Model Predictions (SHAP) — arXiv:1705.07874
  • Sundararajan et al. (2017), Axiomatic Attribution for Deep Networks (Integrated Gradients) — arXiv:1703.01365
  • Vig (2019), A Multiscale Visualization of Attention in the Transformer Model (BERTViz) — arXiv:1906.05714

Further reading on this site