Adversarial Attacks: How a Few Invisible Pixels or One Swapped Word Can Fool an AI

fundamentals
efficiency
Adversarial examples exploit the exact mathematical boundaries a model draws to make decisions. Here’s what these attacks actually look like in images and text, and the concrete defenses that address each one.
Author

Jan Scholtes

Published

August 29, 2026

A picture of a panda, correctly classified by a neural network with 57% confidence, gets a tiny amount of carefully calculated noise added to it — invisible to a human eye, the image still looks exactly like a panda — and the same network now classifies it as a gibbon with over 99% confidence. This isn’t a hypothetical; it’s one of the founding demonstrations of what’s called an adversarial attack, and the underlying vulnerability turns out to affect text models just as much as image models, if you know where to push. This post covers what these attacks actually look like in both domains, and what genuinely works as a defense — following on from the previous post on model compression, which turns out to be part of the answer, but far from all of it.

Why This Is Possible at All

Every trained model draws a decision boundary through an extremely high-dimensional space, separating “panda” from “gibbon,” or “positive review” from “negative review.” That boundary is learned from data, and it’s shaped by whatever correlations happen to exist in the training set — not necessarily by the same features a human would use to make the same distinction. An adversarial attack works by finding the shortest possible path across that boundary: a tiny, deliberately calculated perturbation to the input that pushes it just barely to the other side, without changing anything a human would notice. The vulnerability isn’t a bug in one particular model — it’s a structural consequence of how these decision boundaries get learned in the first place, which is why the problem shows up across model types and domains.

Adversarial Attacks on Images

Gradient-based perturbation attacks. The classic approach — the Fast Gradient Sign Method and its more iterative successor, Projected Gradient Descent — computes the gradient of the model’s loss with respect to the input pixels rather than the model’s weights, and then nudges every pixel by a tiny amount in whichever direction increases the loss (i.e., makes the model more wrong) the fastest. Because the perturbation is spread across the entire image and kept within a small bound, it’s imperceptible to a human viewer while being precisely calibrated to cross the model’s decision boundary.

Patch attacks. Rather than perturbing an entire image subtly, a patch attack places one small, highly visible, often strange-looking sticker or region somewhere in the frame — on a stop sign, on a piece of clothing, in the corner of a photo — that reliably causes a targeted misclassification regardless of everything else in the image. Unlike gradient-based perturbations, these are often robust to being photographed at different angles and distances, which is exactly what makes them a practical physical-world concern (a modified stop sign that a self-driving system misreads as a speed limit sign, for instance) rather than just a digital curiosity.

Black-box and transfer attacks. Both attack families above assume the attacker has access to the model’s internals to compute gradients. In practice, many real attacks don’t need that: adversarial examples crafted against one model frequently transfer and fool a different model trained on similar data, even without access to that second model’s weights at all. This is what makes the threat practical rather than purely academic — an attacker doesn’t need to steal your exact model to attack it.

Adversarial Attacks on Text

Text is discrete — you can’t nudge a word by 0.1% the way you can nudge a pixel’s brightness — so text-based adversarial attacks look structurally different, but the underlying goal is identical: change the input as little and as unnoticeably as possible while flipping the model’s output.

Character-level perturbations. Swapping visually similar characters, inserting or deleting a character, or introducing common typos can flip a classifier’s output while remaining perfectly readable to a human — “excellent” becomes “excel1ent” or “exceellent,” and a sentiment classifier’s confidence collapses even though no reasonable reader would misread the word.

Word-level substitution attacks. A more sophisticated family of attacks replaces individual words with close synonyms — chosen specifically to preserve the sentence’s actual meaning to a human reader while shifting it just enough in the model’s embedding space to cross a decision boundary. Because the substitution is semantically faithful, these are harder to catch with a simple typo filter and harder to dismiss as “obviously broken input.”

Universal adversarial triggers. Rather than crafting a bespoke perturbation for each individual input, some attacks discover a short sequence of words or tokens that, when prepended or appended to almost any input, reliably pushes the model toward a specific wrong output — a kind of reusable “skeleton key” for a given model, rather than a one-off attack tailored to a single sentence.

Prompt injection (the LLM-specific version). In agentic systems specifically, an increasingly relevant variant is prompt injection: text embedded in a document, webpage, or tool output that a language model reads as part of its context — not something the actual user typed — but that’s crafted to look like an instruction the model should follow. This is structurally the same underlying vulnerability as classic adversarial text attacks (a carefully chosen input pushes the model somewhere it shouldn’t go), applied to the specific weakness of a system that can’t always distinguish “data I’m reading” from “instructions I should obey.”

What Actually Works as a Defense

No single technique fully solves this, which is why real deployments layer several together:

Adversarial training. The most direct defense: deliberately generate adversarial examples during training and include them in the training set, so the model learns a decision boundary that isn’t so easily crossed by small perturbations. This can be extended with broader data augmentation — random noise, hand-crafted negations, synonym substitution, and general training-set diversity — so the model doesn’t overfit to a narrow, brittle notion of what “normal” input looks like.

Defensive distillation and feature squeezing. Training a smaller student model on a teacher’s softened output distribution tends to produce a smoother, less sharply-creased decision boundary that’s harder to exploit with a tiny, precisely-calculated nudge. Feature squeezing works in a related but more direct way: it reduces the input’s degrees of freedom before the model ever sees it — for images, this might mean reducing color depth or applying mild smoothing; for text, normalizing spelling variants and character substitutions before classification — specifically to collapse the fine-grained space an attacker needs to operate in.

Input transformations and randomization. Techniques like slight image compression, cropping, or resizing before classification can disrupt the very precise pixel-level calculations an adversarial perturbation depends on, without meaningfully affecting a genuine image. On the text side, spelling normalization and canonicalization serve a similar purpose against character-level attacks.

Ensembles. Running the same input through multiple independently trained models and requiring agreement (or flagging disagreement) raises the bar significantly for an attacker, since a perturbation crafted to fool one specific model’s particular decision boundary often doesn’t transfer cleanly to a differently-trained model with a different boundary.

Detection rather than pure prevention. Rather than trying to make a model unconditionally robust to every possible perturbation, a separate detection layer can flag inputs that look statistically unusual — oddly high-frequency noise patterns in an image, unusual character sequences in text — and route them for additional scrutiny before they ever reach a decision.

For agentic systems specifically: input guarding and deterministic guardrails. Because prompt injection exploits the boundary between “data” and “instructions,” the most effective defense isn’t asking the language model to police itself — it’s routing untrusted input (documents, web content, tool outputs) through a separate, deterministic check before it reaches the main reasoning model, and treating anything that looks like an embedded instruction as suspicious by default rather than as a legitimate command.

The Common Thread

Every one of these defenses is really doing the same underlying thing: making the model’s decision boundary smoother, less sensitive to tiny or narrowly-targeted changes, and less trusting of any single unverified signal. None of them make a model perfectly unbreakable — this remains a genuinely active arms race between attack and defense research, on both images and text — but layering several of these together (adversarial training plus distillation plus input normalization plus detection) meaningfully raises the cost and difficulty of a successful attack, which in most real deployments is exactly the practical goal.

Key papers

  • Goodfellow et al. (2014), Explaining and Harnessing Adversarial Examples (FGSM) — arXiv:1412.6572
  • Madry et al. (2017), Towards Deep Learning Models Resistant to Adversarial Attacks (PGD) — arXiv:1706.06083
  • Wallace et al. (2019), Universal Adversarial Triggers for Attacking and Analyzing NLP — arXiv:1908.07125

Further reading on this site