Why Agents Hallucinate Citations

Why does my agent hallucinate citations, and how do I catch it automatically?

Your agent just told a customer that Berkshire Hathaway's CEO is Warren Buffett. It said it confidently. It even produced a citation — a real article from a reputable outlet, with a working URL. The citation was real; the fact was stale. Greg Abel had held the job for months. Nobody noticed until the customer did. That incident is the shape of the problem: agents fail confidently, and the failure hides behind a plausible-looking source.

The mechanism

The mechanism is simple, and it explains why the citation was real while the fact was wrong. Retrieval and generation are separate acts. The model does not check the source at generation time; it writes the citation shape it has seen in training — a title, a URL, a date — and then fills in the fact from its own parametric memory. That memory is a snapshot, not a live query. The URL is real because the model has seen that URL in its training data, but the fact it attaches to that URL is whatever the model believes, not what the source actually says. A real URL does not mean a checked fact. The model is not lying; it is pattern-matching. It has learned that citations accompany confident claims, so it produces both. The citation is a stylistic artifact, not evidence.

The practice

Catching this automatically requires breaking that pattern. You cannot trust the model to grade its own answer — self-checking inherits the same blind spots. Instead, verify claims adversarially and externally. That means running two independent retrieval passes: one that gathers evidence for the claim, and one that gathers evidence against it. Then a dedicated refuter — a separate model instance whose only job is to kill the claim — examines both sets. The claim survives, gets refuted, or comes back UNVERIFIABLE when the evidence isn't there. An honest "I can't verify this" must outrank a confident guess. On our measured pilot, every fabricated or private claim we threw at the engine was refused, not graded.

But a verification engine is only as good as its calibration. "Confidence 0.9" means nothing unless 0.9-confidence verdicts are right ~90% of the time, measured on labeled data. Our production curve — measured on the retrieval stack that actually serves requests — shows top-bucket realized accuracy of 0.914 (n=116) at 27% abstention, with decidable-commit precision of 0.976 (121/124) across a 204-claim labeled set. That is the number we publish, because it is the stack that serves your requests.

And here is the part worth stealing: we also measured 0.909 (n=121) on our home research substrate — a different retrieval stack. We do not publish that number as our production curve, because a curve measured on one retrieval stack does not transfer to another. Swapping the gather implementation is enough to invalidate it. The keyless stack lands at 0.848 (n=79) — another reminder that the stack is part of the figure. Any verification vendor quoting a single accuracy figure without naming the stack it was measured on is quoting the best of several. Ask which one is serving your requests. When the engine hadn't earned a curve yet, it was hard-capped and the API served a labeled stub — no paid verdict without a measured curve. That gate is the rule we'd suggest for any verification you build or buy: ask the operator "what's your realized accuracy at the confidence you're quoting, and on what n?" If there's no number, it's vibes.

The close

You can build the first property yourself in a weekend: prompt one model instance to refute another's answer before you act on it, and treat "can't refute AND primary source found" as your bar. That alone catches the worst failures. What's expensive to build is the second property — labeled calibration sets and measured curves. That's the part we sell: one HTTP call, $0.05, paid by your agent itself over x402 — verify.clearsigned.com. It returns SUPPORTED / REFUTED / UNVERIFIABLE with evidence, confidence backed by the published curve, and a signed receipt. Or start free: check whether your own site is even legible to agents at clearsigned.com/scan. Run the scanner to see what agents see, or send one claim through the verification engine at clearsigned.com/verify. No hype — just a way to catch the stale fact before your customer does.