Quick context for anyone new here: I've been building CacheVerifier, testing whether a verifier model in front of a semantic cache (the vector-similarity kind of LLM cache, not exact match) actually catches bad cache hits before they get served. The whole thing started from one concrete failure, a cache returning the "pause your subscription" answer to "cancel my subscription" at 0.87 similarity, confidently wrong. Over the last couple of months the comments on these posts kept pointing at specific ways that failure generalizes: same topic, flipped action word (cancel vs pause). Same verb, flipped object ("convert A to B" vs "convert B to A"). Negation. I'd tested the verifier a lot on natural benchmark data, where those pairs turn up by accident, but never on pairs built on purpose to land on those exact axes.
So I finally built the red-team. Used an LLM (DeepSeek) to generate adversarial query pairs on five failure axes I already had evidence for: negation, action-word swap, direction reversal, named-entity swap (the "George H.W. vs George W. Bush" kind), and quantity swap. Each item is a pair of queries plus an answer that's only correct for the second one. Then I encoded both queries with the same embedding model my pipeline actually uses and kept only the pairs that clear the 0.80 similarity gate, since that's the real precondition for a pair to ever reach the verifier in production. 306 of 400 generated pairs qualified.
One method note, because it surprised me: a single round of this (15 to 20 per category) was uselessly noisy. Same unmodified verifier, two back-to-back trial runs, overall false-approve rate swung from 58% to 85%. I had to run four independent rounds of 20 per category and bootstrap a confidence interval to get a number I'd trust.
Result: the off-the-shelf cross-encoder verifier waves through 84% of these pairs (95% CI 80 to 88%). Negation is the worst blind spot at 96%, quantity swap also 96%. One of the highest-confidence wrong approvals: the query is "can I use this gift card on sale items," the candidate answer says essentially "no, you can't," and the verifier scores that pairing 11.32 against a decision threshold of 0. Not a borderline miss. A confident approval of a literally opposite answer. That tracks with what these models were trained for (MS MARCO passage ranking, never optimized for polarity), but the size of the number was still worse than I expected.
Here's the part that actually changed how I think about this. In an earlier post I wrote up how domain fine-tuning, training the same verifier on a dataset's own gray-zone labels, turned my worst-performing dataset from a net loss into 53 wins out of 54 tested operating points against just tuning the threshold. I'd been mentally filing that as "the fix." Same procedure applied to the conversational dataset is the checkpoint I red-teamed here. On the adversarial set it scores 88%, against the off-the-shelf 84%. The confidence intervals almost entirely overlap. Fine-tuning did nothing here. Whatever it learns from real gray-zone feedback, it's the shape of that dataset's natural noise, not robustness to someone constructing a hard pair on purpose. Those turn out to be two different capabilities and one doesn't buy you the other.
Tried the obvious next thing. Generated a second, separate batch of adversarial pairs (223 triples, non-overlapping with the test set), turned them into 446 training rows, mixed them into the 11,271 natural gray-zone training rows so adversarial data is 3.8% of the set, and reran the exact same fine-tune with one variable changed. Adversarial false-approve dropped from 84 to 88% down to 54% (CI 48 to 59%, no overlap with either earlier model). Natural-data AUC was untouched, 0.875 versus 0.88 for the pure-natural fine-tune. Most axes improved hard: action-word swap 59% to 13%, negation 96% to 57%.
Two honest dents in that. One category, named-entity swap, got worse, 49% to 62%. It also had the fewest training triples (28), so it could be undertraining or negative transfer between categories, I don't know yet. And 54% still means more than half the adversarial pairs get through, versus single-digit-to-teens error on natural data. This is "meaningfully mitigated, nowhere near solved."
Bigger caveat: the attacks and the adversarial training data both come from the same model and the same prompt templates. So it's possible I've taught the verifier to recognize DeepSeek's particular style of adversarial pair rather than the underlying failure mode. Haven't tested with a second generator. All of this is one dataset too.
Where I landed: fine-tuning on real feedback and hardening against constructed inputs are separate problems, and the second one doesn't come free with the first. The capability gap is fixable, a tiny fraction of targeted data moved it a lot without costing natural-data performance, but "targeted" means you need a list of failure axes, and you mostly get that list from getting burned or from someone in a comment thread telling you. Curious how people doing verification or guardrail work think about this: is there a way to get adversarial coverage that isn't just whack-a-mole on failure axes you already know about, and has anyone found a clean way to tell "learned the failure mode" apart from "learned the generator's style" without maintaining a whole second red-team stack?
Full writeup with the per-category tables and the confidence intervals is in the repo, PAPER.md sections 5.18 and 5.19: https://github.com/imxinchengyou/CacheVerifier
Source: r/LLMDevs · by /u/Reasonable_Royal_621