My DNS tunneling detection rule was technically correct and still got killed in two days

Spent an afternoon building a DNS tunneling detection rule. The signal set was the standard one: subdomain length, character entropy on the label, the ratio of NXDOMAIN responses against a given parent domain, and how many unique subdomains show up under one registered domain in a short window. To sanity check it, I replayed a handful of tunneling samples generated with a common lab tool. The rule fired clean on every single one. Confident, I shipped it to production that same afternoon.

By evening it had paged on-call three times. None of the three was tunneling. One was a corporate VPN client's routine health-check traffic, which turns out to generate long, high-entropy-looking subdomains as a completely normal part of how it phones home. Structurally, to the rule, it looked almost identical to a slow-drip exfil channel. By the second day nobody on the rotation trusted the alert anymore, and someone disabled the rule rather than keep getting paged for a VPN client.

The signals in the rule weren't wrong. That's what made this annoying to accept. The mistake was validating it only against attack traffic. Lab samples prove a rule catches attacks. They tell you nothing about what your own environment's normal DNS noise looks like: CDNs, telemetry agents, VPN health checks, any SaaS product handing out per-session subdomains, and some of that noise shares the exact statistical shape tunneling detection is built to flag.

What I do differently now: before a rule ships, it runs against a pull of my own environment's real DNS logs, not a clean synthetic sample, actual production traffic, and it has to stay quiet on that traffic before it's allowed anywhere near production. Attack samples it should catch, benign logs it should ignore, both required before ship. That's the whole idea behind treating detection rules as something with a test suite instead of a one-off query you paste in and walk away from.

The other habit that stuck: writing the logic once and porting it correctly to whatever SIEM syntax is actually running, Sigma, Splunk SPL, Sentinel KQL, Elastic EQL, instead of writing four half-tested versions by hand and assuming they behave the same. They don't, and the gap between them is exactly where a translated rule quietly stops doing what the original did.

Codelivly's Detection Engineering Book for SOC Analysts (427 pages) is built around that discipline, the same rule tested and written across all four syntaxes rather than something you figure out by disabling your own rule mid-shift.