Freshworks removed Karpenter’s dependency on CoreDNS across 400+ EKS clusters. The fix was one taint and one dnsPolicy line

TL;DR: Freshworks runs Karpenter across 400+ EKS clusters. The Karpenter controller node was also running CoreDNS because both components shared the same CriticalAddonsOnly toleration so the bootstrap chicken-and-egg would work. That co-location caused nf_conntrack exhaustion (peaks at 120K entries, 91.55% of the limit), CPU throttling, and a circular DNS dependency where Karpenter's own DNS resolution depended on the CoreDNS instance sharing its node. The fix was a Karpenter-only taint and a single dnsPolicy change. nf_conntrack on controller nodes dropped to a flat band under 800 entries (under 1% of the limit), and the change is expected to reduce Karpenter controller spend by about 35% as it rolls out fleet-wide.

The bootstrap problem. Karpenter needs a node to run on before it can provision nodes for anyone else. Freshworks' internal Kubix platform (Terraform-backed) creates a dedicated Karpenter managed node group as part of cluster provisioning. The ASG launches the first EC2 nodes, registers them with the cluster, and Kubernetes schedules the Karpenter controller onto one of them. But for the Karpenter pod to become Running and Ready, CoreDNS had to be available too. So Karpenter could not actually start provisioning nodes until CoreDNS was up, which only happened after the bootstrap node existed. Classic chicken-and-egg.

Why they shared the node. They gave Karpenter and CoreDNS the same toleration (CriticalAddonsOnly=true:NoSchedule) against the same taint on the Karpenter node group. Whichever pod came up first, both could land on that first node. Practical fix for a real problem, and it worked.

Why it bit them later. Once the cluster was running normally, the same design let CoreDNS keep living on the Karpenter controller node permanently. Controller nodes were intentionally small (c6g.large or c6g.xlarge), and now two workloads were fighting over one small node. nf_conntrack exhaustion, CPU throttling, and resource contention between Karpenter and CoreDNS. Karpenter's own DNS resolution (dnsPolicy: ClusterFirst) depended on the very CoreDNS instance sharing its node. One bad neighbor, and Karpenter's ability to scale the whole cluster was on the line.

The fix. Instead of optimizing the shared-node setup, they removed the dependency.

Give Karpenter its own scheduling boundary. New taint, karpenter.sh/controller=true:NoSchedule. Only Karpenter gets the matching toleration. Controller node group is now truly dedicated, CoreDNS no longer schedules there. Remove the CoreDNS dependency. Karpenter deployment changed to dnsPolicy: Default, so it uses the node and VPC DNS configuration instead of relying on CoreDNS. Why this still works. Kubernetes automatically injects KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT into every pod, pointing at the API server's ClusterIP. With service links enabled (the default), Karpenter uses those directly and never needed CoreDNS for API connectivity. The trade-off: with dnsPolicy: Default, Karpenter can no longer resolve arbitrary in-cluster service names like *.svc.cluster.local. For Freshworks' use case that is fine because Karpenter's critical path is API connectivity, not service discovery.

CoreDNS itself is also protected. Once it is running on Karpenter-provisioned worker nodes, they added karpenter.sh/do-not-disrupt: "true" so Karpenter's own consolidation logic will not evict it. DNS is too foundational to leave to chance.

What it bought them, concretely:

Cheaper controller nodes. Once CoreDNS stops sharing the node, Karpenter alone does not need the headroom. Modeled to reduce Karpenter controller spend by ~35% as the same change extends across all 400+ clusters. Author is explicit that this is expected saving, not yet realized at fleet scale. No more nf_conntrack alerts. The contention that caused them is gone because there is nothing left to contend with. Before: sharp repeated spikes, peaking at 120K entries and 91.55% of the connection-tracking limit. After: flat low band, under ~800 entries, well under 1% of the limit, no spikes. A CoreDNS blip can no longer take down node provisioning. Karpenter talks to the API server directly, so it does not inherit CoreDNS's bad days. CoreDNS, protected by do-not-disrupt, no longer inherits Karpenter's consolidation decisions either. One pattern, rolled out 400+ times. It is a taint and a dnsPolicy line, not a migration, so it standardizes across the entire fleet instead of staying a one-cluster fix. What they would tell another platform team:

Bootstrap requirements are not forever. A component may need a dependency to get started, but that does not mean it needs it for life. Ask periodically whether yesterday's bootstrap shortcut is today's steady-state liability. Make "dedicated" mean dedicated. If a node group is meant for one workload, give that workload its own taint and toleration. Do not let two components share a toleration just because they once needed to share a node. Small fixes compound at fleet scale. At 400+ clusters, a change that saves a little per cluster and rolls out uniformly beats a one-off optimization every time.