Kubernetes DNS Resolution Latency: CoreDNS Scaling, ndots:5 & conntrack Race Conditions

1. The Kubernetes DNS Resolution Path: resolv.conf & ndots:5 Query Amplification

Every Pod provisioned inside a Kubernetes cluster automatically inherits a customized DNS configuration mounted at /etc/resolv.conf by the Kubelet. For workloads operating at multi-thousand requests per second, this standard configuration generates severe DNS query amplification that saturates network interfaces and introduces non-deterministic latency spikes.

A typical Pod's /etc/resolv.conf appears as follows:

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local c.us-east-1.internal
options ndots:5

The ndots:5 Lookup Cascade Mechanism

The standard C library (glibc) resolver interprets ndots:5 as: "If a query string contains fewer than 5 dots, query the domain by appending each entry in the search path sequentially before attempting an absolute lookup on the raw string."

Consider an application resolving an external endpoint, such as api.stripe.com (which contains 2 dots):

1. api.stripe.com.default.svc.cluster.local. → NXDOMAIN (Sent to CoreDNS)
2. api.stripe.com.svc.cluster.local. → NXDOMAIN (Sent to CoreDNS)
3. api.stripe.com.cluster.local. → NXDOMAIN (Sent to CoreDNS)
4. api.stripe.com.c.us-east-1.internal. → NXDOMAIN (Sent to Upstream/VPC DNS)
5. api.stripe.com. → NOERROR / A Record (Final resolution)

2. Linux Netfilter conntrack Race Conditions: The 5-Second DNS Timeout Mystery

One of the most elusive anomalies in Kubernetes clusters is the periodic occurrence of exact 5000ms (5-second) DNS query timeouts in microservice logs, despite CoreDNS running at sub-5% CPU utilization.

The Dual-Packet Race in Netfilter / iptables

When an application attempts a domain lookup, glibc dispatches the A and AAAA UDP queries concurrently over two separate local sockets sharing the same source and destination IP. When these two UDP packets hit the Linux kernel Netfilter stack:

• Step 1 (Tuple Creation Race): Packet 1 (A query) and Packet 2 (AAAA query) execute through Netfilter hooks. Both attempt to insert new connection tracking entries into the kernel nf_conntrack hash table simultaneously before SNAT/DNAT rules complete.
• Step 2 (Insertion Conflict): One packet successfully claims the conntrack tuple, while the competing packet experiences an insertion conflict inside __nf_conntrack_confirm.
• Step 3 (Silent Packet Drop): The Linux kernel drops the conflicting UDP packet without notifying user-space or returning an ICMP error.
• Step 4 (glibc 5s Timeout): The glibc resolver waits for the missing response until its default retransmit timeout—exactly 5 seconds—expires before retrying.

3. CoreDNS Architecture: Plugin Chain, Memory Profiling & Autoscaling Math

CoreDNS compiles as a modular, lightweight DNS server written in Go. Inbound queries traverse a strict, linear pipeline of enabled plugins defined inside the Corefile:

.:53 {
    errors
    health {
        lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
        ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf {
        max_concurrent 1000
    }
    cache 30 {
        success 10000
        denial 5000
    }
    loop
    reload
    loadbalance
}

CoreDNS Sizing Formula for High QPS

To prevent CoreDNS OOM kills and CPU throttling under query bursts, calculate memory and replica boundaries using cluster scale metrics:

$$\text{Memory (MB)} = (\text{Total Services} \times 0.005) + (\text{Total Pods} \times 0.002) + 128\text{ MB Baseline}$$

$$\text{Replica Count} = \max\left(2, \frac{\text{Cluster Nodes}}{8}, \frac{\text{Estimated Cluster DNS QPS}}{4000}\right)$$

4. NodeLocal DNSCache: DaemonSet Architecture & iptables NOTRACK Bypasses

The definitive production architectural solution to eliminate both the 5-second conntrack race condition and CoreDNS centralization bottlenecks is deploying NodeLocal DNSCache.

NodeLocal DNSCache runs a lightweight DNS caching agent as a Kubernetes DaemonSet on every worker node, listening on a dedicated local loopback IP (typically 169.254.20.10):

• Local Cache Hit: Resolves cached queries directly on the node with sub-millisecond latency (0.2ms vs 4.5ms over cluster network).
• TCP Transport to CoreDNS: Cache misses are forwarded to the centralized CoreDNS cluster over TCP instead of UDP, completely bypassing Netfilter conntrack UDP race conditions.
• iptables NOTRACK: Inbound traffic to the link-local IP is marked with NOTRACK, avoiding connection state table allocations entirely.

5. Production Incident Postmortem: Cascading External API Latency Collapse

Incident Context

During a high-concurrency payment processing window, checkout APIs experienced p99 latency degradation from 85ms to 5,200ms. Database metrics and payment provider gateways showed zero latency increases, but Go microservice logs were filled with dial tcp: lookup api.stripe.com on 10.96.0.10:53: i/o timeout.

Root-Cause Investigation Sequence

1. CoreDNS Load Inspection: CoreDNS pods had saturated their CPU resource limits (500m), entering severe CPU CFS throttling.
2. Query Distribution Analysis: Query logs showed over 82% of all DNS queries were invalid lookups matching api.stripe.com.default.svc.cluster.local, directly caused by ndots:5.
3. Netfilter Drop Corroboration: Running nstat -az | grep -i drop on worker nodes showed rapidly incrementing IcmpInErrors and conntrack drops matching the 5-second timeout window.
4. Mitigation Strategy: Deployed Fully Qualified Domain Names (FQDNs) with trailing dots (api.stripe.com.) in application endpoints and rolled out NodeLocal DNSCache across all worker pools.

6. Production Kubernetes Manifests: Corefile, HPA & NodeLocal Deployment

1. Pod DNSConfig Optimization (Bypassing ndots:5)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
spec:
  template:
    spec:
      dnsConfig:
        options:
          - name: ndots
            value: "2"
          - name: single-request-reopen
          - name: timeout
            value: "1"
          - name: attempts
            value: "3"
      containers:
        - name: app
          image: payment-service:v2.4.0

2. Cluster Proportional Autoscaler for CoreDNS

apiVersion: apps/v1
kind: Deployment
metadata:
  name: coredns-autoscaler
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: coredns-autoscaler
  template:
    metadata:
      labels:
        k8s-app: coredns-autoscaler
    spec:
      containers:
        - name: autoscaler
          image: registry.k8s.io/cpa/cluster-proportional-autoscaler:v1.8.8
          resources:
            requests:
              cpu: "20m"
              memory: "32Mi"
          command:
            - /cluster-proportional-autoscaler
            - --namespace=kube-system
            - --configmap=coredns-autoscaler
            - --target=Deployment/coredns
            - --default-params={"linear":{"nodesPerReplica":16,"coresPerReplica":256,"min":2,"max":100}}
            - --logtostderr=true
            - --v=2

7. Failure Mode and Effects Analysis (FMEA Matrix) for Cluster DNS

8. Engineering FAQ & IETF RFC DNS Protocol Specifications

Q1: Why does appending a trailing dot (e.g., `api.stripe.com.`) solve the ndots:5 lookup issue?

Specification: RFC 1034 & RFC 1035 (Domain Concepts and Facilities).
Technical Explanation: In DNS nomenclature, a trailing dot designates a Fully Qualified Domain Name (FQDN) anchored to the DNS root zone. When the glibc resolver encounters a trailing dot, it recognizes the domain as absolute and bypasses all local search path expansions configured in /etc/resolv.conf, dispatching a single authoritative lookup immediately.

Q2: What is the exact function of the `single-request-reopen` resolv.conf option?

Technical Explanation: By default, glibc dispatches A and AAAA queries using the same local socket. When single-request-reopen is enabled, glibc closes the socket and opens a brand-new local socket with a distinct source port for the subsequent query. This prevents the Linux kernel from processing two simultaneous UDP packets matching the same conntrack tuple, eliminating the 5-second race condition.