Module 11 — Networking Model & DNS¶

Overview¶

Kubernetes networking can seem complex, but it's built on a few simple rules. This module covers the networking model (pod-to-pod, pod-to-service), how CNI plugins implement it, and how CoreDNS provides service discovery. All three are tested on the CKA exam.

1. Kubernetes Networking Fundamentals¶

1.1 The Four Networking Requirements¶

Every Kubernetes cluster must satisfy these rules:

#	Rule
1	Every pod gets its own unique IP address
2	Pods on any node can communicate with pods on any other node without NAT
3	Agents on a node (kubelet, kube-proxy) can communicate with all pods on that node
4	Pods in the host network of a node can communicate with all pods on all nodes without NAT

Key insight: Kubernetes does NOT implement networking itself. It defines the rules and delegates implementation to a CNI plugin.

1.2 IP Address Ranges¶

A cluster uses three distinct CIDR ranges:

Range	Purpose	Example	Configured by
Node network	Physical/VM node IPs	`192.168.1.0/24`	Infrastructure
Pod network (pod CIDR)	Pod IPs	`10.244.0.0/16`	`--pod-network-cidr` at init
Service network (service CIDR)	ClusterIP addresses	`10.96.0.0/12`	`--service-cidr` at init

# Find pod CIDR
kubectl cluster-info dump | grep -m1 cluster-cidr
# Or check controller-manager
cat /etc/kubernetes/manifests/kube-controller-manager.yaml | grep cluster-cidr

# Find service CIDR
cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep service-cluster-ip-range

These ranges must NOT overlap with each other or with the node network.

1.3 Pod-to-Pod Communication¶

Same Node¶

Pods on the same node communicate through a virtual bridge (cbr0, cni0, or similar):

┌─────────────────────────────────────────┐
│              Node 1                      │
│                                         │
│  ┌─────────┐       ┌─────────┐         │
│  │  Pod A   │       │  Pod B   │         │
│  │ 10.244.1.2│       │ 10.244.1.3│        │
│  └────┬─────┘       └────┬─────┘         │
│       │  veth pair        │  veth pair    │
│       └──────┬────────────┘              │
│              │                           │
│         ┌────▼────┐                      │
│         │  cni0   │  (virtual bridge)    │
│         │10.244.1.1│                      │
│         └─────────┘                      │
└─────────────────────────────────────────┘

Traffic stays local — no encapsulation needed.

Different Nodes¶

Pods on different nodes communicate through the CNI plugin's overlay or routing mechanism:

┌──────────────────┐                    ┌──────────────────┐
│     Node 1       │                    │     Node 2       │
│                  │                    │                  │
│  ┌─────────┐    │                    │    ┌─────────┐  │
│  │  Pod A   │    │                    │    │  Pod C   │  │
│  │10.244.1.2│    │                    │    │10.244.2.3│  │
│  └────┬─────┘    │                    │    └────┬─────┘  │
│       │          │                    │         │        │
│  ┌────▼────┐    │                    │    ┌────▼────┐  │
│  │  cni0   │    │                    │    │  cni0   │  │
│  └────┬────┘    │                    │    └────┬────┘  │
│       │          │                    │         │        │
│  ┌────▼────┐    │    VXLAN/BGP/     │    ┌────▼────┐  │
│  │  eth0   │────┼───  IP-in-IP  ────┼───│  eth0   │  │
│  │192.168.1.10│  │                    │  │192.168.1.11│  │
│  └─────────┘    │                    │    └─────────┘  │
└──────────────────┘                    └──────────────────┘

1.4 Pod-to-Service Communication¶

Pod A (10.244.1.2)
    │
    │  dst: 10.96.0.10:80 (ClusterIP)
    ▼
iptables/ipvs DNAT (programmed by kube-proxy)
    │
    │  dst rewritten to: 10.244.2.3:8080 (Pod IP)
    ▼
Pod C (10.244.2.3)

kube-proxy watches the API server for Service and Endpoint changes, then programs rules on every node:

kube-proxy mode	How it works
iptables (default)	Creates DNAT rules in the nat table; kernel handles packet forwarding
ipvs	Uses Linux IPVS (IP Virtual Server) for load balancing; better performance at scale

# Check kube-proxy mode
kubectl get configmap kube-proxy -n kube-system -o yaml | grep mode

# View iptables rules for a service
iptables -t nat -L KUBE-SERVICES -n | grep <service-name>
iptables -t nat -L KUBE-SVC-<hash> -n

# View ipvs rules
ipvsadm -Ln

1.5 External-to-Service Communication¶

External Client
    │
    │  dst: <NodeIP>:30080 (NodePort)
    ▼
Node iptables (KUBE-NODEPORTS chain)
    │
    │  DNAT to ClusterIP or directly to Pod IP
    ▼
Pod

2. CNI Plugins Overview¶

2.1 What Is CNI?¶

The Container Network Interface is a specification that defines how network plugins integrate with container runtimes. When a pod is created:

kubelet → CRI (containerd) → CNI plugin
                                │
                                ├── Allocates an IP from the pod CIDR
                                ├── Creates a veth pair
                                ├── Attaches one end to the pod, one to the bridge
                                └── Sets up routes/tunnels for cross-node traffic

2.2 CNI Configuration¶

# CNI config directory
ls /etc/cni/net.d/
# 10-calico.conflist
# or
# 10-flannel.conflist

# CNI binary directory
ls /opt/cni/bin/
# bandwidth  bridge  calico  calico-ipam  flannel  host-local  loopback  portmap ...

2.3 Calico¶

The most commonly used CNI in production and CKA exam environments.

Aspect	Detail
Routing modes	BGP (default), VXLAN, IP-in-IP
Network Policy	✅ Full support (the main reason to choose Calico)
Default pod CIDR	`192.168.0.0/16`
IPAM	Calico IPAM (block-based allocation per node)
Encryption	WireGuard (optional)
Performance	High — native routing with BGP avoids encapsulation overhead

# Check Calico pods
kubectl get pods -n kube-system -l k8s-app=calico-node

# Check Calico node status
kubectl exec -n kube-system calico-node-xxxxx -- calico-node -bird-ready

How Calico routes traffic (BGP mode):

Node 1 (10.244.0.0/24)              Node 2 (10.244.1.0/24)
┌──────────────────┐                ┌──────────────────┐
│  Pod 10.244.0.5  │                │  Pod 10.244.1.8  │
│       │          │                │       │          │
│  routing table:  │                │  routing table:  │
│  10.244.1.0/24   │───── BGP ─────│  10.244.0.0/24   │
│  via 192.168.1.11│                │  via 192.168.1.10│
└──────────────────┘                └──────────────────┘

No encapsulation — packets are routed natively using BGP-learned routes.

2.4 Flannel¶

The simplest CNI plugin — easy to set up, limited features.

Aspect	Detail
Routing mode	VXLAN (default), host-gw
Network Policy	❌ Not supported
Default pod CIDR	`10.244.0.0/16`
IPAM	host-local (subnet per node)
Encryption	None
Performance	Moderate — VXLAN adds encapsulation overhead

# Check Flannel pods
kubectl get pods -n kube-flannel
# or
kubectl get pods -n kube-system -l app=flannel

How Flannel routes traffic (VXLAN mode):

Node 1                              Node 2
┌──────────────────┐                ┌──────────────────┐
│  Pod 10.244.0.5  │                │  Pod 10.244.1.8  │
│       │          │                │       │          │
│  ┌────▼────┐     │                │     ┌────▼────┐  │
│  │ cni0    │     │                │     │ cni0    │  │
│  └────┬────┘     │                │     └────┬────┘  │
│  ┌────▼────┐     │                │     ┌────▼────┐  │
│  │flannel.1│     │   VXLAN tunnel │     │flannel.1│  │
│  │(VTEP)   │─────┼── UDP 4789 ───┼─────│(VTEP)   │  │
│  └─────────┘     │                │     └─────────┘  │
└──────────────────┘                └──────────────────┘

Original packet is encapsulated in a VXLAN (UDP) header for cross-node transport.

2.5 Cilium¶

Modern, eBPF-based CNI with advanced features.

Aspect	Detail
Routing mode	VXLAN, native routing, eBPF-based
Network Policy	✅ Full Kubernetes + extended (L7, DNS-aware)
IPAM	Cluster-scope, multi-pool
Encryption	WireGuard, IPsec
Performance	High — eBPF bypasses iptables entirely
Observability	Built-in (Hubble)

# Check Cilium pods
kubectl get pods -n kube-system -l k8s-app=cilium

# Cilium status
kubectl exec -n kube-system cilium-xxxxx -- cilium status

2.6 CNI Comparison Summary¶

Feature	Calico	Flannel	Cilium
Network Policy	✅	❌	✅ (extended)
Routing	BGP / VXLAN / IP-in-IP	VXLAN / host-gw	VXLAN / native / eBPF
Encryption	WireGuard	❌	WireGuard / IPsec
Complexity	Medium	Low	High
Performance	High	Moderate	High
L7 Policy	❌	❌	✅
CKA relevance	Most common	Simple labs	Growing

CKA Tip: The exam usually has a CNI pre-installed. You may need to install one (the command is typically provided). Know that Flannel does NOT support NetworkPolicies — if the question involves NetworkPolicies, Calico or Cilium must be the CNI.

3. DNS in Kubernetes (CoreDNS)¶

3.1 How DNS Works in Kubernetes¶

CoreDNS runs as a Deployment in kube-system and provides DNS resolution for all Services and Pods:

Pod makes DNS query
    │
    ▼
/etc/resolv.conf (points to CoreDNS ClusterIP)
    │
    │  nameserver 10.96.0.10
    │  search default.svc.cluster.local svc.cluster.local cluster.local
    │
    ▼
CoreDNS (kube-dns Service, 10.96.0.10:53)
    │
    ├── Kubernetes plugin: resolves Service/Pod names
    │     my-svc.default.svc.cluster.local → 10.96.0.15
    │
    └── forward plugin: forwards external queries to upstream DNS
          google.com → upstream nameserver (e.g., 8.8.8.8)

3.2 DNS Records Created by Kubernetes¶

Service DNS¶

Record Type	Format	Example
A record (ClusterIP)	`<svc>.<ns>.svc.cluster.local`	`web.default.svc.cluster.local → 10.96.0.15`
A record (Headless)	`<svc>.<ns>.svc.cluster.local`	`web-headless.default.svc.cluster.local → 10.244.1.5, 10.244.2.8`
SRV record	`_<port-name>._<protocol>.<svc>.<ns>.svc.cluster.local`	`_http._tcp.web.default.svc.cluster.local`

Pod DNS (when enabled)¶

Record Type	Format	Example
A record	`<pod-ip-dashed>.<ns>.pod.cluster.local`	`10-244-1-5.default.pod.cluster.local → 10.244.1.5`
StatefulSet pod	`<pod-name>.<headless-svc>.<ns>.svc.cluster.local`	`mysql-0.mysql-headless.default.svc.cluster.local`

3.3 DNS Search Domains¶

Every pod's /etc/resolv.conf includes search domains that allow short names:

kubectl exec <pod> -- cat /etc/resolv.conf
# nameserver 10.96.0.10
# search default.svc.cluster.local svc.cluster.local cluster.local
# options ndots:5

This means you can use:

Query	Resolves via search domain
`web`	`web.default.svc.cluster.local` (same namespace)
`web.other-ns`	`web.other-ns.svc.cluster.local` (cross-namespace)
`web.other-ns.svc`	`web.other-ns.svc.cluster.local`
`web.default.svc.cluster.local`	Fully qualified (no search needed)

The ndots:5 option means any name with fewer than 5 dots is appended with search domains before querying. This is why short names work but can cause extra DNS lookups for external domains.

3.4 CoreDNS Components¶

# CoreDNS Deployment
kubectl get deployment coredns -n kube-system

# CoreDNS Pods
kubectl get pods -n kube-system -l k8s-app=kube-dns

# CoreDNS Service (named "kube-dns" for historical reasons)
kubectl get svc kube-dns -n kube-system
# NAME       TYPE        CLUSTER-IP   PORT(S)
# kube-dns   ClusterIP   10.96.0.10   53/UDP,53/TCP,9153/TCP

3.5 CoreDNS Configuration (Corefile)¶

CoreDNS is configured via a ConfigMap:

kubectl get configmap coredns -n kube-system -o yaml

.:53 {
    errors                          # log errors
    health {                        # health check endpoint on :8080
        lameduck 5s
    }
    ready                           # readiness endpoint on :8181
    kubernetes cluster.local in-addr.arpa ip6.arpa {   # Kubernetes plugin
        pods insecure               # enable pod DNS records
        fallthrough in-addr.arpa ip6.arpa
        ttl 30
    }
    prometheus :9153                # metrics endpoint
    forward . /etc/resolv.conf {   # forward external queries to upstream
        max_concurrent 1000
    }
    cache 30                        # cache DNS responses for 30 seconds
    loop                            # detect and stop forwarding loops
    reload                          # auto-reload Corefile on ConfigMap change
    loadbalance                     # round-robin DNS responses
}

Plugin	Purpose
`kubernetes`	Resolves Service and Pod DNS names from the Kubernetes API
`forward`	Forwards non-cluster queries to upstream DNS servers
`cache`	Caches DNS responses to reduce API server load
`errors`	Logs DNS errors
`health`	Liveness probe endpoint (`:8080/health`)
`ready`	Readiness probe endpoint (`:8181/ready`)
`loop`	Detects forwarding loops
`reload`	Watches ConfigMap for changes and reloads

3.6 Customizing CoreDNS¶

Add a Custom DNS Entry (Stub Domain)¶

Forward queries for example.com to a specific DNS server:

kubectl edit configmap coredns -n kube-system

.:53 {
    # ... existing config ...
    forward . /etc/resolv.conf
}
example.com:53 {                    # add a new server block
    forward . 10.0.0.53             # forward to custom DNS
}

Add Custom Hosts¶

.:53 {
    hosts {
        10.0.0.100 custom.example.com
        fallthrough
    }
    # ... rest of config ...
}

After editing, CoreDNS auto-reloads (thanks to the reload plugin). If not, restart the pods:

kubectl rollout restart deployment coredns -n kube-system

3.7 Pod DNS Configuration¶

Override DNS settings per pod:

apiVersion: v1
kind: Pod
metadata:
  name: custom-dns
spec:
  dnsPolicy: None                    # don't use cluster DNS
  dnsConfig:
    nameservers:
    - 8.8.8.8
    - 8.8.4.4
    searches:
    - my.custom.domain
    options:
    - name: ndots
      value: "2"
  containers:
  - name: app
    image: nginx

dnsPolicy	Behavior
`ClusterFirst` (default)	Use CoreDNS for cluster names, forward external to upstream
`Default`	Inherit DNS config from the node
`None`	Use only `dnsConfig` settings (must provide nameservers)
`ClusterFirstWithHostNet`	Like ClusterFirst, but for pods using `hostNetwork: true`

4. DNS Troubleshooting¶

4.1 Debugging DNS Resolution¶

# 1. Start a debug pod with DNS tools
kubectl run dnsutils --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3 \
  -it --rm --restart=Never -- sh

# Or use busybox (has nslookup)
kubectl run debug --image=busybox -it --rm --restart=Never -- sh

# 2. Test DNS resolution
nslookup kubernetes.default.svc.cluster.local
nslookup <service-name>
nslookup <service-name>.<namespace>.svc.cluster.local

# 3. Check /etc/resolv.conf
cat /etc/resolv.conf

# 4. Test external DNS
nslookup google.com

4.2 Common DNS Issues¶

Symptom	Cause	Fix
`nslookup: can't resolve` for all names	CoreDNS pods not running	`kubectl get pods -n kube-system -l k8s-app=kube-dns` — check logs
Service name resolves but external names don't	`forward` plugin misconfigured	Check Corefile `forward . /etc/resolv.conf`
Intermittent DNS failures	CoreDNS pods crashing or overloaded	Check pod logs, scale up CoreDNS replicas
`NXDOMAIN` for a Service	Service doesn't exist or wrong namespace	Verify: `kubectl get svc -A \\| grep <name>`
DNS works from some pods but not others	Pod `dnsPolicy` set to `Default` or `None`	Check pod spec `dnsPolicy`
Slow DNS resolution	`ndots:5` causing extra lookups for external names	Use FQDN with trailing dot: `google.com.`
CoreDNS `CrashLoopBackOff`	Forwarding loop (node DNS points back to CoreDNS)	Check Corefile `loop` plugin, fix node `/etc/resolv.conf`

4.3 CoreDNS Logs¶

# View CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns

# Enable verbose logging (add "log" plugin to Corefile)
kubectl edit configmap coredns -n kube-system
# Add "log" on a new line inside the .:53 block

# Watch logs after enabling
kubectl logs -n kube-system -l k8s-app=kube-dns -f

4.4 Verifying the Full DNS Chain¶

# 1. Is CoreDNS running?
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl get svc kube-dns -n kube-system

# 2. Is the kube-dns Service reachable?
kubectl run debug --image=busybox -it --rm --restart=Never -- \
  nslookup kubernetes.default 10.96.0.10

# 3. Does the pod's resolv.conf point to kube-dns?
kubectl exec <pod> -- cat /etc/resolv.conf

# 4. Can the pod resolve Service names?
kubectl exec <pod> -- nslookup <service-name>

# 5. Can the pod resolve external names?
kubectl exec <pod> -- nslookup google.com

# 6. Check CoreDNS ConfigMap
kubectl get cm coredns -n kube-system -o yaml

5. Practice Exercises¶

Exercise 1 — Explore Pod Networking¶

# 1. Create two pods on different nodes
kubectl run pod-a --image=busybox --command -- sleep 3600
kubectl run pod-b --image=busybox --command -- sleep 3600

# 2. Get their IPs and nodes
kubectl get pods -o wide

# 3. Test pod-to-pod connectivity
kubectl exec pod-a -- ping -c 3 <pod-b-ip>

# 4. Check the pod CIDR
kubectl cluster-info dump | grep -m1 cluster-cidr

# 5. Clean up
kubectl delete pod pod-a pod-b

Exercise 2 — Explore DNS¶

# 1. Create a deployment and service
kubectl create deployment dns-test --image=nginx --replicas=2
kubectl expose deployment dns-test --port=80

# 2. Start a debug pod
kubectl run debug --image=busybox -it --rm --restart=Never -- sh

# Inside the debug pod:
# 3. Resolve the service (short name)
nslookup dns-test

# 4. Resolve with full FQDN
nslookup dns-test.default.svc.cluster.local

# 5. Check resolv.conf
cat /etc/resolv.conf

# 6. Resolve an external name
nslookup google.com

# 7. Exit and clean up
exit
kubectl delete deployment dns-test
kubectl delete svc dns-test

Exercise 3 — Inspect CoreDNS¶

# 1. Check CoreDNS deployment
kubectl get deployment coredns -n kube-system

# 2. View the Corefile
kubectl get configmap coredns -n kube-system -o yaml

# 3. Check CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=20

# 4. What ClusterIP does kube-dns use?
kubectl get svc kube-dns -n kube-system

# 5. Verify a pod's resolv.conf points to that IP
kubectl run check --image=busybox -it --rm --restart=Never -- cat /etc/resolv.conf

Exercise 4 — Troubleshoot DNS Failure¶

# 1. Scale CoreDNS to 0 (simulate failure)
kubectl scale deployment coredns -n kube-system --replicas=0

# 2. Try DNS resolution from a new pod
kubectl run debug --image=busybox -it --rm --restart=Never -- nslookup kubernetes
# Should fail — no CoreDNS running

# 3. Fix it
kubectl scale deployment coredns -n kube-system --replicas=2

# 4. Retry
kubectl run debug --image=busybox -it --rm --restart=Never -- nslookup kubernetes
# Should succeed now

Exercise 5 — Cross-Namespace DNS¶

# 1. Create a service in a different namespace
kubectl create namespace backend
kubectl create deployment api --image=nginx -n backend
kubectl expose deployment api --port=80 -n backend

# 2. Resolve from default namespace
kubectl run debug --image=busybox -it --rm --restart=Never -- sh
  nslookup api.backend
  nslookup api.backend.svc.cluster.local
  wget -qO- http://api.backend:80
  exit

# 3. Clean up
kubectl delete namespace backend

Exercise 6 — Identify the CNI Plugin¶

# 1. Check CNI config
ls /etc/cni/net.d/
cat /etc/cni/net.d/*.conflist | head -20

# 2. Check CNI binaries
ls /opt/cni/bin/

# 3. Check CNI-related pods
kubectl get pods -n kube-system | grep -E "calico|flannel|cilium|weave"

# 4. Check pod CIDR allocation per node
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDR}{"\n"}{end}'

6. Key Takeaways for the CKA Exam¶

Point	Detail
Every pod gets a unique IP	No NAT between pods — this is the fundamental rule
Three CIDR ranges	Node, pod, and service — they must not overlap
CNI implements the network	Kubernetes defines rules, CNI plugin does the work
Flannel = no NetworkPolicy	If you need NetworkPolicies, use Calico or Cilium
CoreDNS = kube-dns Service	ClusterIP is in every pod's `/etc/resolv.conf`
Service DNS format	`<svc>.<ns>.svc.cluster.local`
Short names work	`my-svc` resolves in the same namespace via search domains
Cross-namespace	`my-svc.other-ns` or `my-svc.other-ns.svc.cluster.local`
CoreDNS ConfigMap	`kubectl get cm coredns -n kube-system -o yaml`
`ndots:5`	Names with < 5 dots get search domains appended first
Debug DNS with nslookup	`kubectl run debug --image=busybox -it --rm --restart=Never -- nslookup <name>`
kube-proxy modes	iptables (default) or ipvs — programs rules, doesn't proxy traffic
CNI config	`/etc/cni/net.d/` (config), `/opt/cni/bin/` (binaries)

Previous: 10-services.md — Services

Next: 12-ingress.md — Ingress