Kubernetes/DNS

DNS

Since 1.13, CoreOS has replaced kube-dns with core-dns written in Go. It supports DNS over TLS in short dot. core-dns pods are running as a deployment.

#svc-name     ns   /BaseDomainName\ 
kubernetes.default.svc.cluster.local
10-10-20-1.default.pod.cluster.local
#pod ip       ns   \BaseDomainName/

Core-dns runs as a deployment

kubectl -n kube-system get pod -owide | grep core
coredns-86c58d9df4-7dl5d                                  1/1     Running   59         12d   10.100.0.18      master-1.acme.com   <none>           <none>
coredns-86c58d9df4-rsxct                                  1/1     Running   59         12d   10.100.0.19      master-1.acme.com   <none>           <none>

kubectl -n kube-system get deployments -owide
NAME      READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                     SELECTOR
coredns   2/2     2            2           12d   coredns      k8s.gcr.io/coredns:1.2.6   k8s-app=kube-dns

#Service that perform LoadBalancing. note it's named 'kube-dns' to support backward compatibnility for workloads relaying on kube-dns
ubectl -n kube-system get service -owide
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE   SELECTOR
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP   12d   k8s-app=kube-dns

Interact with DNS, using BusyBox container

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - image: busybox:1.28.4
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox
  restartPolicy: Always

#Deploy
kubectl apply -f busybox.yaml

#Verify dns settings
kubectl exec -t busybox -- cat /etc/resolv.conf 
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local mylabserver.com
options ndots:5

#Check DNS operations
kubectl exec -it busybox -- nslookup  <service-name>
kubectl exec -it busybox -- nslookup  <pod-ip-v-4>.default.pod.cluster.local #pod
kubectl exec -it busybox -- nslookup     <svcName>.default.svc.cluster.local #service
kubectl exec -it busybox -- nslookup  kube-dns.kube-system.svc.cluster.local #service

CoreDNS manages following records for Services:

• A records (not headless) - Services are assigned a DNS A record for a name of the form my-svc.my-namespace.svc.cluster-domain.example. This resolves to the cluster IP of the Service.
• A records (headless without cluster IP) - Services are also assigned a DNS A record for a name of the form my-svc.my-namespace.svc.cluster-domain.example. Unlike normal Services, this resolves to the set of IPs of the pods selected by the Service
• SRV records - SRV Records are created for named ports that are part of normal or Headless Services. read more...

Headless service

Headless services, its a service without a clusterIP, will respond with set of IPs that belong to a POD. These IPs are current IPs that service consider healthy pods.

apiVersion: v1
kind: Service
metadata:
  name: kube-headless
spec:
  clusterIP: None #set to none
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: kubserve2

Custom DNS Entries For Kubernetes

Custom dns settings can be set per pod basis. Default is cluster first, pod inherits DNS settings from a node it's running on

per pod dns settings nameservers, search, /etc/hosts

<syntaxhighlightjs lang="yaml"> apiVersion: v1 kind: Pod metadata:

 namespace: default
 name: pod-dns-custom-setup

spec:

 containers:
   - name: test
     image: nginx
 dnsPolicy: "None" # do not inherit settings from a node
 dnsConfig:
   nameservers:
     - 8.8.4.4
   searches:
     - ns1.svc.cluster.local
     - my.dns.search.suffix
   options:
     - name: ndots
       value: "3"
     - name: edns0

</syntaxhighlightjs>

alias, rewrite

<syntaxhighlightjs lang="yaml">

1. kubectl -n kube-system edit cm coredns

   .:53 {
       errors
       log
       health
       rewrite name foo.example.com foo.default.svc.cluster.local
       kubernetes cluster.local 10.0.0.0/24
       file /etc/coredns/example.db example.org
       proxy . /etc/resolv.conf
       cache 30
   }

</syntaxhighlightjs>

Trooubleshot core-dns

Coredns runs as pods

kubectl -n kube-system get pods | grep dns
coredns-86c58d9df4-7dl5d                                  1/1     Running   60         13d
coredns-86c58d9df4-rsxct                                  1/1     Running   60         13d

See logs

kubectl -n kube-system logs coredns-86c58d9df4-7dl5d -f
.:53
2019-07-18T06:33:08.165Z [INFO] CoreDNS-1.2.6
2019-07-18T06:33:08.165Z [INFO] linux/amd64, go1.11.2, 756749c
CoreDNS-1.2.6
linux/amd64, go1.11.2, 756749c
 [INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769
 [ERROR] plugin/errors: 2 8726527267836830687.8090630885983783330. HINFO: unreachable backend: read udp 10.100.0.20:33021->172.31.0.2:53: i/o timeout

Enable resolution logs, the config reload may take a minute or two

kubectl -n kube-system edit cm coredns

apiVersion: v1
data:
  Corefile: |
    .:53 {
        log     # <- add info logs
        errors
        health
..

Example of info-logs

# nslookup test 1
kubectl -n default run --image=busybox busybox-1 --rm -it -- sh
/ $ nslookup -type=A -debug wp.pl 8.8.8.8
Server:		8.8.8.8
Address:	8.8.8.8:53

Query #0 completed in 18ms:
Non-authoritative answer:
Name:	wp.pl
Address: 212.77.98.9

# nslookup test 2
kubectl -n default run --image=busybox busybox-1 --rm -it -- nslookup kubernetes.default
# | Usage: nslookup [-type=QUERY_TYPE] [-debug] HOST [DNS_SERVER]

# Watch logs using stern, sourceIP 10.1.1.1 it is busybox's pod IP 
# | on EKS, it's a secondary-IP assigned to the node <code>busybox</code> pod is running on
$ stern --all-namespaces core
kube-systemkube-system coredns-bb46f865d-vt82x coredns 2019-12-30T15:56:09.450Z [INFO] 10.1.1.1:51168 - 3 "AAAA IN kubernetes.default. udp 36 false 512" NXDOMAIN qr,rd,ra 111 0.000733313s
 coredns-bb46f865d-dsfkv coredns 2019-12-30T15:56:09.449Z [INFO] 10.1.1.1:50216 - 2 "PTR IN 10.0.20.172.in-addr.arpa. udp 42 false 512" NOERROR qr,aa,rd 118 0.000258372s
kube-system coredns-bb46f865d-dsfkv coredns 2019-12-30T15:56:09.451Z [INFO] 10.1.1.1:51070 - 4 "AAAA IN kubernetes.default.default.svc.cluster.local. udp 62 false 512" NXDOMAIN qr,aa,rd 155 0.000145002s
kube-system coredns-bb46f865d-dsfkv coredns 2019-12-30T15:56:09.452Z [INFO] 10.1.1.1:38472 - 5 "AAAA IN kubernetes.default.svc.cluster.local. udp 54 false 512" NOERROR qr,aa,rd 147 0.000097828s
kube-system coredns-bb46f865d-dsfkv coredns 2019-12-30T15:56:09.456Z [INFO] 10.1.1.1:48263 - 6 "A IN kubernetes.default. udp 36 false 512" NXDOMAIN qr,rd,ra 111 0.002978255s
kube-system coredns-bb46f865d-dsfkv kube-system coredns 2019-12-30T15:56:09.456Z [INFO] 10.1.1.1:50766 - 7 "A IN kubernetes.default.default.svc.cluster.local. udp 62 false 512" NXDOMAIN qr,aa,rd 155 0.000099745s
coredns-bb46f865d-vt82x coredns 2019-12-30T15:56:09.457Z [INFO] 10.1.1.1:40487 - 8 "A IN kubernetes.default.svc.cluster.local. udp 54 false 512" NOERROR qr,aa,rd 106 0.000107216s
kube-system coredns-bb46f865d-vt82x coredns 2019-12-30T15:56:09.458Z [INFO] 10.1.1.1:45808 - 9 "PTR IN 1.0.20.172.in-addr.arpa. udp 41 false 512" NOERROR qr,aa,rd 114 0.000087131s