Actions

Difference between revisions of "Kubernetes/Install Master and nodes"

From Ever changing code

< Kubernetes
(Testing cluster end-to-end)
(Upgrade)
(4 intermediate revisions by the same user not shown)
Line 192: Line 192:
 
#access a pod directly using port-forwarding by creating local listener on port 8080 that forwards traffic to the pod's port 80.
 
#access a pod directly using port-forwarding by creating local listener on port 8080 that forwards traffic to the pod's port 80.
 
kubectl port-forward $pod_name 8080:80&
 
kubectl port-forward $pod_name 8080:80&
curl --head  http://127.0.0.1:8080 #Check a response from the 'nginx' pod directly
+
 
 +
#Check a response from the 'nginx' pod directly. Run from a master or where kubectl is installed
 +
curl --head  http://127.0.0.1:8080  
  
 
kubectl logs $pod_name            #view pod's logs
 
kubectl logs $pod_name            #view pod's logs
Line 206: Line 208:
  
 
#Check a response from the service
 
#Check a response from the service
curl -I localhost:$node_port #run from a worker node ()
+
curl -I localhost:$node_port #run on a worker node
 +
HTTP/1.1 200 OK
 +
Server: nginx/1.17.1
 +
Date: Sun, 07 Jul 2019 07:36:00 GMT
 +
Content-Type: text/html
 +
Content-Length: 612
 +
Last-Modified: Tue, 25 Jun 2019 12:19:45 GMT
 +
Connection: keep-alive
 +
ETag: "5d121161-264"
 +
Accept-Ranges: bytes
  
 
#Check information about nodes
 
#Check information about nodes
Line 212: Line 223:
 
kubectl describe nodes
 
kubectl describe nodes
 
</source>
 
</source>
 +
 +
= Upgrade =
 +
<code>kubeadm</code> allows us to upgrade a cluster components in controlled smart way it means the upgrade process will verify what components can be upgraded. Output examples are based on Amazon EKS, but EKS does not allow access for Control Place access, thus cannot see API Server or kube-controller.
 +
<source lang=bash>
 +
#kubectl version --short #check version for a API client (kubelet) and API server
 +
Client Version: v1.15.0
 +
Server Version: v1.12.6-eks-d69f1b
 +
 +
kubectl get pods -n kube-system kube-controller-manager-<podUID> -o yaml #version of the scheduler and controller manager
 +
kubectl get pods -n kube-system                                          #kube-controller pod
 +
 +
#Set the VERSION and ARCH you want to upgrade to
 +
export VERSION=v1.14.1; export ARCH=amd64
 +
 +
#Download Kubernetes binaries
 +
curl -sSL https://dl.k8s.io/release/${VERSION}/bin/linux/${ARCH}/kubeadm > kubeadm
 +
 +
sudo install -o root -g root -m 0755 ./kubeadm /usr/bin/kubeadm #install kubeadm
 +
sudo kubeadm version
 +
sudo kubeadm upgrade plan    #plan upgrade
 +
kubeadm upgrade apply v1.14.1 #apply upgrade
 +
 +
#Check differences between the old and new manifests:
 +
diff kube-controller-manager.yaml /etc/kubernetes/manifests/kube-controller-manager.yaml
 +
 +
#Download the latest version of kubelet:
 +
curl -sSL https://dl.k8s.io/release/${VERSION}/bin/linux/${ARCH}/kubelet > kubelet
 +
sudo install -o root -g root -m 0755 ./kubelet /usr/bin/kubelet #install kublet
 +
sudo systemctl restart kubelet.service #restart is required
 +
kubectl get nodes -w #check nodes versions
 +
</source>
 +
 +
Controlling nodes during upgrade
 +
* if a node downtime is <5 minutes a node controller will restart a pod on a node but if it's greater then it will delete the pod
 +
* so upgrading a node, you should check pod running on it, then evict them
 +
<source lang=bash>
 +
kubectl drain <node> --ignore-deamonsets
 +
kubectl get nodes #the drained node will be in 'Ready,SchedulingDisabled' status
 +
#the node now can be upgraded
 +
kubectl uncordon <node> #bring back to participate in a cluster
 +
kubectl delete <node> #delete node from a cluster, remember to drain it first
 +
</source>
 +
 +
= Add new node to a cluster =
 +
On master node create connection token
 +
<source lang=bash>
 +
sudo kubeadm token generate #below is a token-name not a secret!
 +
123abc.abcabcabcabc
 +
sudo kubeadm token list
 +
sudo kubeadm token create 123abc.abcabcabcabc --ttl 2h --print-join-command
 +
 +
#Run join command on the new worker-node after you install all required packages: etc. Docker
 +
sudo kuebadm join <IP>:6443 --token 123abc.abcabcabcabc --discovery-token-ca-cert-hash sha256:****
 +
</source>
 +
== References ==
 +
*[https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-upgrade/ kubeadm upgrade] K8s docs
 +
*[https://kubernetes.io/docs/tasks/administer-cluster/cluster-management/#maintenance-on-a-node Maintenance on a Node] K8s docs
  
 
= References =
 
= References =

Revision as of 09:51, 10 July 2019

This example is based on Ubuntu 16 LTS

Install binaries

#Docker gpg key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
#Docker repository
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" 
#Kubernetes gpg key
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
#Kubernetes repository
cat << EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF

#Install software
sudo apt-get update
sudo apt-get install -y docker-ce=18.06.1~ce~3-0~ubuntu kubelet=1.13.5-00 kubeadm=1.13.5-00 kubectl=1.13.5-00
# Set packages at the current versions so they won't autoupdate
sudo apt-mark hold docker-ce kubelet kubeadm kubectl
#Add the iptables rule to sysctl.conf; then enable imidiately
echo "net.bridge.bridge-nf-call-iptables=1" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

Initialize a cluster

Run only on the master to initialize the cluster

sudo kubeadm init --pod-network-cidr=10.100.0.0/16

I0705 06:23:54.675905   24293 version.go:237] remote version is much newer: v1.15.0; falling back to: stable-1.13
[init] Using Kubernetes version: v1.13.7
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [kubemaster.acme.com localhost] and IPs [172.31.115.255 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [kubemaster.acme.com localhost] and IPs [172.31.115.255 127.0.0.1 ::1]
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubemaster.acme.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.31.115.255]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 20.002150 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "kubemaster.acme.com" as an annotation
[mark-control-plane] Marking the node kubemaster.acme.com as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node kubemaster.acme.com as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: xkcoul.0i2m*******ockj
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 172.31.115.255:6443 --token xkcoul.0i2m*******ockj --discovery-token-ca-cert-hash sha256:808*******6a


Set up local kubeconfig:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config


Create Flannel CNI network overlay to allow nodes to communicate with each other

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created


Join worker nodes

Run on worker nodes - join worker nodes to the master

sudo kubeadm join 172.31.115.255:6443 --token xkcoul.0i2m*******ockj --discovery-token-ca-cert-hash sha256:808*******6a
[preflight] Running pre-flight checks
[discovery] Trying to connect to API Server "172.31.115.255:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://172.31.115.255:6443"
[discovery] Requesting info from "https://172.31.115.255:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "172.31.115.255:6443"
[discovery] Successfully established connection with API Server "172.31.115.255:6443"
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "kubeworker1.acme.com" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.


Verify nodes joined the cluster

$ kubectl get nodes
NAME                  STATUS   ROLES    AGE     VERSION
kubemaster.acme.com   Ready    master   16m     v1.13.5
kubeworker1.acme.com  Ready    <none>   2m45s   v1.13.5
kubeworker2.acme.com  Ready    <none>   2m39s   v1.13.5

Highly Available Kubernetes Cluster

In the output below you can see out-of-box K8s installation. Note that not all components are duplicated. But even we douplicate components not all can run in active mode. Eg. because Controller Manger and Scheduler are constantly watching for a cluster events only one can be in active mode, the rest would be in standby mode.

#View cluster components
kubectl get pods -o custom-columns=POD:metadata.name,NODE:spec.nodeName --sort-by spec.nodeName -n kube-system
POD                                            NODE
coredns-86c58d9df4-cdl5a                       kube-master.acme.com
coredns-86c58d9df4-csxca                       kube-master.acme.com
etcd-kube-master.acme.com                      kube-master.acme.com
kube-apiserver-kube-master.acme.com            kube-master.acme.com
kube-controller-manager-kube-master.acme.com   kube-master.acme.com
kube-scheduler-kube-master.acme.com            kube-master.acme.com
kube-flannel-ds-amd64-cwd74                    kube-master.acme.com
kube-proxy-z264w                               kube-master.acme.com
kube-proxy-fxl6f                               kube-worker-1.acme.com
kube-flannel-ds-amd64-c7hva                    kube-worker-1.acme.com
kube-flannel-ds-amd64-c5p9a                    kube-worker-2.acme.com
kube-proxy-jtbwm                               kube-worker-2.acme.com

#View details of components
kubectl get endpoints kube-scheduler -n kube-system -o yaml

References

Testing cluster end-to-end

kubectl run nginx --image=nginx #deployment test, run eg. nginx
kubectl get deployments

#access a pod directly using port-forwarding by creating local listener on port 8080 that forwards traffic to the pod's port 80.
kubectl port-forward $pod_name 8080:80&

#Check a response from the 'nginx' pod directly. Run from a master or where kubectl is installed
curl --head   http://127.0.0.1:8080 

kubectl logs $pod_name             #view pod's logs
kubectl exec -it nginx -- nginx -v #run a commands directly on a container

#Create a service by exposing port 80 of the nginx deployment
kubectl expose deployment nginx --port 80 --type NodePort

#List the services in your cluster, you can see the service is exposed on port <tt>31839</tt>
kubectl get service nginx -o wide
NAME    TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE     SELECTOR
nginx   NodePort   10.110.225.169   <none>        80:31839/TCP   2m26s   run=nginx

#Check a response from the service
curl -I localhost:$node_port #run on a worker node
HTTP/1.1 200 OK
Server: nginx/1.17.1
Date: Sun, 07 Jul 2019 07:36:00 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 25 Jun 2019 12:19:45 GMT
Connection: keep-alive
ETag: "5d121161-264"
Accept-Ranges: bytes

#Check information about nodes
kubectl get nodes
kubectl describe nodes

Upgrade

kubeadm allows us to upgrade a cluster components in controlled smart way it means the upgrade process will verify what components can be upgraded. Output examples are based on Amazon EKS, but EKS does not allow access for Control Place access, thus cannot see API Server or kube-controller.

#kubectl version --short #check version for a API client (kubelet) and API server 
Client Version: v1.15.0
Server Version: v1.12.6-eks-d69f1b

kubectl get pods -n kube-system kube-controller-manager-<podUID> -o yaml #version of the scheduler and controller manager
kubectl get pods -n kube-system                                          #kube-controller pod

#Set the VERSION and ARCH you want to upgrade to
export VERSION=v1.14.1; export ARCH=amd64

#Download Kubernetes binaries
curl -sSL https://dl.k8s.io/release/${VERSION}/bin/linux/${ARCH}/kubeadm > kubeadm

sudo install -o root -g root -m 0755 ./kubeadm /usr/bin/kubeadm #install kubeadm
sudo kubeadm version
sudo kubeadm upgrade plan     #plan upgrade
kubeadm upgrade apply v1.14.1 #apply upgrade

#Check differences between the old and new manifests:
diff kube-controller-manager.yaml /etc/kubernetes/manifests/kube-controller-manager.yaml

#Download the latest version of kubelet:
curl -sSL https://dl.k8s.io/release/${VERSION}/bin/linux/${ARCH}/kubelet > kubelet
sudo install -o root -g root -m 0755 ./kubelet /usr/bin/kubelet #install kublet
sudo systemctl restart kubelet.service #restart is required
kubectl get nodes -w #check nodes versions

Controlling nodes during upgrade

  • if a node downtime is <5 minutes a node controller will restart a pod on a node but if it's greater then it will delete the pod
  • so upgrading a node, you should check pod running on it, then evict them
kubectl drain <node> --ignore-deamonsets
kubectl get nodes #the drained node will be in 'Ready,SchedulingDisabled' status
#the node now can be upgraded
kubectl uncordon <node> #bring back to participate in a cluster
kubectl delete <node> #delete node from a cluster, remember to drain it first

Add new node to a cluster

On master node create connection token

sudo kubeadm token generate #below is a token-name not a secret!
123abc.abcabcabcabc
sudo kubeadm token list
sudo kubeadm token create 123abc.abcabcabcabc --ttl 2h --print-join-command

#Run join command on the new worker-node after you install all required packages: etc. Docker
sudo kuebadm join <IP>:6443 --token 123abc.abcabcabcabc --discovery-token-ca-cert-hash sha256:****

References

References