Actions

Kubernetes/Tools

From Ever changing code

< Kubernetes

kubectl

Install
#Latest
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl

# Specific version
$(VER=v1.13.7; ARCH=amd64; curl -LO https://storage.googleapis.com/kubernetes-release/release/$VER/bin/linux/$ARCH/kubectl)
chmod +x ./kubectl; sudo mv ./kubectl /usr/local/bin/kubectl

# Verify, kubectl should not be more than -/+ 1 minor version difference then api-server
kubectl version --short 
Client Version: v1.13.7
Server Version: v1.13.7-gke.8

# Autocompletion
source <(kubectl completion bash); alias k=kubectl; complete -F __start_kubectl k

# Set default namespace
kubectl config set-context $(kubectl config current-context) --namespace=dev

vi ~/.kube/config
...
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
    namespace: web       # default namespace
  name: dev-frontend
...


api-resources aka kubernetes objects

List all available resources in a cluster.

$ kubectl api-resources -owide
NAME                              SHORTNAMES   APIGROUP                       NAMESPACED   KIND                             VERBS
bindings                                                                      true         Binding                          [create]
componentstatuses                 cs                                          false        ComponentStatus                  [get list]
configmaps                        cm                                          true         ConfigMap                        [create delete deletecollection get list patch update watch]
endpoints                         ep                                          true         Endpoints                        [create delete deletecollection get list patch update watch]
events                            ev                                          true         Event                            [create delete deletecollection get list patch update watch]
limitranges                       limits                                      true         LimitRange                       [create delete deletecollection get list patch update watch]
namespaces                        ns                                          false        Namespace                        [create delete get list patch update watch]
nodes                             no                                          false        Node                             [create delete deletecollection get list patch update watch]
persistentvolumeclaims            pvc                                         true         PersistentVolumeClaim            [create delete deletecollection get list patch update watch]
persistentvolumes                 pv                                          false        PersistentVolume                 [create delete deletecollection get list patch update watch]
pods                              po                                          true         Pod                              [create delete deletecollection get list patch update watch]
podtemplates                                                                  true         PodTemplate                      [create delete deletecollection get list patch update watch]
replicationcontrollers            rc                                          true         ReplicationController            [create delete deletecollection get list patch update watch]
resourcequotas                    quota                                       true         ResourceQuota                    [create delete deletecollection get list patch update watch]
secrets                                                                       true         Secret                           [create delete deletecollection get list patch update watch]
serviceaccounts                   sa                                          true         ServiceAccount                   [create delete deletecollection get list patch update watch]
services                          svc                                         true         Service                          [create delete get list patch update watch]
mutatingwebhookconfigurations                  admissionregistration.k8s.io   false        MutatingWebhookConfiguration     [create delete deletecollection get list patch update watch]
validatingwebhookconfigurations                admissionregistration.k8s.io   false        ValidatingWebhookConfiguration   [create delete deletecollection get list patch update watch]
customresourcedefinitions         crd,crds     apiextensions.k8s.io           false        CustomResourceDefinition         [create delete deletecollection get list patch update watch]
apiservices                                    apiregistration.k8s.io         false        APIService                       [create delete deletecollection get list patch update watch]
controllerrevisions                            apps                           true         ControllerRevision               [create delete deletecollection get list patch update watch]
daemonsets                        ds           apps                           true         DaemonSet                        [create delete deletecollection get list patch update watch]
deployments                       deploy       apps                           true         Deployment                       [create delete deletecollection get list patch update watch]
replicasets                       rs           apps                           true         ReplicaSet                       [create delete deletecollection get list patch update watch]
statefulsets                      sts          apps                           true         StatefulSet                      [create delete deletecollection get list patch update watch]
tokenreviews                                   authentication.k8s.io          false        TokenReview                      [create]
localsubjectaccessreviews                      authorization.k8s.io           true         LocalSubjectAccessReview         [create]
selfsubjectaccessreviews                       authorization.k8s.io           false        SelfSubjectAccessReview          [create]
selfsubjectrulesreviews                        authorization.k8s.io           false        SelfSubjectRulesReview           [create]
subjectaccessreviews                           authorization.k8s.io           false        SubjectAccessReview              [create]
horizontalpodautoscalers          hpa          autoscaling                    true         HorizontalPodAutoscaler          [create delete deletecollection get list patch update watch]
cronjobs                          cj           batch                          true         CronJob                          [create delete deletecollection get list patch update watch]
jobs                                           batch                          true         Job                              [create delete deletecollection get list patch update watch]
certificatesigningrequests        csr          certificates.k8s.io            false        CertificateSigningRequest        [create delete deletecollection get list patch update watch]
leases                                         coordination.k8s.io            true         Lease                            [create delete deletecollection get list patch update watch]
events                            ev           events.k8s.io                  true         Event                            [create delete deletecollection get list patch update watch]
daemonsets                        ds           extensions                     true         DaemonSet                        [create delete deletecollection get list patch update watch]
deployments                       deploy       extensions                     true         Deployment                       [create delete deletecollection get list patch update watch]
ingresses                         ing          extensions                     true         Ingress                          [create delete deletecollection get list patch update watch]
networkpolicies                   netpol       extensions                     true         NetworkPolicy                    [create delete deletecollection get list patch update watch]
podsecuritypolicies               psp          extensions                     false        PodSecurityPolicy                [create delete deletecollection get list patch update watch]
replicasets                       rs           extensions                     true         ReplicaSet                       [create delete deletecollection get list patch update watch]
networkpolicies                   netpol       networking.k8s.io              true         NetworkPolicy                    [create delete deletecollection get list patch update watch]
poddisruptionbudgets              pdb          policy                         true         PodDisruptionBudget              [create delete deletecollection get list patch update watch]
podsecuritypolicies               psp          policy                         false        PodSecurityPolicy                [create delete deletecollection get list patch update watch]
clusterrolebindings                            rbac.authorization.k8s.io      false        ClusterRoleBinding               [create delete deletecollection get list patch update watch]
clusterroles                                   rbac.authorization.k8s.io      false        ClusterRole                      [create delete deletecollection get list patch update watch]
rolebindings                                   rbac.authorization.k8s.io      true         RoleBinding                      [create delete deletecollection get list patch update watch]
roles                                          rbac.authorization.k8s.io      true         Role                             [create delete deletecollection get list patch update watch]
priorityclasses                   pc           scheduling.k8s.io              false        PriorityClass                    [create delete deletecollection get list patch update watch]
storageclasses                    sc           storage.k8s.io                 false        StorageClass                     [create delete deletecollection get list patch update watch]
volumeattachments                              storage.k8s.io                 false        VolumeAttachment                 [create delete deletecollection get list patch update watch]


Each object has to properties:

  • spec - describes desired state
  • status - current status

kubectl diff

Shows the differences between the current live object and the new dry-run object.

kubectl diff -f webfront-deploy.yaml
diff -u -N /tmp/LIVE-761963756/apps.v1.Deployment.default.webfront-deploy /tmp/MERGED-431884635/apps.v1.Deployment.default.webfront-deploy
--- /tmp/LIVE-761963756/apps.v1.Deployment.default.webfront-deploy      2019-10-13 17:46:59.784000000 +0000
+++ /tmp/MERGED-431884635/apps.v1.Deployment.default.webfront-deploy    2019-10-13 17:46:59.788000000 +0000
@@ -4,7 +4,7 @@
   annotations:
     deployment.kubernetes.io/revision: "1"
   creationTimestamp: "2019-10-13T16:38:43Z"
-  generation: 2
+  generation: 3
   labels:
     app: webfront-deploy
   name: webfront-deploy
@@ -14,7 +14,7 @@
   uid: ebaf757e-edd7-11e9-8060-0a2fb3cdd79a
 spec:
   progressDeadlineSeconds: 600
-  replicas: 2
+  replicas: 1
   revisionHistoryLimit: 10
   selector:
     matchLabels:
@@ -29,6 +29,7 @@
       creationTimestamp: null
       labels:
         app: webfront-deploy
+        role: webfront
     spec:
       containers:
       - image: nginx:1.7.8
exit status 1


Kubectl-plugins


Getting help like manpages
$ kubectl --help
$ kubectl get --help
$ kubectl explain --help
$ kubectl explain pod.spec.containers #kubectl knows cluster version, so gives you correct schema details

Stern

Log tailing and landscape viewing tool. It connects to kubeapi and streams logs from all pods. Thus using this external tool with clusters that have 100ts of containers can be put significant load on kubeapi.


It will re-use kubectl config file to connect to your clusters, so works oob.


Install

# Govendor - this module manager is required
export GOPATH=$HOME/go        # path where go modules can be found, used by 'go get -u <url>'
export PATH=$PATH:$GOPATH/bin # path to the additional 'go' binaries
go get -u github.com/kardianos/govendor  # there will be no output

# Stern (official)
mkdir -p $GOPATH/src/github.com/wercker
cd $GOPATH/src/github.com/wercker
git clone https://github.com/wercker/stern.git && cd stern
govendor sync # there will be no output, may take 2 min
go install    # no output

# Stern install by getting binary direct, no need for govendor
sudo curl -L https://github.com/wercker/stern/releases/download/1.11.0/stern_linux_amd64 -o /usr/local/bin/stern 
sudo chmod +x /usr/local/bin/stern


Help

$ stern
Tail multiple pods and containers from Kubernetes

Usage:
  stern pod-query [flags]

Flags:
      --all-namespaces             If present, tail across all namespaces. A specific namespace is ignored even if specified with --namespace.
      --color string               Color output. Can be 'always', 'never', or 'auto' (default "auto")
      --completion string          Outputs stern command-line completion code for the specified shell. Can be 'bash' or 'zsh'
  -c, --container string           Container name when multiple containers in pod (default ".*")
      --container-state string     If present, tail containers with status in running, waiting or terminated. Default to running. (default "running")
      --context string             Kubernetes context to use. Default to current context configured in kubeconfig.
  -e, --exclude strings            Regex of log lines to exclude
  -E, --exclude-container string   Exclude a Container name
  -h, --help                       help for stern
  -i, --include strings            Regex of log lines to include
      --kubeconfig string          Path to kubeconfig file to use
  -n, --namespace string           Kubernetes namespace to use. Default to namespace configured in Kubernetes context
  -o, --output string              Specify predefined template. Currently support: [default, raw, json] (default "default")
  -l, --selector string            Selector (label query) to filter on. If present, default to ".*" for the pod-query.
  -s, --since duration             Return logs newer than a relative duration like 5s, 2m, or 3h. Defaults to 48h.
      --tail int                   The number of lines from the end of the logs to show. Defaults to -1, showing all logs. (default -1)
      --template string            Template to use for log lines, leave empty to use --output flag
  -t, --timestamps                 Print timestamps
  -v, --version                    Print the version and exit


Usage
stern <pod>
stern --tail 1 busybox -n <namespace> #this is RegEx that matches busybox1|2|etc

kui terminal

kui is a terminal with visualizations, provided by IBM

curl -sL https://raw.githubusercontent.com/IBM/kui/master/tools/install.sh | sh

k9s

ClipCapIt-190826-152830.PNG

Install

curl -O k9s_0.8.2_Linux_x86_64.tar.gz https://github.com/derailed/k9s/releases/download/0.8.2/k9s_0.8.2_Linux_x86_64.tar.gz
tar xf k9s_0.8.2_Linux_x86_64.tar.gz --remove-files k9s
sudo install k9s /usr/local/bin

kubecolor

Kubecolor is a bash function that colorizes the output of kubectl get events -w.

ClipCapIt-190831-113158.PNG
# This script is not working
git clone https://github.com/droctothorpe/kubecolor.git ~/.kubecolor
echo "source ~/.kubecolor/kubecolor.bash" >> ~/.bash_profile # (or ~/.bashrc)
source ~/.bash_profile # (or ~/.bashrc)

# You can source this function instead
kube-events() {
    kubectl get events --all-namespaces --watch \
    -o 'go-template={{.lastTimestamp}} ^ {{.involvedObject.kind}} ^ {{.message}} ^ ({{.involvedObject.name}}){{"\n"}}' \
    | awk -F^ \
    -v   black=$(tput setaf 0) \
    -v     red=$(tput setaf 1) \
    -v   green=$(tput setaf 2) \
    -v  yellow=$(tput setaf 3) \
    -v    blue=$(tput setaf 4) \
    -v magenta=$(tput setaf 5) \
    -v    cyan=$(tput setaf 6) \
    -v   white=$(tput setaf 7) \
    '{ $1=blue $1; $2=green $2; $3=white $3; }  1'
}

# Usage
kube-events
kubectl get events -A -w
kubectl get events --all-namespaces --watch -o 'go-template={{.lastTimestamp}} {{.involvedObject.kind}} {{.message}} ({{.involvedObject.name}}){{"\n"}}'

Kubernetes online yaml validator

argo-rollouts

Argo Rollouts introduces a new custom resource called a Rollout to provide additional deployment strategies such as Blue Green and Canary to Kubernetes.

Multi-node clusters

Build multi node cluster for development. On a single machine

  • kube-spawn tool for creating a multi-node Kubernetes (>= 1.8) cluster on a single Linux machine
  • kubernetes-dind-cluster Kubernetes multi-node cluster for developer of Kubernetes that launches in 36 seconds
  • kind is a tool for running local Kubernetes clusters using Docker container “nodes”
  • Vagrant full documentation in thsi article


Full cluster provisioning

  • kubespray Deploy a Production Ready Kubernetes Cluster
  • kops get a production grade Kubernetes cluster up and running

Docker - for a single missing commands

If you ever miss some commands you can use docker container package with it:

# curl - missing on minikube node that runs CoreOS
minikube -p metrics ip; minikube ssh
docker run appropriate/curl -- http://<NodeIP>:10255/stats/summary # check kubelet-metrics non secure endpoint

crictl

CLI and validation tools for Kubelet Container Runtime Interface (CRI). Used for debugging Kubernetes nodes with crictl. crictl requires a Linux operating system with a CRI runtime. Creating containers with this tool on K8s cluster, will eventually cause that Kubernetes will delete these containers.

References

Cheatsheets