Kubernetes/minikube

From Ever changing code
Jump to navigation Jump to search

Minikube features v1.6.2

  • LoadBalancer - using minikube tunnel
  • Multi-cluster - using minikube start -p <name>
  • NodePorts - using minikube service
  • Persistent Volumes
  • Ingress
  • RBAC
  • Dashboard - minikube dashboard
  • Container runtimes - start --container-runtime
  • Configure apiserver and kubelet options via command-line flags

Developer friendly features:


Release notes CHANGELOG.mb

Documentation minikube.sigs.k8s.io

Addons

List addons 

$ minikube addons list 
- addon-manager: enabled
- dashboard: enabled
- default-storageclass: enabled
- efk: disabled
- freshpod: disabled
- gvisor: disabled
- heapster: disabled
- helm-tiller: disabled
- ingress: disabled
- ingress-dns: disabled
- logviewer: disabled
- metrics-server: disabled
- nvidia-driver-installer: disabled
- nvidia-gpu-device-plugin: disabled
- registry: disabled
- registry-creds: disabled
- storage-provisioner: enabled
- storage-provisioner-gluster: disabled


Usage 

minikube -p efk start  --memory 8192
minikube -p efk addons enable efk     # pulling images and setting up can take ~10 min
minikube -p efk addons open efk       # opens Kibana

minikube addons enable heapster #get some insight into CPU

Install or upgrade on Linux

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 \
      && sudo install minikube-linux-amd64 /usr/local/bin/minikube

curl -LO https://github.com/kubernetes/minikube/releases/download/v1.6.2/minikube-linux-amd64 \
      && sudo install minikube-linux-amd64 /usr/local/bin/minikube

$> minikube version 
minikube version: v1.6.2
commit: 54f28ac5d3a815d1196cd5d57d707439ee4bb392

Start on Windows

Install <syntaxhighlightjs lang="powershell"> choco install minikube kubernetes-cli </syntaxhighlightjs>


HyperV <syntaxhighlightjs lang="powershell">

  1. HyperV set config (option 1)

minikube config set vm-driver hyperv minikube config set hyperv-virtual-switch "Default Switch" minikube start

  1. HyperV single command (option 2)

minikube start --vm-driver hyperv --hyperv-virtual-switch k8s-node-1 --profile k8s-node-1

  1. --hyperv-virtual-switch :- need to exist

</syntaxhighlightjs>


Virtualbox <syntaxhighlightjs lang="powershell"> minikube start --profile k8s-node-1

  • minikube v1.2.0 on windows (amd64)
  • Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
  • Configuring environment for Kubernetes v1.15.0 on Docker 18.09.6
  • Downloading kubelet v1.15.0
  • Downloading kubeadm v1.15.0
  • Pulling images ...
  • Launching Kubernetes ...
  • Verifying: apiserver proxy etcd scheduler controller dns
  • Done! kubectl is now configured to use "k8s-node-1"

PS C:\Users\Sylwia> kubectl.exe get nodes -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME minikube Ready master 23m v1.15.0 10.0.2.15 <none> Buildroot 2018.05.3 4.15.0 docker://18.9.6 </syntaxhighlightjs>

Start

Example of start sequence 

minikube start 
😄  minikube v1.2.0 on linux (amd64)
🔥  Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
🐳  Configuring environment for Kubernetes v1.15.0 on Docker 18.09.6
E0728 00:14:31.657125   26453 start.go:403] Error caching images:  Caching images for kubeadm: caching images: caching image /home/user1/.minikube/cache/images/k8s.gcr.io/k8s-dns-kube-dns-amd64_1.14.13: stream error: stream ID 25; INTERNAL_ERROR
❌  Unable to load cached images: loading cached images: loading image /home/pipaw/.minikube/cache/images/k8s.gcr.io/k8s-dns-sidecar-amd64_1.14.13: stat /home/user1/.minikube/cache/images/k8s.gcr.io/k8s-dns-sidecar-amd64_1.14.13: no such file or directory
💾  Downloading kubeadm v1.15.0
💾  Downloading kubelet v1.15.0
🚜  Pulling images ...
🚀  Launching Kubernetes ... 
⌛  Verifying: apiserver proxy etcd scheduler controller dns
🏄  Done! kubectl is now configured to use "minikube"

$ minikube status
host: Running
kubelet: Running
apiserver: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.101

$ kubectl get nodes -o wide
NAME       STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE              KERNEL-VERSION   CONTAINER-RUNTIME
minikube   Ready    master   8h    v1.15.0   10.0.2.15     <none>        Buildroot 2018.05.3   4.15.0           docker://18.9.6

$ kubectl get all -o wide --all-namespaces
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
kube-system   pod/coredns-5c98db65d4-d688k           1/1     Running   0          8h    172.17.0.2   minikube   <none>           <none>
kube-system   pod/coredns-5c98db65d4-xdq6d           1/1     Running   0          8h    172.17.0.3   minikube   <none>           <none>
kube-system   pod/etcd-minikube                      1/1     Running   0          8h    10.0.2.15    minikube   <none>           <none>
kube-system   pod/kube-addon-manager-minikube        1/1     Running   0          8h    10.0.2.15    minikube   <none>           <none>
kube-system   pod/kube-apiserver-minikube            1/1     Running   0          8h    10.0.2.15    minikube   <none>           <none>
kube-system   pod/kube-controller-manager-minikube   1/1     Running   0          8h    10.0.2.15    minikube   <none>           <none>
kube-system   pod/kube-proxy-j5f2h                   1/1     Running   0          8h    10.0.2.15    minikube   <none>           <none>
kube-system   pod/kube-scheduler-minikube            1/1     Running   0          8h    10.0.2.15    minikube   <none>           <none>
kube-system   pod/storage-provisioner                1/1     Running   0          8h    10.0.2.15    minikube   <none>           <none>

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE   SELECTOR
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  8h    <none>
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   8h    k8s-app=kube-dns

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE   CONTAINERS   IMAGES                          SELECTOR
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           beta.kubernetes.io/os=linux   8h    kube-proxy   k8s.gcr.io/kube-proxy:v1.15.0   k8s-app=kube-proxy

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                     SELECTOR
kube-system   deployment.apps/coredns   2/2     2            2           8h    coredns      k8s.gcr.io/coredns:1.3.1   k8s-app=kube-dns

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES                     SELECTOR
kube-system   replicaset.apps/coredns-5c98db65d4   2         2         2       8h    coredns      k8s.gcr.io/coredns:1.3.1   k8s-app=kube-dns,pod-template-hash=5c98db65d4

Usage

time minikube start                              #start a cluster by running
# Start another local clusters named: --profile can be abbreviated with -p, default is 'minikube'
time minikube start --kubernetes-version v1.14.9 --profile minikube-v1.14.9-aws # AWS versions v1.14.9-eks-c0eccc
time minikube start --kubernetes-version v1.15.0 --profile minikube-v1.15.0


Multi-profiles 

$ minikube profile list
|-----------------|------------|----------------|-----------|--------------------|
|     Profile     | VM Driver  |     NodeIP     | Node Port | Kubernetes Version |
|-----------------|------------|----------------|-----------|--------------------|
| k8s-v1.13.7-aws | virtualbox | 192.168.99.105 |      8443 | v1.15.0            |
| k8s-v1.15.0     | virtualbox | 192.168.99.104 |      8443 | v1.15.0            |
|-----------------|------------|----------------|-----------|--------------------|


Get port exposed to host via Virtualbox used by minikube to interact with a cluster 

$ sudo ss -ltpn
State  Recv-Q  Send-Q    Local Address:Port    Peer Address:Port                                               
LISTEN 0       128       127.0.0.53%lo:53           0.0.0.0:*      users:(("systemd-resolve",pid=1009,fd=13))  
LISTEN 0       10            127.0.0.1:45111 #<--   0.0.0.0:*      users:(("VBoxHeadless",pid=26918,fd=21))    
LISTEN 0       5             127.0.0.1:631          0.0.0.0:*      users:(("cupsd",pid=27629,fd=7))            
LISTEN 0       5                 [::1]:631             [::]:*      users:(("cupsd",pid=27629,fd=6))

$ vboxmanage showvminfo k8s-v1.15.0 | 'grep port ='                            vv
NIC 1 Rule(0):   name = ssh, protocol = tcp, host ip = 127.0.0.1, host port = 45111, guest ip = , guest port = 22
                                                                               ^^


Access to minikube VM using ssh
minikube ssh cat /etc/*rel* 
$ minikube ssh 
                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$
 
cat /etc/*rel*
NAME=Buildroot
VERSION=2018.05.3
ID=buildroot
VERSION_ID=2018.05.3
PRETTY_NAME="Buildroot 2018.05.3"
NAME=Buildroot
VERSION=2018.05.3
ID=buildroot
VERSION_ID=2018.05.3
PRETTY_NAME="Buildroot 2018.05.3"


#access Kubernetes Dashboard within Minikube
minikube dashboard
🤔  Verifying dashboard health ...
🚀  Launching proxy ...
🤔  Verifying proxy health ...
🎉  Opening http://127.0.0.1:44835/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ in your default browser...

#Use kubectl to interact with the local cluster
kubectl run --generator=run-pod/v1         hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #correct
kubectl run --generator=deployment/apps.v1 hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #deprecated
kubectl run       ^^default^^              hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #deprecated

#Expose a service as a NodePort
kubectl expose pod        hello-minikube --type=NodePort
kubectl expose deployment hello-minikube --type=NodePort #when pod was created via deployment deprecated option

#Determine the NodePort of the service
kubectl get service <hello-minikube> --output='jsonpath="{.spec.ports[0].nodePort}"'

#curl this endpoint
minikube ssh
curl -s -H 'Cache-Control: no-cache' http://<svc-cluster-ip>:8080 #note 'watch won't work as curl or service proxy 

#minikube makes it easy to open this exposed endpoint in your browser
minikube service hello-minikube

minikube stop   #stop your local cluster
minikube delete #delete your local cluster


Working with profiles 

$ ls -1 ~/.minikube/profiles/ #list all profiles
aws-v1.13.7-cluster
minikube             #default profile

#Minikube show current profile
$ minikube profile
aws-v1.13.7-cluster
$ cat ~/.minikube/config/config.json | jq -r .profile #alternative option

#Kubectl  show current context
$ kubectl config current-context
$ kubectl config get-contexts   #get-clusters
CURRENT   NAME                  CLUSTER               AUTHINFO              NAMESPACE
*         aws-v1.13.7-cluster   aws-v1.13.7-cluster   aws-v1.13.7-cluster   
          minikube              minikube              minikube


Switching the Minikube profile does automatically switch the k8s context, and vice versa (v1.2.0 - 2019-06-24) 

#Switch minikube profile
$ minikube profile minikube #switch to profile: minikube
✅  minikube profile was successfully set to minikube

#Switch kubectl context
$ kubectl config set-context minikube
Context "minikube" modified.

Clean up if needed 

minikube delete -p <profile>
kubectl config delete-context <context-name>
kubectl config delete-context <delete-cluster>


Example of ~/.minikube/profiles/<minikube-profile>/config.json
jq .MachineConfig config.json jq .KubernetesConfig config.json 
{
  "KubernetesVersion": "v1.13.7",
  "NodeIP": "192.168.99.102",
  "NodePort": 8443,
  "NodeName": "minikube",
  "APIServerName": "minikubeCA",
  "APIServerNames": null,
  "APIServerIPs": null,
  "DNSDomain": "cluster.local",
  "ContainerRuntime": "docker",
  "CRISocket": "",
  "NetworkPlugin": "",
  "FeatureGates": "",
  "ServiceCIDR": "10.96.0.0/12",
  "ImageRepository": "",
  "ExtraOptions": null,
  "ShouldLoadCachedImages": true,
  "EnableDefaultCNI": false
}
 
{
  "KeepContext": false,
  "MinikubeISO": "https://storage.googleapis.com/minikube/iso/minikube-v1.2.0.iso",
  "Memory": 2048,
  "CPUs": 2,
  "DiskSize": 20000,
  "VMDriver": "virtualbox",
  "ContainerRuntime": "docker",
  "HyperkitVpnKitSock": "",
  "HyperkitVSockPorts": [],
  "XhyveDiskDriver": "ahci-hd",
  "DockerEnv": null,
  "InsecureRegistry": null,
  "RegistryMirror": null,
  "HostOnlyCIDR": "192.168.99.1/24",
  "HypervVirtualSwitch": "",
  "KvmNetwork": "default",
  "DockerOpt": null,
  "DisableDriverMounts": false,
  "NFSShare": [],
  "NFSSharesRoot": "/nfsshares",
  "UUID": "",
  "GPU": false,
  "Hidden": false,
  "NoVTXCheck": false
}

Loadbalancer - minikube tunnel

A LoadBalancer service is the standard way to expose a service to the internet. With this method, each service gets it’s own IP address. In minikube services of type LoadBalancer can be exposed via the minikube tunnel command. It will run until Ctrl-C is hit.


minikube tunnel runs as a separate daemon, creating a network route on the host to the service CIDR of the cluster using the cluster’s IP address as a gateway. The tunnel command exposes the external IP directly to any program running on the host operating system. The low-level commands used to manage routes are /sbin/ip,/sbin/route that optionally can be added to suedors file to make a life a little easier. 

$ minikube profile list
|-----------------|------------|----------------|-----------|--------------------|
|     Profile     | VM Driver  |     NodeIP     | Node Port | Kubernetes Version |
|-----------------|------------|----------------|-----------|--------------------|
| efk             | virtualbox | 192.168.99.113 |      8443 | v1.16.0            |
|-----------------|------------|----------------|-----------|--------------------|

# Before
$ kubectl -n efk get svc # note <pending> EXTERNAL-IP
NAMESPACE NAME                  TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)       
efk       service/elasticsearch LoadBalancer   10.104.149.221  <pending>       9200:30826/TCP
efk       service/kibana        LoadBalancer   10.97.198.117   <pending>       5601:32540/TCP

$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         vodafone.connec 0.0.0.0         UG    600    0        0 wlp4s0
link-local      0.0.0.0         255.255.0.0     U     1000   0        0 wlp4s0
192.168.1.0     0.0.0.0         255.255.255.0   U     600    0        0 wlp4s0
192.168.99.0    0.0.0.0         255.255.255.0   U     0      0        0 vboxnet2

# After - EXTERNAL-IP have been assigned from a pool of 10.96.0.0/12
NAMESPACE NAME                  TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)       
efk       service/elasticsearch LoadBalancer   10.104.149.221  10.104.149.221  9200:30826/TCP
efk       service/kibana        LoadBalancer   10.97.198.117   10.97.198.117   5601:32540/TCP

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         vodafone.connec 0.0.0.0         UG    600    0        0 wlp4s0
10.96.0.0       192.168.99.113  255.240.0.0     UG    0      0        0 vboxnet2 # <- new route to minikube NodeIP created 
link-local      0.0.0.0         255.255.0.0     U     1000   0        0 wlp4s0
192.168.1.0     0.0.0.0         255.255.255.0   U     600    0        0 wlp4s0
192.168.99.0    0.0.0.0         255.255.255.0   U     0      0        0 vboxnet2

$ minikube --profile efk tunnel
Status:
        machine: efk
        pid: 8721
        route: 10.96.0.0/12 -> 192.168.99.113 # route status
        minikube: Running
        services: [elasticsearch, kibana]     # exposed services
    errors: 
                minikube: no errors
                router: no errors
                loadbalancer emulator: no errors
# Verify
$ nc -zv 10.97.198.117 5601 # kibana
Connection to 10.97.198.117 5601 port [tcp/*] succeeded!
$ nc -zv 10.104.149.221 9200 # elasticsearch
Connection to 10.104.149.221 9200 port [tcp/*] succeeded!
$ curl http://10.97.198.117:5601/status


Clean up based on info ~/.minikube/tunnels.json when shuts unexpected 

minikube tunnel --cleanup


Avoid to typing password for minikube tunnel, by allowing a group or user to execute ip, route commands without asking for password. Example for Ubuntu 18.04 can be seen below. 

$ sudo visudo
...
# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL
%admin ALL=(ALL) NOPASSWD: /sbin/ip,/sbin/route  # <- add this line in this order
...

multi node cluster with kvm

References

Retrieved from ‘http://wiki.ciscolinux.co.uk/index.php?title=Kubernetes/minikube&oldid=4999’