Kubernetes/minikube

Minikube features v1.6.2

• LoadBalancer - using minikube tunnel
• Multi-cluster - using minikube start -p <name>
• NodePorts - using minikube service
• Persistent Volumes
• Ingress
• RBAC
• Dashboard - minikube dashboard
• Container runtimes - start --container-runtime
• Configure apiserver and kubelet options via command-line flags

Developer friendly features:

• Filesystem mounts

Release notes CHANGELOG.mb

Documentation minikube.sigs.k8s.io

Addons

List addons

$ minikube addons list 
- addon-manager: enabled
- dashboard: enabled
- default-storageclass: enabled
- efk: disabled
- freshpod: disabled
- gvisor: disabled
- heapster: disabled
- helm-tiller: disabled
- ingress: disabled
- ingress-dns: disabled
- logviewer: disabled
- metrics-server: disabled
- nvidia-driver-installer: disabled
- nvidia-gpu-device-plugin: disabled
- registry: disabled
- registry-creds: disabled
- storage-provisioner: enabled
- storage-provisioner-gluster: disabled

Usage

minikube -p efk start  --memory 8192
minikube -p efk addons enable efk     # pulling images and setting up can take ~10 min
minikube -p efk addons open efk       # opens Kibana

minikube addons enable heapster #get some insight into CPU

Install or upgrade on Linux

# Latest from GitHub
VERSION=$(curl --silent "https://api.github.com/repos/kubernetes/minikube/releases/latest" | jq -r .tag_name)
curl -LO https://github.com/kubernetes/minikube/releases/download/$VERSION/minikube-linux-amd64 \
      && sudo install minikube-linux-amd64 /usr/local/bin/minikube

# Latest from GoogleAPI
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 \
      && sudo install minikube-linux-amd64 /usr/local/bin/minikube

# Specific version from GitHub
curl -LO https://github.com/kubernetes/minikube/releases/download/v1.7.3/minikube-linux-amd64 \
      && sudo install minikube-linux-amd64 /usr/local/bin/minikube

$> $ minikube version 
minikube version: v1.9.2
commit: 93af9c1e43cab9618e301bc9fa720c63d5efa393

Start on Windows

Install <syntaxhighlightjs lang="powershell"> choco install minikube kubernetes-cli </syntaxhighlightjs>

HyperV <syntaxhighlightjs lang="powershell">

1. HyperV set config (option 1)

minikube config set vm-driver hyperv minikube config set hyperv-virtual-switch "Default Switch" minikube start

1. HyperV single command (option 2)

minikube start --vm-driver hyperv --hyperv-virtual-switch k8s-node-1 --profile k8s-node-1

1. --hyperv-virtual-switch :- need to exist

</syntaxhighlightjs>

Virtualbox <syntaxhighlightjs lang="powershell"> minikube start --profile k8s-node-1

• minikube v1.2.0 on windows (amd64)
• Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
• Configuring environment for Kubernetes v1.15.0 on Docker 18.09.6
• Downloading kubelet v1.15.0
• Downloading kubeadm v1.15.0
• Pulling images ...
• Launching Kubernetes ...
• Verifying: apiserver proxy etcd scheduler controller dns
• Done! kubectl is now configured to use "k8s-node-1"

PS C:\Users\Sylwia> kubectl.exe get nodes -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME minikube Ready master 23m v1.15.0 10.0.2.15 <none> Buildroot 2018.05.3 4.15.0 docker://18.9.6 </syntaxhighlightjs>

Start

Example of start sequence

$> time minikube start --kubernetes-version v1.14.9 --profile minikube-v1.14.9 # minikube v1.9.2
😄  [minikube-v1.14.9] minikube v1.9.2 on Ubuntu 18.04
✨  Automatically selected the virtualbox driver
💿  Downloading VM boot image ...
    > minikube-v1.9.0.iso.sha256: 65 B / 65 B [--------------] 100.00% ? p/s 0s
    > minikube-v1.9.0.iso: 174.93 MiB / 174.93 MiB [-] 100.00% 4.05 MiB p/s 44s
👍  Starting control plane node m01 in cluster minikube-v1.14.9
💾  Downloading Kubernetes v1.14.9 preload ...
    > preloaded-images-k8s-v2-v1.14.9-docker-overlay2-amd64.tar.lz4: 496.25 MiB
🔥  Creating virtualbox VM (CPUs=2, Memory=3900MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.14.9 on Docker 19.03.8 ...
🌟  Enabling addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube-v1.14.9"

real	4m9.678s
user	0m21.710s
sys	0m30.296s


$ minikube profile list
|----------------------------------|------------|---------|----------------|------|---------|---------|
|             Profile              | VM Driver  | Runtime |       IP       | Port | Version | Status  |
|----------------------------------|------------|---------|----------------|------|---------|---------|
| minikube-v1.14.9                 | virtualbox | docker  | 192.168.99.123 | 8443 | v1.14.9 | Running |
|----------------------------------|------------|---------|----------------|------|---------|---------|
$ minikube status -p minikube-v1.14.9
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

$ kubectl get nodes -o wide
NAME             STATUS ROLES  AGE   VERSION INTERNAL-IP    EXTERNAL-IP OS-IMAGE             KERNEL-VERSION CONTAINER-RUNTIME
minikube-v1.14.9 Ready  master 6m57s v1.14.9 192.168.99.123 <none>      Buildroot 2019.02.10 4.19.107       docker://19.3.8

$ kubectl get all -o wide --all-namespaces
NAME                               STATUS   ROLES    AGE     VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE               KERNEL-VERSION   CONTAINER-RUNTIME
minikube-v1.14.9                   Ready    master   6m57s   v1.14.9   192.168.99.123   <none>        Buildroot 2019.02.10   4.19.107         docker://19.3.8
pipaw@pp-tp52s ~ $ kubectl get all -o wide --all-namespaces
NAMESPACE     NAME                                                           READY   STATUS    RESTARTS   AGE     IP               NODE                               NOMINATED NODE   READINESS GATES
kube-system   pod/coredns-6dcc67dcbc-48k6v                                   1/1     Running   0          8m16s   172.17.0.2       minikube-v1.14.9                   <none>           <none>
kube-system   pod/coredns-6dcc67dcbc-4ftc4                                   1/1     Running   0          8m16s   172.17.0.3       minikube-v1.14.9                   <none>           <none>
kube-system   pod/etcd-minikube-v1.14.9                                      1/1     Running   0          7m21s   192.168.99.123   minikube-v1.14.9                   <none>           <none>
kube-system   pod/kube-apiserver-minikube-v1.14.9                            1/1     Running   0          7m21s   192.168.99.123   minikube-v1.14.9                   <none>           <none>
kube-system   pod/kube-controller-manager-minikube-v1.14.9                   1/1     Running   0          7m26s   192.168.99.123   minikube-v1.14.9                   <none>           <none>
kube-system   pod/kube-proxy-kjjrx                                           1/1     Running   0          8m16s   192.168.99.123   minikube-v1.14.9                   <none>           <none>
kube-system   pod/kube-scheduler-minikube-v1.14.9                            1/1     Running   0          7m22s   192.168.99.123   minikube-v1.14.9                   <none>           <none>
kube-system   pod/storage-provisioner                                        1/1     Running   1          8m22s   192.168.99.123   minikube-v1.14.9                   <none>           <none>


NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE     SELECTOR
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  8m25s   <none>
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   8m24s   k8s-app=kube-dns

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE     CONTAINERS   IMAGES                          SELECTOR
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           <none>          8m23s   kube-proxy   k8s.gcr.io/kube-proxy:v1.14.9   k8s-app=kube-proxy

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                     SELECTOR
kube-system   deployment.apps/coredns   2/2     2            2           8m24s   coredns      k8s.gcr.io/coredns:1.3.1   k8s-app=kube-dns

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE     CONTAINERS   IMAGES                     SELECTOR
kube-system   replicaset.apps/coredns-6dcc67dcbc   2         2         2       8m16s   coredns      k8s.gcr.io/coredns:1.3.1   k8s-app=kube-dns,pod-template-hash=6dcc67dcbc

Usage

time minikube start                              #start a cluster by running
# Start another local clusters named: --profile can be abbreviated with -p, default is 'minikube'
time minikube start --kubernetes-version v1.14.9 --profile minikube-v1.14.9-aws # AWS versions v1.14.9-eks-c0eccc
time minikube start --kubernetes-version v1.15.0 --profile minikube-v1.15.0

Multi-profiles

$ minikube profile list
|-----------------|------------|----------------|-----------|--------------------|
|     Profile     | VM Driver  |     NodeIP     | Node Port | Kubernetes Version |
|-----------------|------------|----------------|-----------|--------------------|
| k8s-v1.13.7-aws | virtualbox | 192.168.99.105 |      8443 | v1.15.0            |
| k8s-v1.15.0     | virtualbox | 192.168.99.104 |      8443 | v1.15.0            |
|-----------------|------------|----------------|-----------|--------------------|

Get port exposed to host via Virtualbox used by minikube to interact with a cluster

$ sudo ss -ltpn
State  Recv-Q  Send-Q    Local Address:Port    Peer Address:Port                                               
LISTEN 0       128       127.0.0.53%lo:53           0.0.0.0:*      users:(("systemd-resolve",pid=1009,fd=13))  
LISTEN 0       10            127.0.0.1:45111 #<--   0.0.0.0:*      users:(("VBoxHeadless",pid=26918,fd=21))    
LISTEN 0       5             127.0.0.1:631          0.0.0.0:*      users:(("cupsd",pid=27629,fd=7))            
LISTEN 0       5                 [::1]:631             [::]:*      users:(("cupsd",pid=27629,fd=6))

$ vboxmanage showvminfo k8s-v1.15.0 | 'grep port ='                            vv
NIC 1 Rule(0):   name = ssh, protocol = tcp, host ip = 127.0.0.1, host port = 45111, guest ip = , guest port = 22
                                                                               ^^

Access to minikube VM using ssh

minikube ssh	cat /etc/*rel*
$ minikube ssh _ _ _ _ ( ) ( ) ___ ___ (_) ___ (_)| |/') _ _ | |_ __ /' _ ` _ `\| |/' _ `\| || , < ( ) ( )| '_`\ /'__`\ | ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )( ___/ (_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____) $	cat /etc/*rel* NAME=Buildroot VERSION=2018.05.3 ID=buildroot VERSION_ID=2018.05.3 PRETTY_NAME="Buildroot 2018.05.3" NAME=Buildroot VERSION=2018.05.3 ID=buildroot VERSION_ID=2018.05.3 PRETTY_NAME="Buildroot 2018.05.3"

#access Kubernetes Dashboard within Minikube
minikube dashboard
🤔  Verifying dashboard health ...
🚀  Launching proxy ...
🤔  Verifying proxy health ...
🎉  Opening http://127.0.0.1:44835/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ in your default browser...

#Use kubectl to interact with the local cluster
kubectl run --generator=run-pod/v1         hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #correct
kubectl run --generator=deployment/apps.v1 hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #deprecated
kubectl run       ^^default^^              hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #deprecated

#Expose a service as a NodePort
kubectl expose pod        hello-minikube --type=NodePort
kubectl expose deployment hello-minikube --type=NodePort #when pod was created via deployment deprecated option

#Determine the NodePort of the service
kubectl get service <hello-minikube> --output='jsonpath="{.spec.ports[0].nodePort}"'

#curl this endpoint
minikube ssh
curl -s -H 'Cache-Control: no-cache' http://<svc-cluster-ip>:8080 #note 'watch won't work as curl or service proxy 

#minikube makes it easy to open this exposed endpoint in your browser
minikube service hello-minikube

minikube stop   #stop your local cluster
minikube delete #delete your local cluster

Working with profiles

$ ls -1 ~/.minikube/profiles/ #list all profiles
aws-v1.13.7-cluster
minikube             #default profile

#Minikube show current profile
$ minikube profile
aws-v1.13.7-cluster
$ cat ~/.minikube/config/config.json | jq -r .profile #alternative option

#Kubectl  show current context
$ kubectl config current-context
$ kubectl config get-contexts   #get-clusters
CURRENT   NAME                  CLUSTER               AUTHINFO              NAMESPACE
*         aws-v1.13.7-cluster   aws-v1.13.7-cluster   aws-v1.13.7-cluster   
          minikube              minikube              minikube

Switching the Minikube profile does automatically switch the k8s context, and vice versa (v1.2.0 - 2019-06-24)

#Switch minikube profile
$ minikube profile minikube #switch to profile: minikube
✅  minikube profile was successfully set to minikube

#Switch kubectl context
$ kubectl config set-context minikube
Context "minikube" modified.

Clean up if needed

minikube delete -p <profile>
kubectl config delete-context <context-name>
kubectl config delete-context <delete-cluster>

Example of ~/.minikube/profiles/<minikube-profile>/config.json

jq .MachineConfig config.json

jq .KubernetesConfig config.json

{ "KubernetesVersion": "v1.13.7", "NodeIP": "192.168.99.102", "NodePort": 8443, "NodeName": "minikube", "APIServerName": "minikubeCA", "APIServerNames": null, "APIServerIPs": null, "DNSDomain": "cluster.local", "ContainerRuntime": "docker", "CRISocket": "", "NetworkPlugin": "", "FeatureGates": "", "ServiceCIDR": "10.96.0.0/12", "ImageRepository": "", "ExtraOptions": null, "ShouldLoadCachedImages": true, "EnableDefaultCNI": false }

{ "KeepContext": false, "MinikubeISO": "https://storage.googleapis.com/minikube/iso/minikube-v1.2.0.iso", "Memory": 2048, "CPUs": 2, "DiskSize": 20000, "VMDriver": "virtualbox", "ContainerRuntime": "docker", "HyperkitVpnKitSock": "", "HyperkitVSockPorts": [], "XhyveDiskDriver": "ahci-hd", "DockerEnv": null, "InsecureRegistry": null, "RegistryMirror": null, "HostOnlyCIDR": "192.168.99.1/24", "HypervVirtualSwitch": "", "KvmNetwork": "default", "DockerOpt": null, "DisableDriverMounts": false, "NFSShare": [], "NFSSharesRoot": "/nfsshares", "UUID": "", "GPU": false, "Hidden": false, "NoVTXCheck": false }

Loadbalancer - minikube tunnel

A LoadBalancer service is the standard way to expose a service to the internet. With this method, each service gets it’s own IP address. In minikube services of type LoadBalancer can be exposed via the minikube tunnel command. It will run until Ctrl-C is hit.

minikube tunnel runs as a separate daemon, creating a network route on the host to the service CIDR of the cluster using the cluster’s IP address as a gateway. The tunnel command exposes the external IP directly to any program running on the host operating system. The low-level commands used to manage routes are /sbin/ip,/sbin/route that optionally can be added to suedors file to make a life a little easier.

$ minikube profile list
|-----------------|------------|----------------|-----------|--------------------|
|     Profile     | VM Driver  |     NodeIP     | Node Port | Kubernetes Version |
|-----------------|------------|----------------|-----------|--------------------|
| efk             | virtualbox | 192.168.99.113 |      8443 | v1.16.0            |
|-----------------|------------|----------------|-----------|--------------------|

# Before
$ kubectl -n efk get svc # note <pending> EXTERNAL-IP
NAMESPACE NAME                  TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)       
efk       service/elasticsearch LoadBalancer   10.104.149.221  <pending>       9200:30826/TCP
efk       service/kibana        LoadBalancer   10.97.198.117   <pending>       5601:32540/TCP

$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         vodafone.connec 0.0.0.0         UG    600    0        0 wlp4s0
link-local      0.0.0.0         255.255.0.0     U     1000   0        0 wlp4s0
192.168.1.0     0.0.0.0         255.255.255.0   U     600    0        0 wlp4s0
192.168.99.0    0.0.0.0         255.255.255.0   U     0      0        0 vboxnet2

# After - EXTERNAL-IP have been assigned from a pool of 10.96.0.0/12
NAMESPACE NAME                  TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)       
efk       service/elasticsearch LoadBalancer   10.104.149.221  10.104.149.221  9200:30826/TCP
efk       service/kibana        LoadBalancer   10.97.198.117   10.97.198.117   5601:32540/TCP

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         vodafone.connec 0.0.0.0         UG    600    0        0 wlp4s0
10.96.0.0       192.168.99.113  255.240.0.0     UG    0      0        0 vboxnet2 # <- new route to minikube NodeIP created 
link-local      0.0.0.0         255.255.0.0     U     1000   0        0 wlp4s0
192.168.1.0     0.0.0.0         255.255.255.0   U     600    0        0 wlp4s0
192.168.99.0    0.0.0.0         255.255.255.0   U     0      0        0 vboxnet2

$ minikube --profile efk tunnel
Status:
        machine: efk
        pid: 8721
        route: 10.96.0.0/12 -> 192.168.99.113 # route status
        minikube: Running
        services: [elasticsearch, kibana]     # exposed services
    errors: 
                minikube: no errors
                router: no errors
                loadbalancer emulator: no errors
# Verify
$ nc -zv 10.97.198.117 5601 # kibana
Connection to 10.97.198.117 5601 port [tcp/*] succeeded!
$ nc -zv 10.104.149.221 9200 # elasticsearch
Connection to 10.104.149.221 9200 port [tcp/*] succeeded!
$ curl http://10.97.198.117:5601/status

Clean up based on info ~/.minikube/tunnels.json when shuts unexpected

minikube tunnel --cleanup

Avoid to typing password for minikube tunnel, by allowing a group or user to execute ip, route commands without asking for password. Example for Ubuntu 18.04 can be seen below.

$ sudo visudo
...
# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL
%admin ALL=(ALL) NOPASSWD: /sbin/ip,/sbin/route  # <- add this line in this order
...

References

• minikube Github
• CHANGELOG.md Github
• Networking.md Github