Kubernetes/minikube
Minikube features v1.6.2
- LoadBalancer - using
minikube tunnel
- Multi-cluster - using
minikube start -p <name>
- NodePorts - using
minikube service
- Persistent Volumes
- Ingress
- RBAC
- Dashboard -
minikube dashboard
- Container runtimes -
start --container-runtime
- Configure apiserver and kubelet options via command-line flags
Developer friendly features:
Release notes CHANGELOG.mb
Documentation minikube.sigs.k8s.io
Addons
List addons
$ minikube addons list - addon-manager: enabled - dashboard: enabled - default-storageclass: enabled - efk: disabled - freshpod: disabled - gvisor: disabled - heapster: disabled - helm-tiller: disabled - ingress: disabled - ingress-dns: disabled - logviewer: disabled - metrics-server: disabled - nvidia-driver-installer: disabled - nvidia-gpu-device-plugin: disabled - registry: disabled - registry-creds: disabled - storage-provisioner: enabled - storage-provisioner-gluster: disabled
Usage
minikube -p efk start --memory 8192
minikube -p efk addons enable efk # pulling images and setting up can take ~10 min
minikube -p efk addons open efk # opens Kibana
minikube addons enable heapster #get some insight into CPU
Install or upgrade on Linux
# Latest from GitHub
VERSION=$(curl --silent "https://api.github.com/repos/kubernetes/minikube/releases/latest" | jq -r .tag_name)
curl -LO https://github.com/kubernetes/minikube/releases/download/$VERSION/minikube-linux-amd64 \
&& sudo install minikube-linux-amd64 /usr/local/bin/minikube
# Latest from GoogleAPI
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 \
&& sudo install minikube-linux-amd64 /usr/local/bin/minikube
# Specific version from GitHub
curl -LO https://github.com/kubernetes/minikube/releases/download/v1.7.3/minikube-linux-amd64 \
&& sudo install minikube-linux-amd64 /usr/local/bin/minikube
$> $ minikube version
minikube version: v1.9.2
commit: 93af9c1e43cab9618e301bc9fa720c63d5efa393
Start on Windows
Install <syntaxhighlightjs lang="powershell"> choco install minikube kubernetes-cli </syntaxhighlightjs>
HyperV
<syntaxhighlightjs lang="powershell">
- HyperV set config (option 1)
minikube config set vm-driver hyperv minikube config set hyperv-virtual-switch "Default Switch" minikube start
- HyperV single command (option 2)
minikube start --vm-driver hyperv --hyperv-virtual-switch k8s-node-1 --profile k8s-node-1
- --hyperv-virtual-switch :- need to exist
</syntaxhighlightjs>
Virtualbox
<syntaxhighlightjs lang="powershell">
minikube start --profile k8s-node-1
- minikube v1.2.0 on windows (amd64)
- Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
- Configuring environment for Kubernetes v1.15.0 on Docker 18.09.6
- Downloading kubelet v1.15.0
- Downloading kubeadm v1.15.0
- Pulling images ...
- Launching Kubernetes ...
- Verifying: apiserver proxy etcd scheduler controller dns
- Done! kubectl is now configured to use "k8s-node-1"
PS C:\Users\Sylwia> kubectl.exe get nodes -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME minikube Ready master 23m v1.15.0 10.0.2.15 <none> Buildroot 2018.05.3 4.15.0 docker://18.9.6 </syntaxhighlightjs>
Start
Example of start sequence
$> time minikube start --kubernetes-version v1.14.9 --profile minikube-v1.14.9 # minikube v1.9.2
😄 [minikube-v1.14.9] minikube v1.9.2 on Ubuntu 18.04
✨ Automatically selected the virtualbox driver
💿 Downloading VM boot image ...
> minikube-v1.9.0.iso.sha256: 65 B / 65 B [--------------] 100.00% ? p/s 0s
> minikube-v1.9.0.iso: 174.93 MiB / 174.93 MiB [-] 100.00% 4.05 MiB p/s 44s
👍 Starting control plane node m01 in cluster minikube-v1.14.9
💾 Downloading Kubernetes v1.14.9 preload ...
> preloaded-images-k8s-v2-v1.14.9-docker-overlay2-amd64.tar.lz4: 496.25 MiB
🔥 Creating virtualbox VM (CPUs=2, Memory=3900MB, Disk=20000MB) ...
🐳 Preparing Kubernetes v1.14.9 on Docker 19.03.8 ...
🌟 Enabling addons: default-storageclass, storage-provisioner
🏄 Done! kubectl is now configured to use "minikube-v1.14.9"
real 4m9.678s
user 0m21.710s
sys 0m30.296s
$ minikube profile list
|----------------------------------|------------|---------|----------------|------|---------|---------|
| Profile | VM Driver | Runtime | IP | Port | Version | Status |
|----------------------------------|------------|---------|----------------|------|---------|---------|
| minikube-v1.14.9 | virtualbox | docker | 192.168.99.123 | 8443 | v1.14.9 | Running |
|----------------------------------|------------|---------|----------------|------|---------|---------|
$ minikube status -p minikube-v1.14.9
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
minikube-v1.14.9 Ready master 6m57s v1.14.9 192.168.99.123 <none> Buildroot 2019.02.10 4.19.107 docker://19.3.8
$ kubectl get all -o wide --all-namespaces
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
minikube-v1.14.9 Ready master 6m57s v1.14.9 192.168.99.123 <none> Buildroot 2019.02.10 4.19.107 docker://19.3.8
pipaw@pp-tp52s ~ $ kubectl get all -o wide --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system pod/coredns-6dcc67dcbc-48k6v 1/1 Running 0 8m16s 172.17.0.2 minikube-v1.14.9 <none> <none>
kube-system pod/coredns-6dcc67dcbc-4ftc4 1/1 Running 0 8m16s 172.17.0.3 minikube-v1.14.9 <none> <none>
kube-system pod/etcd-minikube-v1.14.9 1/1 Running 0 7m21s 192.168.99.123 minikube-v1.14.9 <none> <none>
kube-system pod/kube-apiserver-minikube-v1.14.9 1/1 Running 0 7m21s 192.168.99.123 minikube-v1.14.9 <none> <none>
kube-system pod/kube-controller-manager-minikube-v1.14.9 1/1 Running 0 7m26s 192.168.99.123 minikube-v1.14.9 <none> <none>
kube-system pod/kube-proxy-kjjrx 1/1 Running 0 8m16s 192.168.99.123 minikube-v1.14.9 <none> <none>
kube-system pod/kube-scheduler-minikube-v1.14.9 1/1 Running 0 7m22s 192.168.99.123 minikube-v1.14.9 <none> <none>
kube-system pod/storage-provisioner 1/1 Running 1 8m22s 192.168.99.123 minikube-v1.14.9 <none> <none>
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8m25s <none>
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 8m24s k8s-app=kube-dns
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR
kube-system daemonset.apps/kube-proxy 1 1 1 1 1 <none> 8m23s kube-proxy k8s.gcr.io/kube-proxy:v1.14.9 k8s-app=kube-proxy
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
kube-system deployment.apps/coredns 2/2 2 2 8m24s coredns k8s.gcr.io/coredns:1.3.1 k8s-app=kube-dns
NAMESPACE NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
kube-system replicaset.apps/coredns-6dcc67dcbc 2 2 2 8m16s coredns k8s.gcr.io/coredns:1.3.1 k8s-app=kube-dns,pod-template-hash=6dcc67dcbc
Usage
time minikube start #start a cluster by running
# Start another local clusters named: --profile can be abbreviated with -p, default is 'minikube'
time minikube start --kubernetes-version v1.14.9 --profile minikube-v1.14.9-aws # AWS versions v1.14.9-eks-c0eccc
time minikube start --kubernetes-version v1.15.0 --profile minikube-v1.15.0
Multi-profiles
$ minikube profile list
|-----------------|------------|----------------|-----------|--------------------|
| Profile | VM Driver | NodeIP | Node Port | Kubernetes Version |
|-----------------|------------|----------------|-----------|--------------------|
| k8s-v1.13.7-aws | virtualbox | 192.168.99.105 | 8443 | v1.15.0 |
| k8s-v1.15.0 | virtualbox | 192.168.99.104 | 8443 | v1.15.0 |
|-----------------|------------|----------------|-----------|--------------------|
Get port exposed to host via Virtualbox used by minikube to interact with a cluster
$ sudo ss -ltpn
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=1009,fd=13))
LISTEN 0 10 127.0.0.1:45111 #<-- 0.0.0.0:* users:(("VBoxHeadless",pid=26918,fd=21))
LISTEN 0 5 127.0.0.1:631 0.0.0.0:* users:(("cupsd",pid=27629,fd=7))
LISTEN 0 5 [::1]:631 [::]:* users:(("cupsd",pid=27629,fd=6))
$ vboxmanage showvminfo k8s-v1.15.0 | 'grep port =' vv
NIC 1 Rule(0): name = ssh, protocol = tcp, host ip = 127.0.0.1, host port = 45111, guest ip = , guest port = 22
^^
minikube ssh
|
cat /etc/*rel*
|
---|---|
$ minikube ssh
_ _
_ _ ( ) ( )
___ ___ (_) ___ (_)| |/') _ _ | |_ __
/' _ ` _ `\| |/' _ `\| || , < ( ) ( )| '_`\ /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )( ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)
$
|
cat /etc/*rel*
NAME=Buildroot
VERSION=2018.05.3
ID=buildroot
VERSION_ID=2018.05.3
PRETTY_NAME="Buildroot 2018.05.3"
NAME=Buildroot
VERSION=2018.05.3
ID=buildroot
VERSION_ID=2018.05.3
PRETTY_NAME="Buildroot 2018.05.3"
|
#access Kubernetes Dashboard within Minikube
minikube dashboard
🤔 Verifying dashboard health ...
🚀 Launching proxy ...
🤔 Verifying proxy health ...
🎉 Opening http://127.0.0.1:44835/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ in your default browser...
#Use kubectl to interact with the local cluster
kubectl run --generator=run-pod/v1 hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #correct
kubectl run --generator=deployment/apps.v1 hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #deprecated
kubectl run ^^default^^ hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #deprecated
#Expose a service as a NodePort
kubectl expose pod hello-minikube --type=NodePort
kubectl expose deployment hello-minikube --type=NodePort #when pod was created via deployment deprecated option
#Determine the NodePort of the service
kubectl get service <hello-minikube> --output='jsonpath="{.spec.ports[0].nodePort}"'
#curl this endpoint
minikube ssh
curl -s -H 'Cache-Control: no-cache' http://<svc-cluster-ip>:8080 #note 'watch won't work as curl or service proxy
#minikube makes it easy to open this exposed endpoint in your browser
minikube service hello-minikube
minikube stop #stop your local cluster
minikube delete #delete your local cluster
Working with profiles
$ ls -1 ~/.minikube/profiles/ #list all profiles
aws-v1.13.7-cluster
minikube #default profile
#Minikube show current profile
$ minikube profile
aws-v1.13.7-cluster
$ cat ~/.minikube/config/config.json | jq -r .profile #alternative option
#Kubectl show current context
$ kubectl config current-context
$ kubectl config get-contexts #get-clusters
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* aws-v1.13.7-cluster aws-v1.13.7-cluster aws-v1.13.7-cluster
minikube minikube minikube
Switching the Minikube profile does automatically switch the k8s context, and vice versa (v1.2.0 - 2019-06-24)
#Switch minikube profile
$ minikube profile minikube #switch to profile: minikube
✅ minikube profile was successfully set to minikube
#Switch kubectl context
$ kubectl config set-context minikube
Context "minikube" modified.
Clean up if needed
minikube delete -p <profile> kubectl config delete-context <context-name> kubectl config delete-context <delete-cluster>
jq .MachineConfig config.json
|
jq .KubernetesConfig config.json
|
---|---|
{
"KubernetesVersion": "v1.13.7",
"NodeIP": "192.168.99.102",
"NodePort": 8443,
"NodeName": "minikube",
"APIServerName": "minikubeCA",
"APIServerNames": null,
"APIServerIPs": null,
"DNSDomain": "cluster.local",
"ContainerRuntime": "docker",
"CRISocket": "",
"NetworkPlugin": "",
"FeatureGates": "",
"ServiceCIDR": "10.96.0.0/12",
"ImageRepository": "",
"ExtraOptions": null,
"ShouldLoadCachedImages": true,
"EnableDefaultCNI": false
}
|
{
"KeepContext": false,
"MinikubeISO": "https://storage.googleapis.com/minikube/iso/minikube-v1.2.0.iso",
"Memory": 2048,
"CPUs": 2,
"DiskSize": 20000,
"VMDriver": "virtualbox",
"ContainerRuntime": "docker",
"HyperkitVpnKitSock": "",
"HyperkitVSockPorts": [],
"XhyveDiskDriver": "ahci-hd",
"DockerEnv": null,
"InsecureRegistry": null,
"RegistryMirror": null,
"HostOnlyCIDR": "192.168.99.1/24",
"HypervVirtualSwitch": "",
"KvmNetwork": "default",
"DockerOpt": null,
"DisableDriverMounts": false,
"NFSShare": [],
"NFSSharesRoot": "/nfsshares",
"UUID": "",
"GPU": false,
"Hidden": false,
"NoVTXCheck": false
}
|
Loadbalancer - minikube tunnel
A LoadBalancer service is the standard way to expose a service to the internet. With this method, each service gets it’s own IP address. In minikube services of type LoadBalancer can be exposed via the minikube tunnel
command. It will run until Ctrl-C
is hit.
minikube tunnel
runs as a separate daemon, creating a network route on the host to the service CIDR of the cluster using the cluster’s IP address as a gateway. The tunnel command exposes the external IP directly to any program running on the host operating system. The low-level commands used to manage routes are /sbin/ip,/sbin/route
that optionally can be added to suedors file to make a life a little easier.
$ minikube profile list
|-----------------|------------|----------------|-----------|--------------------|
| Profile | VM Driver | NodeIP | Node Port | Kubernetes Version |
|-----------------|------------|----------------|-----------|--------------------|
| efk | virtualbox | 192.168.99.113 | 8443 | v1.16.0 |
|-----------------|------------|----------------|-----------|--------------------|
# Before
$ kubectl -n efk get svc # note <pending> EXTERNAL-IP
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
efk service/elasticsearch LoadBalancer 10.104.149.221 <pending> 9200:30826/TCP
efk service/kibana LoadBalancer 10.97.198.117 <pending> 5601:32540/TCP
$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default vodafone.connec 0.0.0.0 UG 600 0 0 wlp4s0
link-local 0.0.0.0 255.255.0.0 U 1000 0 0 wlp4s0
192.168.1.0 0.0.0.0 255.255.255.0 U 600 0 0 wlp4s0
192.168.99.0 0.0.0.0 255.255.255.0 U 0 0 0 vboxnet2
# After - EXTERNAL-IP have been assigned from a pool of 10.96.0.0/12
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
efk service/elasticsearch LoadBalancer 10.104.149.221 10.104.149.221 9200:30826/TCP
efk service/kibana LoadBalancer 10.97.198.117 10.97.198.117 5601:32540/TCP
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default vodafone.connec 0.0.0.0 UG 600 0 0 wlp4s0
10.96.0.0 192.168.99.113 255.240.0.0 UG 0 0 0 vboxnet2 # <- new route to minikube NodeIP created
link-local 0.0.0.0 255.255.0.0 U 1000 0 0 wlp4s0
192.168.1.0 0.0.0.0 255.255.255.0 U 600 0 0 wlp4s0
192.168.99.0 0.0.0.0 255.255.255.0 U 0 0 0 vboxnet2
$ minikube --profile efk tunnel
Status:
machine: efk
pid: 8721
route: 10.96.0.0/12 -> 192.168.99.113 # route status
minikube: Running
services: [elasticsearch, kibana] # exposed services
errors:
minikube: no errors
router: no errors
loadbalancer emulator: no errors
# Verify
$ nc -zv 10.97.198.117 5601 # kibana
Connection to 10.97.198.117 5601 port [tcp/*] succeeded!
$ nc -zv 10.104.149.221 9200 # elasticsearch
Connection to 10.104.149.221 9200 port [tcp/*] succeeded!
$ curl http://10.97.198.117:5601/status
Clean up based on info ~/.minikube/tunnels.json
when shuts unexpected
minikube tunnel --cleanup
Avoid to typing password for minikube tunnel
, by allowing a group or user to execute ip, route
commands without asking for password. Example for Ubuntu 18.04 can be seen below.
$ sudo visudo
...
# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL
%admin ALL=(ALL) NOPASSWD: /sbin/ip,/sbin/route # <- add this line in this order
...
multi node cluster with kvm
This is now supported natively using --nodes
flag:
minikube start --profile minikube-v1.14.9 --nodes=2 --kubernetes-version v1.14.9 minikube ssh --profile minikube-v1.14.9 --node=m02
Nodes names have following naming convention
- node-1-name:
minikube-v1.14.9
- node-2-name:
minikube-v1.14.9-m02
References
- minikube Github
- CHANGELOG.md Github
- Networking.md Github