Kubernetes/minikube
Minikube features
- LoadBalancer - using
minikube tunnel
- Multi-cluster
minikube start --profile cluster-1
- Multi-node cluster
minikube start --profile cluster-2 --nodes 2
- NodePorts - using
minikube service
- Persistent Volumes
- Ingress
- RBAC
- Dashboard -
minikube dashboard
- Container runtimes -
start --container-runtime
- Configure apiserver and kubelet options via command-line flags
Developer friendly features:
- Filesystem mounts
- Addons - a marketplace for developers to share configurations for running services on minikube
- NVIDIA GPU support - for machine learning
Release notes CHANGELOG.mb
Documentation minikube.sigs.k8s.io
Addons
List addons
$ minikube addons list --profile minikube-v1.14.9 |-----------------------------|----------------------|--------------| | ADDON NAME | PROFILE | STATUS | |-----------------------------|----------------------|--------------| | dashboard | minikube-v1.14.9 | disabled | | default-storageclass | minikube-v1.14.9 | enabled β | | efk | minikube-v1.14.9 | disabled | | freshpod | minikube-v1.14.9 | disabled | | gvisor | minikube-v1.14.9 | disabled | | helm-tiller | minikube-v1.14.9 | disabled | | ingress | minikube-v1.14.9 | enabled β | | ingress-dns | minikube-v1.14.9 | enabled β | | istio | minikube-v1.14.9 | disabled | | istio-provisioner | minikube-v1.14.9 | disabled | | logviewer | minikube-v1.14.9 | disabled | | metrics-server | minikube-v1.14.9 | disabled | | nvidia-driver-installer | minikube-v1.14.9 | disabled | | nvidia-gpu-device-plugin | minikube-v1.14.9 | disabled | | registry | minikube-v1.14.9 | disabled | | registry-aliases | minikube-v1.14.9 | disabled | | registry-creds | minikube-v1.14.9 | disabled | | storage-provisioner | minikube-v1.14.9 | enabled β | | storage-provisioner-gluster | minikube-v1.14.9 | disabled | |-----------------------------|----------------------|--------------|
Usage
minikube -p efk start --memory 8192 minikube -p efk addons enable efk # pulling images and setting up can take ~10 min minikube -p efk addons open efk # opens Kibana minikube addons enable heapster # get some insight into CPU # Show images minikube addons images metrics-server - metrics-server has following images: |---------------|--------------------------------------------------------------------------------------------------------------|------------------| | IMAGE NAME | DEFAULT IMAGE | DEFAULT REGISTRY | |---------------|--------------------------------------------------------------------------------------------------------------|------------------| | MetricsServer | metrics-server/metrics-server:v0.4.2@sha256:dbc33d7d35d2a9cc5ab402005aa7a0d13be6192f3550c7d42cba8d2d5e3a5d62 | k8s.gcr.io | |---------------|--------------------------------------------------------------------------------------------------------------|------------------|
Install or upgrade on Linux
# Latest from GitHub VERSION=$(curl --silent "https://api.github.com/repos/kubernetes/minikube/releases/latest" | jq -r .tag_name); echo $VERSION curl -LO https://github.com/kubernetes/minikube/releases/download/$VERSION/minikube-linux-amd64 \ && sudo install minikube-linux-amd64 /usr/local/bin/minikube # Latest from GoogleAPI curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 \ && sudo install minikube-linux-amd64 /usr/local/bin/minikube # Specific version from GitHub curl -LO https://github.com/kubernetes/minikube/releases/download/v1.7.3/minikube-linux-amd64 \ && sudo install minikube-linux-amd64 /usr/local/bin/minikube $> $ minikube version minikube version: v1.17.0
Start on Windows
Install <syntaxhighlightjs lang="powershell"> choco install minikube kubernetes-cli </syntaxhighlightjs>
HyperV
<syntaxhighlightjs lang="powershell">
- HyperV set config (option 1)
minikube config set vm-driver hyperv minikube config set hyperv-virtual-switch "Default Switch" minikube start
- HyperV single command (option 2)
minikube start --vm-driver hyperv --hyperv-virtual-switch k8s-node-1 --profile k8s-node-1
- --hyperv-virtual-switch :- need to exist
</syntaxhighlightjs>
Virtualbox
<syntaxhighlightjs lang="powershell">
minikube start --profile k8s-node-1
- minikube v1.2.0 on windows (amd64)
- Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
- Configuring environment for Kubernetes v1.15.0 on Docker 18.09.6
- Downloading kubelet v1.15.0
- Downloading kubeadm v1.15.0
- Pulling images ...
- Launching Kubernetes ...
- Verifying: apiserver proxy etcd scheduler controller dns
- Done! kubectl is now configured to use "k8s-node-1"
PS C:\Users\Sylwia> kubectl.exe get nodes -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME minikube Ready master 23m v1.15.0 10.0.2.15 <none> Buildroot 2018.05.3 4.15.0 docker://18.9.6 </syntaxhighlightjs>
Start - Linux (Ubuntu)
Start sequence
VERSION=v1.18.9 time minikube start --kubernetes-version $VERSION --profile minikube-$VERSION π [minikube-v1.18.9] minikube v1.19.0 on Ubuntu 20.04 β¨ Automatically selected the docker driver. Other choices: virtualbox, ssh, none π Starting control plane node minikube-v1.18.9 in cluster minikube-v1.18.9 π Pulling base image ... > gcr.io/k8s-minikube/kicbase...: 357.67 MiB / 357.67 MiB 100.00% 4.70 MiB π₯ Creating docker container (CPUs=2, Memory=3900MB) ... π³ Preparing Kubernetes v1.18.9 on Docker 20.10.5 ... βͺ Generating certificates and keys ... βͺ Booting up control plane ... βͺ Configuring RBAC rules ... π Verifying Kubernetes components... βͺ Using image gcr.io/k8s-minikube/storage-provisioner:v5 π Enabled addons: storage-provisioner, default-storageclass π Done! kubectl is now configured to use "minikube-v1.18.9" cluster and "default" namespace by default real 2m4.966s user 0m6.658s sys 0m3.768s $> minikube profile list |-----------------------------|-----------|---------|--------------|------|---------|---------|-------| | Profile | VM Driver | Runtime | IP | Port | Version | Status | Nodes | |-----------------------------|-----------|---------|--------------|------|---------|---------|-------| | minikube-v1.18.9 | docker | docker | 192.168.58.2 | 8443 | v1.18.9 | Running | 1 | |-----------------------------|-----------|---------|--------------|------|---------|---------|-------| $> minikube status -p minikube-v1.18.9 minikube-v1.18.9 type: Control Plane host: Running kubelet: Running apiserver: Running kubeconfig: Configured $> kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME minikube-v1.18.9 Ready master 2m34s v1.18.9 192.168.58.2 <none> Ubuntu 20.04.1 LTS 5.8.0-50-generic docker://20.10.5 $ kubectl get all -o wide --all-namespaces kubectl get all -o wide --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system pod/coredns-66bff467f8-wzjx7 1/1 Running 0 4m59s 172.17.0.2 minikube-v1.18.9 <none> <none> kube-system pod/etcd-minikube-v1.18.9 1/1 Running 0 5m7s 192.168.49.2 minikube-v1.18.9 <none> <none> kube-system pod/kube-apiserver-minikube-v1.18.9 1/1 Running 0 5m7s 192.168.49.2 minikube-v1.18.9 <none> <none> kube-system pod/kube-controller-manager-minikube-v1.18.9 1/1 Running 0 5m7s 192.168.49.2 minikube-v1.18.9 <none> <none> kube-system pod/kube-proxy-zdqc9 1/1 Running 0 4m59s 192.168.49.2 minikube-v1.18.9 <none> <none> kube-system pod/kube-scheduler-minikube-v1.18.9 1/1 Running 0 5m7s 192.168.49.2 minikube-v1.18.9 <none> <none> kube-system pod/storage-provisioner 1/1 Running 1 5m13s 192.168.49.2 minikube-v1.18.9 <none> <none> NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5m15s <none> kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 5m14s k8s-app=kube-dns NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR kube-system daemonset.apps/kube-proxy 1 1 1 1 1 kubernetes.io/os=linux 5m14s kube-proxy k8s.gcr.io/kube-proxy:v1.18.9 k8s-app=kube-proxy NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR kube-system deployment.apps/coredns 1/1 1 1 5m14s coredns k8s.gcr.io/coredns:1.6.7 k8s-app=kube-dns NAMESPACE NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR kube-system replicaset.apps/coredns-66bff467f8 1 1 1 4m59s coredns k8s.gcr.io/coredns:1.6.7 k8s-app=kube-dns,pod-template-hash=66bff467f8
Usage (VirtualBox)
time minikube start --kubernetes-version v1.15.0 --profile minikube-v1.15.0
Get port exposed to host via Virtualbox used by minikube to interact with a cluster
$ sudo ss -ltpn State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=1009,fd=13)) LISTEN 0 10 127.0.0.1:45111 #<-- 0.0.0.0:* users:(("VBoxHeadless",pid=26918,fd=21)) LISTEN 0 5 127.0.0.1:631 0.0.0.0:* users:(("cupsd",pid=27629,fd=7)) LISTEN 0 5 [::1]:631 [::]:* users:(("cupsd",pid=27629,fd=6)) $ vboxmanage showvminfo k8s-v1.15.0 | 'grep port =' vv NIC 1 Rule(0): name = ssh, protocol = tcp, host ip = 127.0.0.1, host port = 45111, guest ip = , guest port = 22 ^^
minikube ssh
|
cat /etc/*rel*
|
---|---|
$ minikube ssh _ _ _ _ ( ) ( ) ___ ___ (_) ___ (_)| |/') _ _ | |_ __ /' _ ` _ `\| |/' _ `\| || , < ( ) ( )| '_`\ /'__`\ | ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )( ___/ (_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____) $ |
cat /etc/*rel* NAME=Buildroot VERSION=2018.05.3 ID=buildroot VERSION_ID=2018.05.3 PRETTY_NAME="Buildroot 2018.05.3" NAME=Buildroot VERSION=2018.05.3 ID=buildroot VERSION_ID=2018.05.3 PRETTY_NAME="Buildroot 2018.05.3" |
#access Kubernetes Dashboard within Minikube minikube dashboard π€ Verifying dashboard health ... π Launching proxy ... π€ Verifying proxy health ... π Opening http://127.0.0.1:44835/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ in your default browser... # Use kubectl to interact with the local cluster [[Kubernetes/Tools#create_pod]] kubectl run --generator=run-pod/v1 hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #correct kubectl run --generator=deployment/apps.v1 hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #deprecated kubectl run ^^default^^ hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #deprecated # Expose a service as a NodePort kubectl expose pod hello-minikube --type=NodePort kubectl expose deployment hello-minikube --type=NodePort #when pod was created via deployment deprecated option # Determine the NodePort of the service kubectl get service <hello-minikube> --output='jsonpath="{.spec.ports[0].nodePort}"' # curl the endpoint minikube ssh curl -s -H 'Cache-Control: no-cache' http://<svc-cluster-ip>:8080 #note 'watch won't work as curl or service proxy #minikube makes it easy to open this exposed endpoint in your browser minikube service hello-minikube minikube stop #stop your local cluster minikube delete #delete your local cluster
Profiles
- List profiles
$> minikube profile list # minikube v1.9.2 |-------------------|-----------|---------|------------|------|----------|---------| | Profile | VM Driver | Runtime | IP | Port | Version | Status | |-------------------|-----------|---------|------------|------|----------|---------| | minikube-v1.15.11 | docker | docker | 172.17.0.2 | 8443 | v1.15.11 | Stopped | |-------------------|-----------|---------|------------|------|----------|---------|
- Set and switch in between profiles
Setting the minikube
profile does automatically change Kubernetes kubectl config set-context
context.
minikube profile minikube-v1.15.11 # set the profile 'minikube-v1.15.11' β minikube profile was successfully set to minikube-v1.15.11 # Configure the persistent default profile minikube config set profile minikube-v1.15.11 minikube config view - profile: minikube-v1.15.11 # Manually switch kubectl context - not needed when using 'minikube profile' kubectl config set-context minikube Context "minikube" modified.
- Profile local config files
$> ls -1 ~/.minikube/profiles/ # list all profiles minikube-v1.15.11 minikube # Minikube show current profile $> minikube profile minikube-v1.15.11 $> cat ~/.minikube/config/config.json | jq -r .profile #alternative option # Kubectl show current context $ kubectl config current-context $ kubectl config get-contexts #get-clusters CURRENT NAME CLUSTER AUTHINFO NAMESPACE * minikube-v1.15.11 minikube-v1.15.11 minikube-v1.15.11 minikube minikube minikube
- Clean up if needed
minikube delete -p <profile> kubectl config delete-context <context-name> kubectl config delete-context <delete-cluster>
jq .MachineConfig config.json
|
jq .KubernetesConfig config.json
|
---|---|
{ "KubernetesVersion": "v1.13.7", "NodeIP": "192.168.99.102", "NodePort": 8443, "NodeName": "minikube", "APIServerName": "minikubeCA", "APIServerNames": null, "APIServerIPs": null, "DNSDomain": "cluster.local", "ContainerRuntime": "docker", "CRISocket": "", "NetworkPlugin": "", "FeatureGates": "", "ServiceCIDR": "10.96.0.0/12", "ImageRepository": "", "ExtraOptions": null, "ShouldLoadCachedImages": true, "EnableDefaultCNI": false } |
{ "KeepContext": false, "MinikubeISO": "https://storage.googleapis.com/minikube/iso/minikube-v1.2.0.iso", "Memory": 2048, "CPUs": 2, "DiskSize": 20000, "VMDriver": "virtualbox", "ContainerRuntime": "docker", "HyperkitVpnKitSock": "", "HyperkitVSockPorts": [], "XhyveDiskDriver": "ahci-hd", "DockerEnv": null, "InsecureRegistry": null, "RegistryMirror": null, "HostOnlyCIDR": "192.168.99.1/24", "HypervVirtualSwitch": "", "KvmNetwork": "default", "DockerOpt": null, "DisableDriverMounts": false, "NFSShare": [], "NFSSharesRoot": "/nfsshares", "UUID": "", "GPU": false, "Hidden": false, "NoVTXCheck": false } |
Setting default configuration
It's useful to set default configuration, eg. you always work with AWS EKS, so it comes at certain versioning we could set as a default if not explicitly specified.
List configurable options
$ minikube config config modifies minikube config files using subcommands like "minikube config set driver kvm" Configurable fields: * driver * log_dir * ShowDriverDeprecationNotification * vm-driver * kubernetes-version * ShowBootstrapperDeprecationNotification * container-runtime * iso-url * insecure-registry * feature-gates * WantUpdateNotification * hyperv-virtual-switch * v * ReminderWaitPeriodInHours * disable-driver-mounts * cpus * WantReportError * cache * disk-size * WantReportErrorPrompt * embed-certs * host-only-cidr * WantKubectlDownloadMsg * native-ssh * memory * WantNoneDriverWarning * profile * bootstrapper Available Commands: get Gets the value of PROPERTY_NAME from the minikube config file set Sets an individual value in a minikube config file unset unsets an individual value in a minikube config file view Display values currently set in the minikube config file Usage: minikube config SUBCOMMAND [flags] [options]
- Set default version of Kubernetes
$ minikube config set kubernetes-version v1.14.9 $ minikube config view - kubernetes-version: v1.14.9
Loadbalancer - minikube tunnel
A LoadBalancer service is the standard way to expose a service to the internet. With this method, each service gets itβs own IP address. In minikube services of type LoadBalancer can be exposed via the minikube tunnel
command. It will run until Ctrl-C
is hit.
minikube tunnel
runs as a separate daemon, creating a network route on the host to the service CIDR of the cluster using the clusterβs IP address as a gateway. The tunnel command exposes the external IP directly to any program running on the host operating system. The low-level commands used to manage routes are /sbin/ip,/sbin/route
that optionally can be added to suedors file to make a life a little easier.
$ minikube profile list |-----------------|------------|----------------|-----------|--------------------| | Profile | VM Driver | NodeIP | Node Port | Kubernetes Version | |-----------------|------------|----------------|-----------|--------------------| | efk | virtualbox | 192.168.99.113 | 8443 | v1.16.0 | |-----------------|------------|----------------|-----------|--------------------| # Before $ kubectl -n efk get svc # note <pending> EXTERNAL-IP NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) efk service/elasticsearch LoadBalancer 10.104.149.221 <pending> 9200:30826/TCP efk service/kibana LoadBalancer 10.97.198.117 <pending> 5601:32540/TCP $ route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default vodafone.connec 0.0.0.0 UG 600 0 0 wlp4s0 link-local 0.0.0.0 255.255.0.0 U 1000 0 0 wlp4s0 192.168.1.0 0.0.0.0 255.255.255.0 U 600 0 0 wlp4s0 192.168.99.0 0.0.0.0 255.255.255.0 U 0 0 0 vboxnet2 # After - EXTERNAL-IP have been assigned from a pool of 10.96.0.0/12 NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) efk service/elasticsearch LoadBalancer 10.104.149.221 10.104.149.221 9200:30826/TCP efk service/kibana LoadBalancer 10.97.198.117 10.97.198.117 5601:32540/TCP Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default vodafone.connec 0.0.0.0 UG 600 0 0 wlp4s0 10.96.0.0 192.168.99.113 255.240.0.0 UG 0 0 0 vboxnet2 # <- new route to minikube NodeIP created link-local 0.0.0.0 255.255.0.0 U 1000 0 0 wlp4s0 192.168.1.0 0.0.0.0 255.255.255.0 U 600 0 0 wlp4s0 192.168.99.0 0.0.0.0 255.255.255.0 U 0 0 0 vboxnet2 $ minikube --profile efk tunnel Status: machine: efk pid: 8721 route: 10.96.0.0/12 -> 192.168.99.113 # route status minikube: Running services: [elasticsearch, kibana] # exposed services errors: minikube: no errors router: no errors loadbalancer emulator: no errors # Verify $ nc -zv 10.97.198.117 5601 # kibana Connection to 10.97.198.117 5601 port [tcp/*] succeeded! $ nc -zv 10.104.149.221 9200 # elasticsearch Connection to 10.104.149.221 9200 port [tcp/*] succeeded! $ curl http://10.97.198.117:5601/status
Clean up based on info ~/.minikube/tunnels.json
when shuts unexpected
minikube tunnel --cleanup
Avoid to typing password for minikube tunnel
, by allowing a group or user to execute ip, route
commands without asking for password. Example for Ubuntu 18.04 can be seen below.
$ sudo visudo ... # Members of the admin group may gain root privileges %admin ALL=(ALL) ALL %admin ALL=(ALL) NOPASSWD: /sbin/ip,/sbin/route # <- add this line in this order ...
multi node cluster with kvm
This is now supported natively using --nodes
flag:
minikube start --profile minikube-v1.15.11 --nodes=2 --kubernetes-version v1.15.11 minikube ssh --profile minikube-v1.15.11 --node=m02 minikube --profile minikube-v1.15.11 node add # add new node minikube --profile minikube-v1.15.11 node delete m03 # delete $ kubectl get nodes NAME STATUS ROLES AGE VERSION minikube-v1.15.11 Ready master 3h25m v1.15.11 minikube-v1.15.11-m02 NotReady <none> 3h24m v1.15.11 minikube-v1.15.11-m03 NotReady <none> 16m v1.15.11 # <- add / remove
Nodes names have following naming convention
- node-1-name:
minikube-v1.14.9
- node-2-name:
minikube-v1.14.9-m02
ingress and local DNS (ingress-dns)
TODO:
- Setting up ExternalDNS for Headless Services
- minikube-ingress-dns gitlab
- ingress-minikube Kubernetes docs
Add required addons
minikube addons enable ingress --profile minikube-v1.14.9 π The 'ingress' addon is enabled minikube addons enable ingress-dns --profile minikube-v1.14.9 π The 'ingress-dns' addon is enabled
- Add minikube ip as a dns server
Ubuntu 18.04 LTS, configure resolvconf
sudo apt install resolvconf sudo -p /etc/resolvconf/resolv.conf.d # possibly created by installation package sudo vi /etc/resolvconf/resolv.conf.d/base minikube ip --profile minikube-v1.14.9 192.168.99.126 # Update resolverconf sudo vi /etc/resolvconf/resolv.conf.d/base search test nameserver 192.168.99.126 timeout 5 # Apply changes sudo resolvconf -u systemctl disable --now resolvconf.service # Test, Add the test ingress kubectl apply -f https://raw.githubusercontent.com/kubernetes/minikube/master/deploy/addons/ingress-dns/example/example.yaml # Validate DNS queries are returning A records nslookup hello-john.test $(minikube ip --profile minikube-v1.14.9) Server: 192.168.99.126 Address: 192.168.99.126#53 Non-authoritative answer: Name: hello-john.test Address: 192.168.99.126 Name: hello-john.test Address: 192.168.99.126 # <- this is ingress IP address $ kubectl -n kube-system get ingresses.networking.k8s.io example-ingress NAME HOSTS ADDRESS PORTS AGE example-ingress hello-john.test,hello-jane.test 192.168.99.126 80 102s
Note: .local
is a reserved TLD, do not use as this is a reserved TLD for mDNS and bind9 DNS servers
Note: .localhost
domains will not correctly resolve on chromium since it is used as a loopback address. Instead use .test
, .example
, or .invalid
References
- minikube Github
- CHANGELOG.md Github
- Networking.md Github