Difference between revisions of "Kubernetes/minikube"

From Ever changing code
Jump to navigation Jump to search
 
(38 intermediate revisions by the same user not shown)
Line 1: Line 1:
= Minikube features v1.6.2 =
= [https://minikube.sigs.k8s.io/docs/ Minikube] features =
* [https://minikube.sigs.k8s.io/docs/tasks/loadbalancer/ LoadBalancer] - using <code>minikube tunnel</code>
* [https://minikube.sigs.k8s.io/docs/tasks/loadbalancer/ LoadBalancer] - using <code>minikube tunnel</code>
* Multi-cluster - using <code>minikube start -p <name></code>
* Multi-cluster <code>minikube start --profile cluster-1</code>
* Multi-node cluster <code>minikube start --profile cluster-2 --nodes 2</code>
* NodePorts - using <code>minikube service</code>
* NodePorts - using <code>minikube service</code>
* [https://minikube.sigs.k8s.io/docs/reference/persistent_volumes/ Persistent Volumes]
* [https://minikube.sigs.k8s.io/docs/reference/persistent_volumes/ Persistent Volumes]
* Ingress
* [https://kubernetes.io/docs/tasks/access-application-cluster/ingress-minikube/ Ingress]
* RBAC
* RBAC
* [https://minikube.sigs.k8s.io/docs/tasks/dashboard/ Dashboard] - <code>minikube dashboard</code>
* [https://minikube.sigs.k8s.io/docs/tasks/dashboard/ Dashboard] - <code>minikube dashboard</code>
Line 11: Line 12:
Developer friendly features:
Developer friendly features:
* [https://minikube.sigs.k8s.io/docs/tasks/mount/ Filesystem mounts]
* [https://minikube.sigs.k8s.io/docs/tasks/mount/ Filesystem mounts]
 
* Addons - a marketplace for developers to share configurations for running services on minikube
* NVIDIA GPU support - for machine learning


Release notes [https://github.com/kubernetes/minikube/blob/master/CHANGELOG.md CHANGELOG.mb]
Release notes [https://github.com/kubernetes/minikube/blob/master/CHANGELOG.md CHANGELOG.mb]
Line 20: Line 22:
List addons
List addons
<source lang=bash>
<source lang=bash>
$ minikube addons list  
$ minikube addons list --profile minikube-v1.14.9
- addon-manager: enabled
|-----------------------------|----------------------|--------------|
- dashboard: enabled
|        ADDON NAME          |        PROFILE      |    STATUS    |
- default-storageclass: enabled
|-----------------------------|----------------------|--------------|
- efk: disabled
| dashboard                   | minikube-v1.14.9    | disabled    |
- freshpod: disabled
| default-storageclass       | minikube-v1.14.9    | enabled ✅  |
- gvisor: disabled
| efk                        | minikube-v1.14.9    | disabled     |
- heapster: disabled
| freshpod                    | minikube-v1.14.9    | disabled     |
- helm-tiller: disabled
| gvisor                     | minikube-v1.14.9    | disabled     |
- ingress: disabled
| helm-tiller                 | minikube-v1.14.9    | disabled     |
- ingress-dns: disabled
| ingress                    | minikube-v1.14.9    | enabled ✅  |
- logviewer: disabled
| ingress-dns                | minikube-v1.14.9    | enabled ✅  |
- metrics-server: disabled
| istio                      | minikube-v1.14.9    | disabled     |
- nvidia-driver-installer: disabled
| istio-provisioner          | minikube-v1.14.9    | disabled     |
- nvidia-gpu-device-plugin: disabled
| logviewer                  | minikube-v1.14.9    | disabled     |
- registry: disabled
| metrics-server             | minikube-v1.14.9    | disabled     |
- registry-creds: disabled
| nvidia-driver-installer     | minikube-v1.14.9    | disabled     |
- storage-provisioner: enabled
| nvidia-gpu-device-plugin   | minikube-v1.14.9    | disabled     |
- storage-provisioner-gluster: disabled
| registry                    | minikube-v1.14.9    | disabled    |
| registry-aliases            | minikube-v1.14.9    | disabled     |
| registry-creds             | minikube-v1.14.9    | disabled     |
| storage-provisioner         | minikube-v1.14.9    | enabled ✅  |
| storage-provisioner-gluster | minikube-v1.14.9    | disabled     |
|-----------------------------|----------------------|--------------|
</source>
</source>


Line 48: Line 55:
minikube -p efk addons open efk      # opens Kibana
minikube -p efk addons open efk      # opens Kibana


minikube addons enable heapster #get some insight into CPU
minikube addons enable heapster       # get some insight into CPU
 
# Show images
minikube addons images metrics-server
  - metrics-server has following images:
|---------------|--------------------------------------------------------------------------------------------------------------|------------------|
|  IMAGE NAME  |                                                DEFAULT IMAGE                                                | DEFAULT REGISTRY |
|---------------|--------------------------------------------------------------------------------------------------------------|------------------|
| MetricsServer | metrics-server/metrics-server:v0.4.2@sha256:dbc33d7d35d2a9cc5ab402005aa7a0d13be6192f3550c7d42cba8d2d5e3a5d62 | k8s.gcr.io      |
|---------------|--------------------------------------------------------------------------------------------------------------|------------------|
 
</source>
</source>


= Install or upgrade on Linux =
= Install or upgrade on Linux =
<source lang=bash>
<source lang=bash>
# Latest from GitHub
VERSION=$(curl --silent "https://api.github.com/repos/kubernetes/minikube/releases/latest" | jq -r .tag_name); echo $VERSION
curl -LO https://github.com/kubernetes/minikube/releases/download/$VERSION/minikube-linux-amd64 \
      && sudo install minikube-linux-amd64 /usr/local/bin/minikube
# Latest from GoogleAPI
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 \
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 \
       && sudo install minikube-linux-amd64 /usr/local/bin/minikube
       && sudo install minikube-linux-amd64 /usr/local/bin/minikube


curl -LO https://github.com/kubernetes/minikube/releases/download/v1.6.2/minikube-linux-amd64 \
# Specific version from GitHub
curl -LO https://github.com/kubernetes/minikube/releases/download/v1.7.3/minikube-linux-amd64 \
       && sudo install minikube-linux-amd64 /usr/local/bin/minikube
       && sudo install minikube-linux-amd64 /usr/local/bin/minikube


$> minikube version  
$> $ minikube version  
minikube version: v1.6.2
minikube version: v1.17.0
commit: 54f28ac5d3a815d1196cd5d57d707439ee4bb392
</source>
</source>


Line 102: Line 125:
</syntaxhighlightjs>
</syntaxhighlightjs>


= Start =
= Start - Linux (Ubuntu)=
Example of start sequence
Start sequence
<source lang=bash>
<source lang=bash>
minikube start  
VERSION=v1.18.9
😄  minikube v1.2.0 on linux (amd64)
time minikube start --kubernetes-version $VERSION --profile minikube-$VERSION
🔥 Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
😄  [minikube-v1.18.9] minikube v1.19.0 on Ubuntu 20.04
🐳 Configuring environment for Kubernetes v1.15.0 on Docker 18.09.6
Automatically selected the docker driver. Other choices: virtualbox, ssh, none
E0728 00:14:31.657125  26453 start.go:403] Error caching images: Caching images for kubeadm: caching images: caching image /home/user1/.minikube/cache/images/k8s.gcr.io/k8s-dns-kube-dns-amd64_1.14.13: stream error: stream ID 25; INTERNAL_ERROR
👍 Starting control plane node minikube-v1.18.9 in cluster minikube-v1.18.9
Unable to load cached images: loading cached images: loading image /home/pipaw/.minikube/cache/images/k8s.gcr.io/k8s-dns-sidecar-amd64_1.14.13: stat /home/user1/.minikube/cache/images/k8s.gcr.io/k8s-dns-sidecar-amd64_1.14.13: no such file or directory
🚜 Pulling base image ...
💾  Downloading kubeadm v1.15.0
    > gcr.io/k8s-minikube/kicbase...: 357.67 MiB / 357.67 MiB  100.00% 4.70 MiB
💾  Downloading kubelet v1.15.0
🔥 Creating docker container (CPUs=2, Memory=3900MB) ...
🚜  Pulling images ...
🐳  Preparing Kubernetes v1.18.9 on Docker 20.10.5 ...
🚀 Launching Kubernetes ...  
    ▪ Generating certificates and keys ...
Verifying: apiserver proxy etcd scheduler controller dns
    ▪ Booting up control plane ...
🏄  Done! kubectl is now configured to use "minikube"
    ▪ Configuring RBAC rules ...
🔎 Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube-v1.18.9" cluster and "default" namespace by default
 
real 2m4.966s
user 0m6.658s
sys 0m3.768s
 
 
$> minikube profile list
|-----------------------------|-----------|---------|--------------|------|---------|---------|-------|
|          Profile          | VM Driver | Runtime |      IP      | Port | Version | Status  | Nodes |
|-----------------------------|-----------|---------|--------------|------|---------|---------|-------|
| minikube-v1.18.9            | docker    | docker  | 192.168.58.2 | 8443 | v1.18.9 | Running |    1 |
|-----------------------------|-----------|---------|--------------|------|---------|---------|-------|


$ minikube status
$> minikube status -p minikube-v1.18.9
minikube-v1.18.9
type: Control Plane
host: Running
host: Running
kubelet: Running
kubelet: Running
apiserver: Running
apiserver: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.101
kubeconfig: Configured
 
$> kubectl get nodes -o wide
NAME              STATUS  ROLES    AGE    VERSION  INTERNAL-IP    EXTERNAL-IP  OS-IMAGE            KERNEL-VERSION    CONTAINER-RUNTIME
minikube-v1.18.9  Ready    master  2m34s  v1.18.9  192.168.58.2  <none>        Ubuntu 20.04.1 LTS  5.8.0-50-generic  docker://20.10.5


$ kubectl get nodes -o wide
NAME      STATUS  ROLES    AGE  VERSION  INTERNAL-IP  EXTERNAL-IP  OS-IMAGE              KERNEL-VERSION  CONTAINER-RUNTIME
minikube  Ready    master  8h    v1.15.0  10.0.2.15    <none>        Buildroot 2018.05.3  4.15.0          docker://18.9.6


$ kubectl get all -o wide --all-namespaces
$ kubectl get all -o wide --all-namespaces
NAMESPACE    NAME                                   READY  STATUS    RESTARTS  AGE   IP           NODE       NOMINATED NODE  READINESS GATES
kubectl get all -o wide --all-namespaces
kube-system  pod/coredns-5c98db65d4-d688k          1/1    Running  0          8h    172.17.0.2   minikube   <none>          <none>
NAMESPACE    NAME                                                 READY  STATUS    RESTARTS  AGE     IP             NODE                     NOMINATED NODE  READINESS GATES
kube-system  pod/coredns-5c98db65d4-xdq6d          1/1    Running  0          8h    172.17.0.3  minikube  <none>          <none>
kube-system  pod/coredns-66bff467f8-wzjx7                        1/1    Running  0          4m59s  172.17.0.2     minikube-v1.18.9        <none>          <none>
kube-system  pod/etcd-minikube                     1/1    Running  0          8h   10.0.2.15    minikube   <none>          <none>
kube-system  pod/etcd-minikube-v1.18.9                            1/1    Running  0          5m7s   192.168.49.2   minikube-v1.18.9        <none>          <none>
kube-system  pod/kube-addon-manager-minikube        1/1    Running  0          8h    10.0.2.15    minikube  <none>          <none>
kube-system  pod/kube-apiserver-minikube-v1.18.9                  1/1    Running  0          5m7s   192.168.49.2   minikube-v1.18.9        <none>          <none>
kube-system  pod/kube-apiserver-minikube           1/1    Running  0          8h   10.0.2.15    minikube  <none>          <none>
kube-system  pod/kube-controller-manager-minikube-v1.18.9        1/1    Running  0          5m7s   192.168.49.2   minikube-v1.18.9        <none>          <none>
kube-system  pod/kube-controller-manager-minikube   1/1    Running  0          8h   10.0.2.15    minikube  <none>          <none>
kube-system  pod/kube-proxy-zdqc9                                1/1    Running  0          4m59s  192.168.49.2   minikube-v1.18.9        <none>          <none>
kube-system  pod/kube-proxy-j5f2h                  1/1    Running  0          8h    10.0.2.15    minikube  <none>          <none>
kube-system  pod/kube-scheduler-minikube-v1.18.9                  1/1    Running  0          5m7s   192.168.49.2   minikube-v1.18.9        <none>          <none>
kube-system  pod/kube-scheduler-minikube           1/1    Running  0          8h   10.0.2.15    minikube  <none>          <none>
kube-system  pod/storage-provisioner                             1/1    Running  1         5m13s  192.168.49.2   minikube-v1.18.9        <none>          <none>
kube-system  pod/storage-provisioner               1/1    Running  0         8h    10.0.2.15    minikube  <none>          <none>


NAMESPACE    NAME                TYPE        CLUSTER-IP  EXTERNAL-IP  PORT(S)                  AGE   SELECTOR
NAMESPACE    NAME                TYPE        CLUSTER-IP  EXTERNAL-IP  PORT(S)                  AGE     SELECTOR
default      service/kubernetes  ClusterIP  10.96.0.1    <none>        443/TCP                  8h    <none>
default      service/kubernetes  ClusterIP  10.96.0.1    <none>        443/TCP                  5m15s  <none>
kube-system  service/kube-dns    ClusterIP  10.96.0.10  <none>        53/UDP,53/TCP,9153/TCP  8h    k8s-app=kube-dns
kube-system  service/kube-dns    ClusterIP  10.96.0.10  <none>        53/UDP,53/TCP,9153/TCP  5m14s  k8s-app=kube-dns


NAMESPACE    NAME                        DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR                 AGE   CONTAINERS  IMAGES                          SELECTOR
NAMESPACE    NAME                        DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR           AGE     CONTAINERS  IMAGES                          SELECTOR
kube-system  daemonset.apps/kube-proxy  1        1        1      1            1          beta.kubernetes.io/os=linux  8h    kube-proxy  k8s.gcr.io/kube-proxy:v1.15.0   k8s-app=kube-proxy
kube-system  daemonset.apps/kube-proxy  1        1        1      1            1          kubernetes.io/os=linux  5m14s  kube-proxy  k8s.gcr.io/kube-proxy:v1.18.9   k8s-app=kube-proxy


NAMESPACE    NAME                      READY  UP-TO-DATE  AVAILABLE  AGE   CONTAINERS  IMAGES                    SELECTOR
NAMESPACE    NAME                      READY  UP-TO-DATE  AVAILABLE  AGE     CONTAINERS  IMAGES                    SELECTOR
kube-system  deployment.apps/coredns  2/2     2           2           8h    coredns      k8s.gcr.io/coredns:1.3.1   k8s-app=kube-dns
kube-system  deployment.apps/coredns  1/1     1           1           5m14s  coredns      k8s.gcr.io/coredns:1.6.7   k8s-app=kube-dns


NAMESPACE    NAME                                DESIRED  CURRENT  READY  AGE   CONTAINERS  IMAGES                    SELECTOR
NAMESPACE    NAME                                DESIRED  CURRENT  READY  AGE     CONTAINERS  IMAGES                    SELECTOR
kube-system  replicaset.apps/coredns-5c98db65d4   2         2         2       8h    coredns      k8s.gcr.io/coredns:1.3.1   k8s-app=kube-dns,pod-template-hash=5c98db65d4
kube-system  replicaset.apps/coredns-66bff467f8   1         1         1       4m59s  coredns      k8s.gcr.io/coredns:1.6.7   k8s-app=kube-dns,pod-template-hash=66bff467f8
</source>
</source>


= Usage =
= Usage (VirtualBox) =
<source lang=bash>
<source lang=bash>
time minikube start                              #start a cluster by running
time minikube start --kubernetes-version v1.15.0 --profile minikube-v1.15.0
# Start another local clusters named: --profile can be abbreviated with -p, default is 'minikube'
time minikube start --kubernetes-version v1.13.7 --profile k8s-v1.13.7-aws #AWS versions 1.13.7,1.12.6
time minikube start --kubernetes-version v1.15.0 --profile k8s-v1.15.0
</source>
 
 
Multi-profiles
<source>
$ minikube profile list
|-----------------|------------|----------------|-----------|--------------------|
|    Profile    | VM Driver  |    NodeIP    | Node Port | Kubernetes Version |
|-----------------|------------|----------------|-----------|--------------------|
| k8s-v1.13.7-aws | virtualbox | 192.168.99.105 |      8443 | v1.15.0           |
| k8s-v1.15.0    | virtualbox | 192.168.99.104 |      8443 | v1.15.0            |
|-----------------|------------|----------------|-----------|--------------------|
</source>
</source>


Line 227: Line 253:
🎉  Opening http://127.0.0.1:44835/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ in your default browser...
🎉  Opening http://127.0.0.1:44835/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ in your default browser...


#Use kubectl to interact with the local cluster
# Use kubectl to interact with the local cluster [[Kubernetes/Tools#create_pod]]
kubectl run --generator=run-pod/v1        hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #correct
kubectl run --generator=run-pod/v1        hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #correct
kubectl run --generator=deployment/apps.v1 hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #deprecated
kubectl run --generator=deployment/apps.v1 hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #deprecated
kubectl run      ^^default^^              hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #deprecated
kubectl run      ^^default^^              hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #deprecated


#Expose a service as a NodePort
# Expose a service as a NodePort
kubectl expose pod        hello-minikube --type=NodePort
kubectl expose pod        hello-minikube --type=NodePort
kubectl expose deployment hello-minikube --type=NodePort #when pod was created via deployment deprecated option
kubectl expose deployment hello-minikube --type=NodePort #when pod was created via deployment deprecated option


#Determine the NodePort of the service
# Determine the NodePort of the service
kubectl get service <hello-minikube> --output='jsonpath="{.spec.ports[0].nodePort}"'
kubectl get service <hello-minikube> --output='jsonpath="{.spec.ports[0].nodePort}"'


#curl this endpoint
# curl the endpoint
minikube ssh
minikube ssh
curl -s -H 'Cache-Control: no-cache' http://<svc-cluster-ip>:8080 #note 'watch won't work as curl or service proxy  
curl -s -H 'Cache-Control: no-cache' http://<svc-cluster-ip>:8080 #note 'watch won't work as curl or service proxy  
Line 248: Line 274:
minikube stop  #stop your local cluster
minikube stop  #stop your local cluster
minikube delete #delete your local cluster
minikube delete #delete your local cluster
</source>
== Profiles ==
;List profiles
<source>
$> minikube profile list # minikube v1.9.2
|-------------------|-----------|---------|------------|------|----------|---------|
|          Profile | VM Driver | Runtime |    IP    | Port | Version  | Status  |
|-------------------|-----------|---------|------------|------|----------|---------|
| minikube-v1.15.11 | docker    | docker  | 172.17.0.2 | 8443 | v1.15.11 | Stopped |
|-------------------|-----------|---------|------------|------|----------|---------|
</source>
</source>




Working with profiles
;Set and switch in between profiles
Setting the <code>minikube</code> profile does automatically change Kubernetes <code>kubectl config set-context</code> context.
<source lang=bash>
<source lang=bash>
$ ls -1 ~/.minikube/profiles/ #list all profiles
minikube profile minikube-v1.15.11    # set the profile 'minikube-v1.15.11'
aws-v1.13.7-cluster
✅  minikube profile was successfully set to minikube-v1.15.11
minikube             #default profile


#Minikube show current profile
# Configure the persistent default profile
$ minikube profile
minikube config set profile minikube-v1.15.11
aws-v1.13.7-cluster
minikube config view
$ cat ~/.minikube/config/config.json | jq -r .profile #alternative option
- profile: minikube-v1.15.11


#Kubectl  show current context
# Manually switch kubectl context - not needed when using 'minikube profile'
$ kubectl config current-context
kubectl config set-context minikube
$ kubectl config get-contexts  #get-clusters
Context "minikube" modified.
CURRENT  NAME                  CLUSTER              AUTHINFO              NAMESPACE
*        aws-v1.13.7-cluster  aws-v1.13.7-cluster  aws-v1.13.7-cluster 
          minikube              minikube              minikube
</source>
</source>




Switching the Minikube profile does automatically switch the k8s context, and vice versa (v1.2.0 - 2019-06-24)
;Profile local config files
<source lang=bash>
<source lang=bash>
#Switch minikube profile
$> ls -1 ~/.minikube/profiles/ # list all profiles
$ minikube profile minikube #switch to profile: minikube
minikube-v1.15.11
✅  minikube profile was successfully set to minikube
minikube
 
# Minikube show current profile
$> minikube profile
minikube-v1.15.11
$> cat ~/.minikube/config/config.json | jq -r .profile #alternative option


#Switch kubectl context
# Kubectl  show current context
$ kubectl config set-context minikube
$ kubectl config current-context
Context "minikube" modified.
$ kubectl config get-contexts  #get-clusters
CURRENT  NAME                  CLUSTER            AUTHINFO          NAMESPACE
*        minikube-v1.15.11    minikube-v1.15.11  minikube-v1.15.11
          minikube              minikube            minikube
</source>
</source>


Clean up if needed
 
;Clean up if needed
<source lang=bash>
<source lang=bash>
minikube delete -p <profile>
minikube delete -p <profile>
Line 344: Line 387:
} </source>
} </source>
|}
|}
= Setting default configuration =
It's useful to set default configuration, eg. you always work with AWS EKS, so it comes at certain versioning we could set as a default if not explicitly specified.
List configurable options
<source lang=bash>
$ minikube config
config modifies minikube config files using subcommands like "minikube config set driver kvm"
Configurable fields:
* driver              * log_dir                    * ShowDriverDeprecationNotification
* vm-driver          * kubernetes-version          * ShowBootstrapperDeprecationNotification
* container-runtime  * iso-url                    * insecure-registry
* feature-gates      * WantUpdateNotification      * hyperv-virtual-switch
* v                  * ReminderWaitPeriodInHours  * disable-driver-mounts
* cpus                * WantReportError            * cache
* disk-size          * WantReportErrorPrompt      * embed-certs
* host-only-cidr      * WantKubectlDownloadMsg      * native-ssh
* memory              * WantNoneDriverWarning
                      * profile
                      * bootstrapper
Available Commands:
  get        Gets the value of PROPERTY_NAME from the minikube config file
  set        Sets an individual value in a minikube config file
  unset      unsets an individual value in a minikube config file
  view        Display values currently set in the minikube config file
Usage:
  minikube config SUBCOMMAND [flags] [options]
</source>
; Set default version of Kubernetes
<source lang=bash>
$ minikube config set kubernetes-version v1.14.9
$ minikube config view
- kubernetes-version: v1.14.9
</source>


= Loadbalancer - <code>minikube tunnel</code> =
= Loadbalancer - <code>minikube tunnel</code> =
Line 423: Line 505:


= [https://gist.github.com/alexellis/eec21a96906726d08a071d58aee66ab9#file-kvm_minikube-md multi node cluster with kvm] =
= [https://gist.github.com/alexellis/eec21a96906726d08a071d58aee66ab9#file-kvm_minikube-md multi node cluster with kvm] =
This is now supported natively using <code>--nodes</code> flag:
<source lang=bash>
minikube start --profile minikube-v1.15.11 --nodes=2  --kubernetes-version v1.15.11
minikube ssh  --profile minikube-v1.15.11 --node=m02
minikube      --profile minikube-v1.15.11  node add        # add new node
minikube      --profile minikube-v1.15.11  node delete m03  # delete
$ kubectl get nodes
NAME                    STATUS    ROLES    AGE    VERSION
minikube-v1.15.11      Ready      master  3h25m  v1.15.11
minikube-v1.15.11-m02  NotReady  <none>  3h24m  v1.15.11
minikube-v1.15.11-m03  NotReady  <none>  16m    v1.15.11 # <- add / remove
</source>
Nodes names have following naming convention
*node-1-name: <code>minikube-v1.14.9</code>
*node-2-name: <code>minikube-v1.14.9-m02</code>
= ingress and local DNS ([https://github.com/kubernetes/minikube/tree/master/deploy/addons/ingress-dns ingress-dns]) =
TODO:
* [https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/hostport.md Setting up ExternalDNS for Headless Services]
* [https://github.com/superbrothers/minikube-ingress-dns minikube-ingress-dns] [https://gitlab.com/cryptexlabs/public/development/minikube-ingress-dns gitlab]
* [https://kubernetes.io/docs/tasks/access-application-cluster/ingress-minikube/ ingress-minikube] Kubernetes docs
Add required addons
<source lang=bash>
minikube addons enable ingress --profile minikube-v1.14.9
🌟  The 'ingress' addon is enabled
minikube addons enable ingress-dns --profile minikube-v1.14.9
🌟  The 'ingress-dns' addon is enabled
</source>
;Add minikube ip as a dns server
Ubuntu 18.04 LTS, configure <code>resolvconf</code>
<source lang=bash>
sudo apt install resolvconf
sudo -p /etc/resolvconf/resolv.conf.d      # possibly created by installation package
sudo vi /etc/resolvconf/resolv.conf.d/base
minikube ip --profile minikube-v1.14.9
192.168.99.126
# Update resolverconf
sudo vi /etc/resolvconf/resolv.conf.d/base
search test
nameserver 192.168.99.126
timeout 5
# Apply changes
sudo resolvconf -u
systemctl disable --now resolvconf.service
# Test, Add the test ingress
kubectl apply -f https://raw.githubusercontent.com/kubernetes/minikube/master/deploy/addons/ingress-dns/example/example.yaml
# Validate DNS queries are returning A records
nslookup hello-john.test $(minikube ip --profile minikube-v1.14.9)
Server: 192.168.99.126
Address: 192.168.99.126#53
Non-authoritative answer:
Name: hello-john.test
Address: 192.168.99.126
Name: hello-john.test
Address: 192.168.99.126 # <- this is ingress IP address
$ kubectl -n kube-system get ingresses.networking.k8s.io example-ingress
NAME              HOSTS                            ADDRESS          PORTS  AGE
example-ingress  hello-john.test,hello-jane.test  192.168.99.126  80      102s
</source>
{{Note|<code>.local</code> is a reserved TLD, do not use as this is a reserved TLD for mDNS and bind9 DNS servers}}
{{Note|<code>.localhost</code> domains will not correctly resolve on chromium since it is used as a loopback address. Instead use <code>.test</code>, <code>.example</code>, or <code>.invalid</code>}}
= References =
= References =
*[https://github.com/kubernetes/minikube minikube] Github
*[https://github.com/kubernetes/minikube minikube] Github
*[https://github.com/kubernetes/minikube/blob/master/CHANGELOG.md CHANGELOG.md] Github
*[https://github.com/kubernetes/minikube/blob/master/CHANGELOG.md CHANGELOG.md] Github
*[https://github.com/kubernetes/minikube/blob/master/docs/networking.md Networking.md] Github
*[https://github.com/kubernetes/minikube/blob/master/docs/networking.md Networking.md] Github

Latest revision as of 11:13, 8 May 2021

Minikube features

  • LoadBalancer - using minikube tunnel
  • Multi-cluster minikube start --profile cluster-1
  • Multi-node cluster minikube start --profile cluster-2 --nodes 2
  • NodePorts - using minikube service
  • Persistent Volumes
  • Ingress
  • RBAC
  • Dashboard - minikube dashboard
  • Container runtimes - start --container-runtime
  • Configure apiserver and kubelet options via command-line flags

Developer friendly features:

  • Filesystem mounts
  • Addons - a marketplace for developers to share configurations for running services on minikube
  • NVIDIA GPU support - for machine learning

Release notes CHANGELOG.mb

Documentation minikube.sigs.k8s.io

Addons

List addons

$ minikube addons list --profile minikube-v1.14.9
|-----------------------------|----------------------|--------------|
|         ADDON NAME          |        PROFILE       |    STATUS    |
|-----------------------------|----------------------|--------------|
| dashboard                   | minikube-v1.14.9     | disabled     |
| default-storageclass        | minikube-v1.14.9     | enabled ✅   |
| efk                         | minikube-v1.14.9     | disabled     |
| freshpod                    | minikube-v1.14.9     | disabled     |
| gvisor                      | minikube-v1.14.9     | disabled     |
| helm-tiller                 | minikube-v1.14.9     | disabled     |
| ingress                     | minikube-v1.14.9     | enabled ✅   |
| ingress-dns                 | minikube-v1.14.9     | enabled ✅   |
| istio                       | minikube-v1.14.9     | disabled     |
| istio-provisioner           | minikube-v1.14.9     | disabled     |
| logviewer                   | minikube-v1.14.9     | disabled     |
| metrics-server              | minikube-v1.14.9     | disabled     |
| nvidia-driver-installer     | minikube-v1.14.9     | disabled     |
| nvidia-gpu-device-plugin    | minikube-v1.14.9     | disabled     |
| registry                    | minikube-v1.14.9     | disabled     |
| registry-aliases            | minikube-v1.14.9     | disabled     |
| registry-creds              | minikube-v1.14.9     | disabled     |
| storage-provisioner         | minikube-v1.14.9     | enabled ✅   |
| storage-provisioner-gluster | minikube-v1.14.9     | disabled     |
|-----------------------------|----------------------|--------------|


Usage

minikube -p efk start  --memory 8192
minikube -p efk addons enable efk     # pulling images and setting up can take ~10 min
minikube -p efk addons open efk       # opens Kibana

minikube addons enable heapster       # get some insight into CPU

# Show images
minikube addons images metrics-server
  - metrics-server has following images:
|---------------|--------------------------------------------------------------------------------------------------------------|------------------|
|  IMAGE NAME   |                                                DEFAULT IMAGE                                                 | DEFAULT REGISTRY |
|---------------|--------------------------------------------------------------------------------------------------------------|------------------|
| MetricsServer | metrics-server/metrics-server:v0.4.2@sha256:dbc33d7d35d2a9cc5ab402005aa7a0d13be6192f3550c7d42cba8d2d5e3a5d62 | k8s.gcr.io       |
|---------------|--------------------------------------------------------------------------------------------------------------|------------------|

Install or upgrade on Linux

# Latest from GitHub
VERSION=$(curl --silent "https://api.github.com/repos/kubernetes/minikube/releases/latest" | jq -r .tag_name); echo $VERSION
curl -LO https://github.com/kubernetes/minikube/releases/download/$VERSION/minikube-linux-amd64 \
      && sudo install minikube-linux-amd64 /usr/local/bin/minikube

# Latest from GoogleAPI
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 \
      && sudo install minikube-linux-amd64 /usr/local/bin/minikube

# Specific version from GitHub
curl -LO https://github.com/kubernetes/minikube/releases/download/v1.7.3/minikube-linux-amd64 \
      && sudo install minikube-linux-amd64 /usr/local/bin/minikube

$> $ minikube version 
minikube version: v1.17.0

Start on Windows

Install <syntaxhighlightjs lang="powershell"> choco install minikube kubernetes-cli </syntaxhighlightjs>


HyperV <syntaxhighlightjs lang="powershell">

  1. HyperV set config (option 1)

minikube config set vm-driver hyperv minikube config set hyperv-virtual-switch "Default Switch" minikube start

  1. HyperV single command (option 2)

minikube start --vm-driver hyperv --hyperv-virtual-switch k8s-node-1 --profile k8s-node-1

  1. --hyperv-virtual-switch :- need to exist

</syntaxhighlightjs>


Virtualbox <syntaxhighlightjs lang="powershell"> minikube start --profile k8s-node-1

  • minikube v1.2.0 on windows (amd64)
  • Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
  • Configuring environment for Kubernetes v1.15.0 on Docker 18.09.6
  • Downloading kubelet v1.15.0
  • Downloading kubeadm v1.15.0
  • Pulling images ...
  • Launching Kubernetes ...
  • Verifying: apiserver proxy etcd scheduler controller dns
  • Done! kubectl is now configured to use "k8s-node-1"

PS C:\Users\Sylwia> kubectl.exe get nodes -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME minikube Ready master 23m v1.15.0 10.0.2.15 <none> Buildroot 2018.05.3 4.15.0 docker://18.9.6 </syntaxhighlightjs>

Start - Linux (Ubuntu)

Start sequence

VERSION=v1.18.9
time minikube start --kubernetes-version $VERSION --profile minikube-$VERSION
😄  [minikube-v1.18.9] minikube v1.19.0 on Ubuntu 20.04
✨  Automatically selected the docker driver. Other choices: virtualbox, ssh, none
👍  Starting control plane node minikube-v1.18.9 in cluster minikube-v1.18.9
🚜  Pulling base image ...
    > gcr.io/k8s-minikube/kicbase...: 357.67 MiB / 357.67 MiB  100.00% 4.70 MiB
🔥  Creating docker container (CPUs=2, Memory=3900MB) ...
🐳  Preparing Kubernetes v1.18.9 on Docker 20.10.5 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube-v1.18.9" cluster and "default" namespace by default

real	2m4.966s
user	0m6.658s
sys	0m3.768s


$> minikube profile list
|-----------------------------|-----------|---------|--------------|------|---------|---------|-------|
|           Profile           | VM Driver | Runtime |      IP      | Port | Version | Status  | Nodes |
|-----------------------------|-----------|---------|--------------|------|---------|---------|-------|
| minikube-v1.18.9            | docker    | docker  | 192.168.58.2 | 8443 | v1.18.9 | Running |     1 |
|-----------------------------|-----------|---------|--------------|------|---------|---------|-------|

$> minikube status -p minikube-v1.18.9
minikube-v1.18.9
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

$> kubectl get nodes -o wide
NAME               STATUS   ROLES    AGE     VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
minikube-v1.18.9   Ready    master   2m34s   v1.18.9   192.168.58.2   <none>        Ubuntu 20.04.1 LTS   5.8.0-50-generic   docker://20.10.5


$ kubectl get all -o wide --all-namespaces
kubectl get all -o wide --all-namespaces
NAMESPACE     NAME                                                 READY   STATUS    RESTARTS   AGE     IP             NODE                     NOMINATED NODE   READINESS GATES
kube-system   pod/coredns-66bff467f8-wzjx7                         1/1     Running   0          4m59s   172.17.0.2     minikube-v1.18.9         <none>           <none>
kube-system   pod/etcd-minikube-v1.18.9                            1/1     Running   0          5m7s    192.168.49.2   minikube-v1.18.9         <none>           <none>
kube-system   pod/kube-apiserver-minikube-v1.18.9                  1/1     Running   0          5m7s    192.168.49.2   minikube-v1.18.9         <none>           <none>
kube-system   pod/kube-controller-manager-minikube-v1.18.9         1/1     Running   0          5m7s    192.168.49.2   minikube-v1.18.9         <none>           <none>
kube-system   pod/kube-proxy-zdqc9                                 1/1     Running   0          4m59s   192.168.49.2   minikube-v1.18.9         <none>           <none>
kube-system   pod/kube-scheduler-minikube-v1.18.9                  1/1     Running   0          5m7s    192.168.49.2   minikube-v1.18.9         <none>           <none>
kube-system   pod/storage-provisioner                              1/1     Running   1          5m13s   192.168.49.2   minikube-v1.18.9         <none>           <none>

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE     SELECTOR
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  5m15s   <none>
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   5m14s   k8s-app=kube-dns

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE     CONTAINERS   IMAGES                          SELECTOR
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux   5m14s   kube-proxy   k8s.gcr.io/kube-proxy:v1.18.9   k8s-app=kube-proxy

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                     SELECTOR
kube-system   deployment.apps/coredns   1/1     1            1           5m14s   coredns      k8s.gcr.io/coredns:1.6.7   k8s-app=kube-dns

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE     CONTAINERS   IMAGES                     SELECTOR
kube-system   replicaset.apps/coredns-66bff467f8   1         1         1       4m59s   coredns      k8s.gcr.io/coredns:1.6.7   k8s-app=kube-dns,pod-template-hash=66bff467f8

Usage (VirtualBox)

time minikube start --kubernetes-version v1.15.0 --profile minikube-v1.15.0


Get port exposed to host via Virtualbox used by minikube to interact with a cluster

$ sudo ss -ltpn
State  Recv-Q  Send-Q    Local Address:Port    Peer Address:Port                                               
LISTEN 0       128       127.0.0.53%lo:53           0.0.0.0:*      users:(("systemd-resolve",pid=1009,fd=13))  
LISTEN 0       10            127.0.0.1:45111 #<--   0.0.0.0:*      users:(("VBoxHeadless",pid=26918,fd=21))    
LISTEN 0       5             127.0.0.1:631          0.0.0.0:*      users:(("cupsd",pid=27629,fd=7))            
LISTEN 0       5                 [::1]:631             [::]:*      users:(("cupsd",pid=27629,fd=6))

$ vboxmanage showvminfo k8s-v1.15.0 | 'grep port ='                            vv
NIC 1 Rule(0):   name = ssh, protocol = tcp, host ip = 127.0.0.1, host port = 45111, guest ip = , guest port = 22
                                                                               ^^


Access to minikube VM using ssh
minikube ssh cat /etc/*rel*
$ minikube ssh 
                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$
cat /etc/*rel*
NAME=Buildroot
VERSION=2018.05.3
ID=buildroot
VERSION_ID=2018.05.3
PRETTY_NAME="Buildroot 2018.05.3"
NAME=Buildroot
VERSION=2018.05.3
ID=buildroot
VERSION_ID=2018.05.3
PRETTY_NAME="Buildroot 2018.05.3"


#access Kubernetes Dashboard within Minikube
minikube dashboard
🤔  Verifying dashboard health ...
🚀  Launching proxy ...
🤔  Verifying proxy health ...
🎉  Opening http://127.0.0.1:44835/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ in your default browser...

# Use kubectl to interact with the local cluster [[Kubernetes/Tools#create_pod]]
kubectl run --generator=run-pod/v1         hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #correct
kubectl run --generator=deployment/apps.v1 hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #deprecated
kubectl run       ^^default^^              hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #deprecated

# Expose a service as a NodePort
kubectl expose pod        hello-minikube --type=NodePort
kubectl expose deployment hello-minikube --type=NodePort #when pod was created via deployment deprecated option

# Determine the NodePort of the service
kubectl get service <hello-minikube> --output='jsonpath="{.spec.ports[0].nodePort}"'

# curl the endpoint
minikube ssh
curl -s -H 'Cache-Control: no-cache' http://<svc-cluster-ip>:8080 #note 'watch won't work as curl or service proxy 

#minikube makes it easy to open this exposed endpoint in your browser
minikube service hello-minikube

minikube stop   #stop your local cluster
minikube delete #delete your local cluster

Profiles

List profiles
$> minikube profile list # minikube v1.9.2
|-------------------|-----------|---------|------------|------|----------|---------|
|           Profile | VM Driver | Runtime |     IP     | Port | Version  | Status  |
|-------------------|-----------|---------|------------|------|----------|---------|
| minikube-v1.15.11 | docker    | docker  | 172.17.0.2 | 8443 | v1.15.11 | Stopped |
|-------------------|-----------|---------|------------|------|----------|---------|


Set and switch in between profiles

Setting the minikube profile does automatically change Kubernetes kubectl config set-context context.

minikube profile minikube-v1.15.11    # set the profile 'minikube-v1.15.11'
✅  minikube profile was successfully set to minikube-v1.15.11

# Configure the persistent default profile
minikube config set profile minikube-v1.15.11
minikube config view 
- profile: minikube-v1.15.11

# Manually switch kubectl context - not needed when using 'minikube profile'
kubectl config set-context minikube
Context "minikube" modified.


Profile local config files
$> ls -1 ~/.minikube/profiles/ # list all profiles
minikube-v1.15.11
minikube

# Minikube show current profile
$> minikube profile
minikube-v1.15.11
$> cat ~/.minikube/config/config.json | jq -r .profile #alternative option

# Kubectl  show current context
$ kubectl config current-context
$ kubectl config get-contexts   #get-clusters
CURRENT   NAME                  CLUSTER             AUTHINFO           NAMESPACE
*         minikube-v1.15.11     minikube-v1.15.11   minikube-v1.15.11
          minikube              minikube            minikube


Clean up if needed
minikube delete -p <profile>
kubectl config delete-context <context-name>
kubectl config delete-context <delete-cluster>


Example of ~/.minikube/profiles/<minikube-profile>/config.json
jq .MachineConfig config.json jq .KubernetesConfig config.json
{
  "KubernetesVersion": "v1.13.7",
  "NodeIP": "192.168.99.102",
  "NodePort": 8443,
  "NodeName": "minikube",
  "APIServerName": "minikubeCA",
  "APIServerNames": null,
  "APIServerIPs": null,
  "DNSDomain": "cluster.local",
  "ContainerRuntime": "docker",
  "CRISocket": "",
  "NetworkPlugin": "",
  "FeatureGates": "",
  "ServiceCIDR": "10.96.0.0/12",
  "ImageRepository": "",
  "ExtraOptions": null,
  "ShouldLoadCachedImages": true,
  "EnableDefaultCNI": false
}
{
  "KeepContext": false,
  "MinikubeISO": "https://storage.googleapis.com/minikube/iso/minikube-v1.2.0.iso",
  "Memory": 2048,
  "CPUs": 2,
  "DiskSize": 20000,
  "VMDriver": "virtualbox",
  "ContainerRuntime": "docker",
  "HyperkitVpnKitSock": "",
  "HyperkitVSockPorts": [],
  "XhyveDiskDriver": "ahci-hd",
  "DockerEnv": null,
  "InsecureRegistry": null,
  "RegistryMirror": null,
  "HostOnlyCIDR": "192.168.99.1/24",
  "HypervVirtualSwitch": "",
  "KvmNetwork": "default",
  "DockerOpt": null,
  "DisableDriverMounts": false,
  "NFSShare": [],
  "NFSSharesRoot": "/nfsshares",
  "UUID": "",
  "GPU": false,
  "Hidden": false,
  "NoVTXCheck": false
}

Setting default configuration

It's useful to set default configuration, eg. you always work with AWS EKS, so it comes at certain versioning we could set as a default if not explicitly specified.


List configurable options

$ minikube config 
config modifies minikube config files using subcommands like "minikube config set driver kvm"
Configurable fields: 

 * driver              * log_dir                     * ShowDriverDeprecationNotification
 * vm-driver           * kubernetes-version          * ShowBootstrapperDeprecationNotification
 * container-runtime   * iso-url                     * insecure-registry
 * feature-gates       * WantUpdateNotification      * hyperv-virtual-switch
 * v                   * ReminderWaitPeriodInHours   * disable-driver-mounts
 * cpus                * WantReportError             * cache
 * disk-size           * WantReportErrorPrompt       * embed-certs
 * host-only-cidr      * WantKubectlDownloadMsg      * native-ssh
 * memory              * WantNoneDriverWarning
                       * profile
                       * bootstrapper
Available Commands:
  get         Gets the value of PROPERTY_NAME from the minikube config file
  set         Sets an individual value in a minikube config file
  unset       unsets an individual value in a minikube config file
  view        Display values currently set in the minikube config file

Usage:
  minikube config SUBCOMMAND [flags] [options]


Set default version of Kubernetes
$ minikube config set kubernetes-version v1.14.9
$ minikube config view 
- kubernetes-version: v1.14.9

Loadbalancer - minikube tunnel

A LoadBalancer service is the standard way to expose a service to the internet. With this method, each service gets it’s own IP address. In minikube services of type LoadBalancer can be exposed via the minikube tunnel command. It will run until Ctrl-C is hit.


minikube tunnel runs as a separate daemon, creating a network route on the host to the service CIDR of the cluster using the cluster’s IP address as a gateway. The tunnel command exposes the external IP directly to any program running on the host operating system. The low-level commands used to manage routes are /sbin/ip,/sbin/route that optionally can be added to suedors file to make a life a little easier.

$ minikube profile list
|-----------------|------------|----------------|-----------|--------------------|
|     Profile     | VM Driver  |     NodeIP     | Node Port | Kubernetes Version |
|-----------------|------------|----------------|-----------|--------------------|
| efk             | virtualbox | 192.168.99.113 |      8443 | v1.16.0            |
|-----------------|------------|----------------|-----------|--------------------|

# Before
$ kubectl -n efk get svc # note <pending> EXTERNAL-IP
NAMESPACE NAME                  TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)       
efk       service/elasticsearch LoadBalancer   10.104.149.221  <pending>       9200:30826/TCP
efk       service/kibana        LoadBalancer   10.97.198.117   <pending>       5601:32540/TCP

$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         vodafone.connec 0.0.0.0         UG    600    0        0 wlp4s0
link-local      0.0.0.0         255.255.0.0     U     1000   0        0 wlp4s0
192.168.1.0     0.0.0.0         255.255.255.0   U     600    0        0 wlp4s0
192.168.99.0    0.0.0.0         255.255.255.0   U     0      0        0 vboxnet2

# After - EXTERNAL-IP have been assigned from a pool of 10.96.0.0/12
NAMESPACE NAME                  TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)       
efk       service/elasticsearch LoadBalancer   10.104.149.221  10.104.149.221  9200:30826/TCP
efk       service/kibana        LoadBalancer   10.97.198.117   10.97.198.117   5601:32540/TCP

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         vodafone.connec 0.0.0.0         UG    600    0        0 wlp4s0
10.96.0.0       192.168.99.113  255.240.0.0     UG    0      0        0 vboxnet2 # <- new route to minikube NodeIP created 
link-local      0.0.0.0         255.255.0.0     U     1000   0        0 wlp4s0
192.168.1.0     0.0.0.0         255.255.255.0   U     600    0        0 wlp4s0
192.168.99.0    0.0.0.0         255.255.255.0   U     0      0        0 vboxnet2

$ minikube --profile efk tunnel
Status:
        machine: efk
        pid: 8721
        route: 10.96.0.0/12 -> 192.168.99.113 # route status
        minikube: Running
        services: [elasticsearch, kibana]     # exposed services
    errors: 
                minikube: no errors
                router: no errors
                loadbalancer emulator: no errors
# Verify
$ nc -zv 10.97.198.117 5601 # kibana
Connection to 10.97.198.117 5601 port [tcp/*] succeeded!
$ nc -zv 10.104.149.221 9200 # elasticsearch
Connection to 10.104.149.221 9200 port [tcp/*] succeeded!
$ curl http://10.97.198.117:5601/status


Clean up based on info ~/.minikube/tunnels.json when shuts unexpected

minikube tunnel --cleanup


Avoid to typing password for minikube tunnel, by allowing a group or user to execute ip, route commands without asking for password. Example for Ubuntu 18.04 can be seen below.

$ sudo visudo
...
# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL
%admin ALL=(ALL) NOPASSWD: /sbin/ip,/sbin/route  # <- add this line in this order
...

multi node cluster with kvm

This is now supported natively using --nodes flag:

minikube start --profile minikube-v1.15.11 --nodes=2  --kubernetes-version v1.15.11
minikube ssh   --profile minikube-v1.15.11 --node=m02
minikube       --profile minikube-v1.15.11   node add         # add new node
minikube       --profile minikube-v1.15.11   node delete m03  # delete

$ kubectl get nodes
NAME                    STATUS     ROLES    AGE     VERSION
minikube-v1.15.11       Ready      master   3h25m   v1.15.11
minikube-v1.15.11-m02   NotReady   <none>   3h24m   v1.15.11
minikube-v1.15.11-m03   NotReady   <none>   16m     v1.15.11 # <- add / remove


Nodes names have following naming convention

  • node-1-name: minikube-v1.14.9
  • node-2-name: minikube-v1.14.9-m02

ingress and local DNS (ingress-dns)

TODO:

Add required addons

minikube addons enable ingress --profile minikube-v1.14.9
🌟  The 'ingress' addon is enabled
minikube addons enable ingress-dns --profile minikube-v1.14.9
🌟  The 'ingress-dns' addon is enabled
Add minikube ip as a dns server

Ubuntu 18.04 LTS, configure resolvconf

sudo apt install resolvconf
sudo -p /etc/resolvconf/resolv.conf.d       # possibly created by installation package
sudo vi /etc/resolvconf/resolv.conf.d/base
minikube ip --profile minikube-v1.14.9
192.168.99.126

# Update resolverconf
sudo vi /etc/resolvconf/resolv.conf.d/base
search test
nameserver 192.168.99.126
timeout 5

# Apply changes
sudo resolvconf -u
systemctl disable --now resolvconf.service 

# Test, Add the test ingress
kubectl apply -f https://raw.githubusercontent.com/kubernetes/minikube/master/deploy/addons/ingress-dns/example/example.yaml

# Validate DNS queries are returning A records
nslookup hello-john.test $(minikube ip --profile minikube-v1.14.9)
Server:		192.168.99.126
Address:	192.168.99.126#53

Non-authoritative answer:
Name:	hello-john.test
Address: 192.168.99.126
Name:	hello-john.test
Address: 192.168.99.126 # <- this is ingress IP address

$ kubectl -n kube-system get ingresses.networking.k8s.io example-ingress 
NAME              HOSTS                             ADDRESS          PORTS   AGE
example-ingress   hello-john.test,hello-jane.test   192.168.99.126   80      102s


Note: .local is a reserved TLD, do not use as this is a reserved TLD for mDNS and bind9 DNS servers


Note: .localhost domains will not correctly resolve on chromium since it is used as a loopback address. Instead use .test, .example, or .invalid

References