Difference between revisions of "Kubernetes/minikube"

From Ever changing code
Jump to navigation Jump to search
 
(4 intermediate revisions by the same user not shown)
Line 55: Line 55:
minikube -p efk addons open efk      # opens Kibana
minikube -p efk addons open efk      # opens Kibana


minikube addons enable heapster #get some insight into CPU
minikube addons enable heapster       # get some insight into CPU
 
# Show images
minikube addons images metrics-server
  - metrics-server has following images:
|---------------|--------------------------------------------------------------------------------------------------------------|------------------|
|  IMAGE NAME  |                                                DEFAULT IMAGE                                                | DEFAULT REGISTRY |
|---------------|--------------------------------------------------------------------------------------------------------------|------------------|
| MetricsServer | metrics-server/metrics-server:v0.4.2@sha256:dbc33d7d35d2a9cc5ab402005aa7a0d13be6192f3550c7d42cba8d2d5e3a5d62 | k8s.gcr.io      |
|---------------|--------------------------------------------------------------------------------------------------------------|------------------|
 
</source>
</source>


Line 115: Line 125:
</syntaxhighlightjs>
</syntaxhighlightjs>


= Start =
= Start - Linux (Ubuntu)=
Example of start sequence
Start sequence
<source lang=bash>
<source lang=bash>
$> time minikube start --kubernetes-version v1.14.9 --profile minikube-v1.14.9 # minikube v1.9.2
VERSION=v1.18.9
😄  [minikube-v1.14.9] minikube v1.9.2 on Ubuntu 18.04
time minikube start --kubernetes-version $VERSION --profile minikube-$VERSION
✨  Automatically selected the virtualbox driver
😄  [minikube-v1.18.9] minikube v1.19.0 on Ubuntu 20.04
💿  Downloading VM boot image ...
✨  Automatically selected the docker driver. Other choices: virtualbox, ssh, none
    > minikube-v1.9.0.iso.sha256: 65 B / 65 B [--------------] 100.00% ? p/s 0s
👍  Starting control plane node minikube-v1.18.9 in cluster minikube-v1.18.9
    > minikube-v1.9.0.iso: 174.93 MiB / 174.93 MiB [-] 100.00% 4.05 MiB p/s 44s
🚜 Pulling base image ...
👍  Starting control plane node m01 in cluster minikube-v1.14.9
     > gcr.io/k8s-minikube/kicbase...: 357.67 MiB / 357.67 MiB  100.00% 4.70 MiB
💾 Downloading Kubernetes v1.14.9 preload ...
🔥  Creating docker container (CPUs=2, Memory=3900MB) ...
     > preloaded-images-k8s-v2-v1.14.9-docker-overlay2-amd64.tar.lz4: 496.25 MiB
🐳  Preparing Kubernetes v1.18.9 on Docker 20.10.5 ...
🔥  Creating virtualbox VM (CPUs=2, Memory=3900MB, Disk=20000MB) ...
    ▪ Generating certificates and keys ...
🐳  Preparing Kubernetes v1.14.9 on Docker 19.03.8 ...
    ▪ Booting up control plane ...
🌟  Enabling addons: default-storageclass, storage-provisioner
    ▪ Configuring RBAC rules ...
🏄  Done! kubectl is now configured to use "minikube-v1.14.9"
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube-v1.18.9" cluster and "default" namespace by default


real 4m9.678s
real 2m4.966s
user 0m21.710s
user 0m6.658s
sys 0m30.296s
sys 0m3.768s




$ minikube profile list
$> minikube profile list
|----------------------------------|------------|---------|----------------|------|---------|---------|
|-----------------------------|-----------|---------|--------------|------|---------|---------|-------|
|             Profile             | VM Driver | Runtime |       IP       | Port | Version | Status  |
|           Profile           | VM Driver | Runtime |     IP     | Port | Version | Status  | Nodes |
|----------------------------------|------------|---------|----------------|------|---------|---------|
|-----------------------------|-----------|---------|--------------|------|---------|---------|-------|
| minikube-v1.14.9                 | virtualbox | docker  | 192.168.99.123 | 8443 | v1.14.9 | Running |
| minikube-v1.18.9           | docker    | docker  | 192.168.58.2 | 8443 | v1.18.9 | Running |    1 |
|----------------------------------|------------|---------|----------------|------|---------|---------|
|-----------------------------|-----------|---------|--------------|------|---------|---------|-------|
$ minikube status -p minikube-v1.14.9
 
$> minikube status -p minikube-v1.18.9
minikube-v1.18.9
type: Control Plane
host: Running
host: Running
kubelet: Running
kubelet: Running
Line 149: Line 165:
kubeconfig: Configured
kubeconfig: Configured


$ kubectl get nodes -o wide
$> kubectl get nodes -o wide
NAME             STATUS ROLES AGE  VERSION INTERNAL-IP    EXTERNAL-IP OS-IMAGE            KERNEL-VERSION CONTAINER-RUNTIME
NAME               STATUS   ROLES   AGE     VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE            KERNEL-VERSION     CONTAINER-RUNTIME
minikube-v1.14.9 Ready master 6m57s v1.14.9 192.168.99.123 <none>     Buildroot 2019.02.10 4.19.107      docker://19.3.8
minikube-v1.18.9   Ready   master   2m34s  v1.18.9   192.168.58.<none>       Ubuntu 20.04.1 LTS  5.8.0-50-generic  docker://20.10.5
 


$ kubectl get all -o wide --all-namespaces
$ kubectl get all -o wide --all-namespaces
NAME                              STATUS  ROLES    AGE    VERSION  INTERNAL-IP      EXTERNAL-IP  OS-IMAGE              KERNEL-VERSION  CONTAINER-RUNTIME
kubectl get all -o wide --all-namespaces
minikube-v1.14.9                  Ready    master  6m57s  v1.14.9  192.168.99.123  <none>        Buildroot 2019.02.10  4.19.107        docker://19.3.8
NAMESPACE    NAME                                                 READY  STATUS    RESTARTS  AGE    IP             NODE                     NOMINATED NODE  READINESS GATES
pipaw@pp-tp52s ~ $ kubectl get all -o wide --all-namespaces
kube-system  pod/coredns-66bff467f8-wzjx7                        1/1    Running  0          4m59s   172.17.0.2    minikube-v1.18.9         <none>          <none>
NAMESPACE    NAME                                                           READY  STATUS    RESTARTS  AGE    IP               NODE                               NOMINATED NODE  READINESS GATES
kube-system  pod/etcd-minikube-v1.18.9                           1/1    Running  0          5m7s    192.168.49.2   minikube-v1.18.9         <none>          <none>
kube-system  pod/coredns-6dcc67dcbc-48k6v                                  1/1    Running  0          8m16s   172.17.0.2       minikube-v1.14.9                  <none>          <none>
kube-system  pod/kube-apiserver-minikube-v1.18.9                 1/1    Running  0          5m7s    192.168.49.2   minikube-v1.18.9         <none>          <none>
kube-system  pod/coredns-6dcc67dcbc-4ftc4                                  1/1     Running  0          8m16s  172.17.0.3      minikube-v1.14.9                   <none>          <none>
kube-system  pod/kube-controller-manager-minikube-v1.18.9         1/1    Running  0          5m7s    192.168.49.2   minikube-v1.18.9         <none>          <none>
kube-system  pod/etcd-minikube-v1.14.9                                     1/1    Running  0          7m21s  192.168.99.123   minikube-v1.14.9                   <none>          <none>
kube-system  pod/kube-proxy-zdqc9                                1/1    Running  0          4m59s   192.168.49.2   minikube-v1.18.9         <none>          <none>
kube-system  pod/kube-apiserver-minikube-v1.14.9                           1/1    Running  0          7m21s  192.168.99.123   minikube-v1.14.9                   <none>          <none>
kube-system  pod/kube-scheduler-minikube-v1.18.9                 1/1    Running  0          5m7s    192.168.49.2   minikube-v1.18.9         <none>          <none>
kube-system  pod/kube-controller-manager-minikube-v1.14.9                   1/1    Running  0          7m26s  192.168.99.123   minikube-v1.14.9                   <none>          <none>
kube-system  pod/storage-provisioner                             1/1    Running  1          5m13s   192.168.49.2   minikube-v1.18.9         <none>          <none>
kube-system  pod/kube-proxy-kjjrx                                          1/1    Running  0          8m16s   192.168.99.123   minikube-v1.14.9                   <none>          <none>
kube-system  pod/kube-scheduler-minikube-v1.14.9                           1/1    Running  0          7m22s  192.168.99.123   minikube-v1.14.9                   <none>          <none>
kube-system  pod/storage-provisioner                                       1/1    Running  1          8m22s   192.168.99.123   minikube-v1.14.9                   <none>          <none>
 


NAMESPACE    NAME                TYPE        CLUSTER-IP  EXTERNAL-IP  PORT(S)                  AGE    SELECTOR
NAMESPACE    NAME                TYPE        CLUSTER-IP  EXTERNAL-IP  PORT(S)                  AGE    SELECTOR
default      service/kubernetes  ClusterIP  10.96.0.1    <none>        443/TCP                  8m25s   <none>
default      service/kubernetes  ClusterIP  10.96.0.1    <none>        443/TCP                  5m15s   <none>
kube-system  service/kube-dns    ClusterIP  10.96.0.10  <none>        53/UDP,53/TCP,9153/TCP  8m24s   k8s-app=kube-dns
kube-system  service/kube-dns    ClusterIP  10.96.0.10  <none>        53/UDP,53/TCP,9153/TCP  5m14s   k8s-app=kube-dns


NAMESPACE    NAME                        DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR   AGE    CONTAINERS  IMAGES                          SELECTOR
NAMESPACE    NAME                        DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR           AGE    CONTAINERS  IMAGES                          SELECTOR
kube-system  daemonset.apps/kube-proxy  1        1        1      1            1          <none>          8m23s   kube-proxy  k8s.gcr.io/kube-proxy:v1.14.9  k8s-app=kube-proxy
kube-system  daemonset.apps/kube-proxy  1        1        1      1            1          kubernetes.io/os=linux  5m14s   kube-proxy  k8s.gcr.io/kube-proxy:v1.18.9  k8s-app=kube-proxy


NAMESPACE    NAME                      READY  UP-TO-DATE  AVAILABLE  AGE    CONTAINERS  IMAGES                    SELECTOR
NAMESPACE    NAME                      READY  UP-TO-DATE  AVAILABLE  AGE    CONTAINERS  IMAGES                    SELECTOR
kube-system  deployment.apps/coredns  2/2     2           2           8m24s   coredns      k8s.gcr.io/coredns:1.3.1   k8s-app=kube-dns
kube-system  deployment.apps/coredns  1/1     1           1           5m14s   coredns      k8s.gcr.io/coredns:1.6.7   k8s-app=kube-dns


NAMESPACE    NAME                                DESIRED  CURRENT  READY  AGE    CONTAINERS  IMAGES                    SELECTOR
NAMESPACE    NAME                                DESIRED  CURRENT  READY  AGE    CONTAINERS  IMAGES                    SELECTOR
kube-system  replicaset.apps/coredns-6dcc67dcbc   2         2         2       8m16s   coredns      k8s.gcr.io/coredns:1.3.1   k8s-app=kube-dns,pod-template-hash=6dcc67dcbc
kube-system  replicaset.apps/coredns-66bff467f8   1         1         1       4m59s   coredns      k8s.gcr.io/coredns:1.6.7   k8s-app=kube-dns,pod-template-hash=66bff467f8
</source>
</source>


= Usage =
= Usage (VirtualBox) =
<source lang=bash>
<source lang=bash>
time minikube start                              #start a cluster by running
# Start another local clusters named: --profile can be abbreviated with -p, default is 'minikube'
time minikube start --kubernetes-version v1.14.9 --profile minikube-v1.14.9-aws # AWS versions v1.14.9-eks-c0eccc
time minikube start --kubernetes-version v1.15.0 --profile minikube-v1.15.0
time minikube start --kubernetes-version v1.15.0 --profile minikube-v1.15.0
</source>
</source>
Line 266: Line 276:
</source>
</source>


== Working with profiles ==
== Profiles ==
;List profiles
;List profiles
<source>
<source>
Line 278: Line 288:




;Switich profiles
;Set and switch in between profiles
Switching <code>minikube</code> profile does automatically switch Kubernetes <code>kubectl</code> context
Setting the <code>minikube</code> profile does automatically change Kubernetes <code>kubectl config set-context</code> context.
<source lang=bash>
<source lang=bash>
# Switch minikube profile - in current context
minikube profile minikube-v1.15.11   # set the profile 'minikube-v1.15.11'
$> minikube profile minikube   # switch to profile: minikube
✅  minikube profile was successfully set to minikube-v1.15.11
✅  minikube profile was successfully set to minikube


# Configure minikube setting default profile
# Configure the persistent default profile
$> minikube config set profile minikube-v1.15.11
minikube config set profile minikube-v1.15.11
$> minikube config view  
minikube config view  
- profile: minikube-v1.15.11
- profile: minikube-v1.15.11
...


# Manually switch kubectl context - not needed if switched using 'minikube profile'
# Manually switch kubectl context - not needed when using 'minikube profile'
$> kubectl config set-context minikube
kubectl config set-context minikube
Context "minikube" modified.
Context "minikube" modified.
</source>
</source>

Latest revision as of 11:13, 8 May 2021

Minikube features

  • LoadBalancer - using minikube tunnel
  • Multi-cluster minikube start --profile cluster-1
  • Multi-node cluster minikube start --profile cluster-2 --nodes 2
  • NodePorts - using minikube service
  • Persistent Volumes
  • Ingress
  • RBAC
  • Dashboard - minikube dashboard
  • Container runtimes - start --container-runtime
  • Configure apiserver and kubelet options via command-line flags

Developer friendly features:

  • Filesystem mounts
  • Addons - a marketplace for developers to share configurations for running services on minikube
  • NVIDIA GPU support - for machine learning

Release notes CHANGELOG.mb

Documentation minikube.sigs.k8s.io

Addons

List addons

$ minikube addons list --profile minikube-v1.14.9
|-----------------------------|----------------------|--------------|
|         ADDON NAME          |        PROFILE       |    STATUS    |
|-----------------------------|----------------------|--------------|
| dashboard                   | minikube-v1.14.9     | disabled     |
| default-storageclass        | minikube-v1.14.9     | enabled ✅   |
| efk                         | minikube-v1.14.9     | disabled     |
| freshpod                    | minikube-v1.14.9     | disabled     |
| gvisor                      | minikube-v1.14.9     | disabled     |
| helm-tiller                 | minikube-v1.14.9     | disabled     |
| ingress                     | minikube-v1.14.9     | enabled ✅   |
| ingress-dns                 | minikube-v1.14.9     | enabled ✅   |
| istio                       | minikube-v1.14.9     | disabled     |
| istio-provisioner           | minikube-v1.14.9     | disabled     |
| logviewer                   | minikube-v1.14.9     | disabled     |
| metrics-server              | minikube-v1.14.9     | disabled     |
| nvidia-driver-installer     | minikube-v1.14.9     | disabled     |
| nvidia-gpu-device-plugin    | minikube-v1.14.9     | disabled     |
| registry                    | minikube-v1.14.9     | disabled     |
| registry-aliases            | minikube-v1.14.9     | disabled     |
| registry-creds              | minikube-v1.14.9     | disabled     |
| storage-provisioner         | minikube-v1.14.9     | enabled ✅   |
| storage-provisioner-gluster | minikube-v1.14.9     | disabled     |
|-----------------------------|----------------------|--------------|


Usage

minikube -p efk start  --memory 8192
minikube -p efk addons enable efk     # pulling images and setting up can take ~10 min
minikube -p efk addons open efk       # opens Kibana

minikube addons enable heapster       # get some insight into CPU

# Show images
minikube addons images metrics-server
  - metrics-server has following images:
|---------------|--------------------------------------------------------------------------------------------------------------|------------------|
|  IMAGE NAME   |                                                DEFAULT IMAGE                                                 | DEFAULT REGISTRY |
|---------------|--------------------------------------------------------------------------------------------------------------|------------------|
| MetricsServer | metrics-server/metrics-server:v0.4.2@sha256:dbc33d7d35d2a9cc5ab402005aa7a0d13be6192f3550c7d42cba8d2d5e3a5d62 | k8s.gcr.io       |
|---------------|--------------------------------------------------------------------------------------------------------------|------------------|

Install or upgrade on Linux

# Latest from GitHub
VERSION=$(curl --silent "https://api.github.com/repos/kubernetes/minikube/releases/latest" | jq -r .tag_name); echo $VERSION
curl -LO https://github.com/kubernetes/minikube/releases/download/$VERSION/minikube-linux-amd64 \
      && sudo install minikube-linux-amd64 /usr/local/bin/minikube

# Latest from GoogleAPI
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 \
      && sudo install minikube-linux-amd64 /usr/local/bin/minikube

# Specific version from GitHub
curl -LO https://github.com/kubernetes/minikube/releases/download/v1.7.3/minikube-linux-amd64 \
      && sudo install minikube-linux-amd64 /usr/local/bin/minikube

$> $ minikube version 
minikube version: v1.17.0

Start on Windows

Install <syntaxhighlightjs lang="powershell"> choco install minikube kubernetes-cli </syntaxhighlightjs>


HyperV <syntaxhighlightjs lang="powershell">

  1. HyperV set config (option 1)

minikube config set vm-driver hyperv minikube config set hyperv-virtual-switch "Default Switch" minikube start

  1. HyperV single command (option 2)

minikube start --vm-driver hyperv --hyperv-virtual-switch k8s-node-1 --profile k8s-node-1

  1. --hyperv-virtual-switch :- need to exist

</syntaxhighlightjs>


Virtualbox <syntaxhighlightjs lang="powershell"> minikube start --profile k8s-node-1

  • minikube v1.2.0 on windows (amd64)
  • Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
  • Configuring environment for Kubernetes v1.15.0 on Docker 18.09.6
  • Downloading kubelet v1.15.0
  • Downloading kubeadm v1.15.0
  • Pulling images ...
  • Launching Kubernetes ...
  • Verifying: apiserver proxy etcd scheduler controller dns
  • Done! kubectl is now configured to use "k8s-node-1"

PS C:\Users\Sylwia> kubectl.exe get nodes -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME minikube Ready master 23m v1.15.0 10.0.2.15 <none> Buildroot 2018.05.3 4.15.0 docker://18.9.6 </syntaxhighlightjs>

Start - Linux (Ubuntu)

Start sequence

VERSION=v1.18.9
time minikube start --kubernetes-version $VERSION --profile minikube-$VERSION
😄  [minikube-v1.18.9] minikube v1.19.0 on Ubuntu 20.04
✨  Automatically selected the docker driver. Other choices: virtualbox, ssh, none
👍  Starting control plane node minikube-v1.18.9 in cluster minikube-v1.18.9
🚜  Pulling base image ...
    > gcr.io/k8s-minikube/kicbase...: 357.67 MiB / 357.67 MiB  100.00% 4.70 MiB
🔥  Creating docker container (CPUs=2, Memory=3900MB) ...
🐳  Preparing Kubernetes v1.18.9 on Docker 20.10.5 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube-v1.18.9" cluster and "default" namespace by default

real	2m4.966s
user	0m6.658s
sys	0m3.768s


$> minikube profile list
|-----------------------------|-----------|---------|--------------|------|---------|---------|-------|
|           Profile           | VM Driver | Runtime |      IP      | Port | Version | Status  | Nodes |
|-----------------------------|-----------|---------|--------------|------|---------|---------|-------|
| minikube-v1.18.9            | docker    | docker  | 192.168.58.2 | 8443 | v1.18.9 | Running |     1 |
|-----------------------------|-----------|---------|--------------|------|---------|---------|-------|

$> minikube status -p minikube-v1.18.9
minikube-v1.18.9
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

$> kubectl get nodes -o wide
NAME               STATUS   ROLES    AGE     VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
minikube-v1.18.9   Ready    master   2m34s   v1.18.9   192.168.58.2   <none>        Ubuntu 20.04.1 LTS   5.8.0-50-generic   docker://20.10.5


$ kubectl get all -o wide --all-namespaces
kubectl get all -o wide --all-namespaces
NAMESPACE     NAME                                                 READY   STATUS    RESTARTS   AGE     IP             NODE                     NOMINATED NODE   READINESS GATES
kube-system   pod/coredns-66bff467f8-wzjx7                         1/1     Running   0          4m59s   172.17.0.2     minikube-v1.18.9         <none>           <none>
kube-system   pod/etcd-minikube-v1.18.9                            1/1     Running   0          5m7s    192.168.49.2   minikube-v1.18.9         <none>           <none>
kube-system   pod/kube-apiserver-minikube-v1.18.9                  1/1     Running   0          5m7s    192.168.49.2   minikube-v1.18.9         <none>           <none>
kube-system   pod/kube-controller-manager-minikube-v1.18.9         1/1     Running   0          5m7s    192.168.49.2   minikube-v1.18.9         <none>           <none>
kube-system   pod/kube-proxy-zdqc9                                 1/1     Running   0          4m59s   192.168.49.2   minikube-v1.18.9         <none>           <none>
kube-system   pod/kube-scheduler-minikube-v1.18.9                  1/1     Running   0          5m7s    192.168.49.2   minikube-v1.18.9         <none>           <none>
kube-system   pod/storage-provisioner                              1/1     Running   1          5m13s   192.168.49.2   minikube-v1.18.9         <none>           <none>

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE     SELECTOR
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  5m15s   <none>
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   5m14s   k8s-app=kube-dns

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE     CONTAINERS   IMAGES                          SELECTOR
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux   5m14s   kube-proxy   k8s.gcr.io/kube-proxy:v1.18.9   k8s-app=kube-proxy

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                     SELECTOR
kube-system   deployment.apps/coredns   1/1     1            1           5m14s   coredns      k8s.gcr.io/coredns:1.6.7   k8s-app=kube-dns

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE     CONTAINERS   IMAGES                     SELECTOR
kube-system   replicaset.apps/coredns-66bff467f8   1         1         1       4m59s   coredns      k8s.gcr.io/coredns:1.6.7   k8s-app=kube-dns,pod-template-hash=66bff467f8

Usage (VirtualBox)

time minikube start --kubernetes-version v1.15.0 --profile minikube-v1.15.0


Get port exposed to host via Virtualbox used by minikube to interact with a cluster

$ sudo ss -ltpn
State  Recv-Q  Send-Q    Local Address:Port    Peer Address:Port                                               
LISTEN 0       128       127.0.0.53%lo:53           0.0.0.0:*      users:(("systemd-resolve",pid=1009,fd=13))  
LISTEN 0       10            127.0.0.1:45111 #<--   0.0.0.0:*      users:(("VBoxHeadless",pid=26918,fd=21))    
LISTEN 0       5             127.0.0.1:631          0.0.0.0:*      users:(("cupsd",pid=27629,fd=7))            
LISTEN 0       5                 [::1]:631             [::]:*      users:(("cupsd",pid=27629,fd=6))

$ vboxmanage showvminfo k8s-v1.15.0 | 'grep port ='                            vv
NIC 1 Rule(0):   name = ssh, protocol = tcp, host ip = 127.0.0.1, host port = 45111, guest ip = , guest port = 22
                                                                               ^^


Access to minikube VM using ssh
minikube ssh cat /etc/*rel*
$ minikube ssh 
                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$
cat /etc/*rel*
NAME=Buildroot
VERSION=2018.05.3
ID=buildroot
VERSION_ID=2018.05.3
PRETTY_NAME="Buildroot 2018.05.3"
NAME=Buildroot
VERSION=2018.05.3
ID=buildroot
VERSION_ID=2018.05.3
PRETTY_NAME="Buildroot 2018.05.3"


#access Kubernetes Dashboard within Minikube
minikube dashboard
🤔  Verifying dashboard health ...
🚀  Launching proxy ...
🤔  Verifying proxy health ...
🎉  Opening http://127.0.0.1:44835/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ in your default browser...

# Use kubectl to interact with the local cluster [[Kubernetes/Tools#create_pod]]
kubectl run --generator=run-pod/v1         hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #correct
kubectl run --generator=deployment/apps.v1 hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #deprecated
kubectl run       ^^default^^              hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #deprecated

# Expose a service as a NodePort
kubectl expose pod        hello-minikube --type=NodePort
kubectl expose deployment hello-minikube --type=NodePort #when pod was created via deployment deprecated option

# Determine the NodePort of the service
kubectl get service <hello-minikube> --output='jsonpath="{.spec.ports[0].nodePort}"'

# curl the endpoint
minikube ssh
curl -s -H 'Cache-Control: no-cache' http://<svc-cluster-ip>:8080 #note 'watch won't work as curl or service proxy 

#minikube makes it easy to open this exposed endpoint in your browser
minikube service hello-minikube

minikube stop   #stop your local cluster
minikube delete #delete your local cluster

Profiles

List profiles
$> minikube profile list # minikube v1.9.2
|-------------------|-----------|---------|------------|------|----------|---------|
|           Profile | VM Driver | Runtime |     IP     | Port | Version  | Status  |
|-------------------|-----------|---------|------------|------|----------|---------|
| minikube-v1.15.11 | docker    | docker  | 172.17.0.2 | 8443 | v1.15.11 | Stopped |
|-------------------|-----------|---------|------------|------|----------|---------|


Set and switch in between profiles

Setting the minikube profile does automatically change Kubernetes kubectl config set-context context.

minikube profile minikube-v1.15.11    # set the profile 'minikube-v1.15.11'
✅  minikube profile was successfully set to minikube-v1.15.11

# Configure the persistent default profile
minikube config set profile minikube-v1.15.11
minikube config view 
- profile: minikube-v1.15.11

# Manually switch kubectl context - not needed when using 'minikube profile'
kubectl config set-context minikube
Context "minikube" modified.


Profile local config files
$> ls -1 ~/.minikube/profiles/ # list all profiles
minikube-v1.15.11
minikube

# Minikube show current profile
$> minikube profile
minikube-v1.15.11
$> cat ~/.minikube/config/config.json | jq -r .profile #alternative option

# Kubectl  show current context
$ kubectl config current-context
$ kubectl config get-contexts   #get-clusters
CURRENT   NAME                  CLUSTER             AUTHINFO           NAMESPACE
*         minikube-v1.15.11     minikube-v1.15.11   minikube-v1.15.11
          minikube              minikube            minikube


Clean up if needed
minikube delete -p <profile>
kubectl config delete-context <context-name>
kubectl config delete-context <delete-cluster>


Example of ~/.minikube/profiles/<minikube-profile>/config.json
jq .MachineConfig config.json jq .KubernetesConfig config.json
{
  "KubernetesVersion": "v1.13.7",
  "NodeIP": "192.168.99.102",
  "NodePort": 8443,
  "NodeName": "minikube",
  "APIServerName": "minikubeCA",
  "APIServerNames": null,
  "APIServerIPs": null,
  "DNSDomain": "cluster.local",
  "ContainerRuntime": "docker",
  "CRISocket": "",
  "NetworkPlugin": "",
  "FeatureGates": "",
  "ServiceCIDR": "10.96.0.0/12",
  "ImageRepository": "",
  "ExtraOptions": null,
  "ShouldLoadCachedImages": true,
  "EnableDefaultCNI": false
}
{
  "KeepContext": false,
  "MinikubeISO": "https://storage.googleapis.com/minikube/iso/minikube-v1.2.0.iso",
  "Memory": 2048,
  "CPUs": 2,
  "DiskSize": 20000,
  "VMDriver": "virtualbox",
  "ContainerRuntime": "docker",
  "HyperkitVpnKitSock": "",
  "HyperkitVSockPorts": [],
  "XhyveDiskDriver": "ahci-hd",
  "DockerEnv": null,
  "InsecureRegistry": null,
  "RegistryMirror": null,
  "HostOnlyCIDR": "192.168.99.1/24",
  "HypervVirtualSwitch": "",
  "KvmNetwork": "default",
  "DockerOpt": null,
  "DisableDriverMounts": false,
  "NFSShare": [],
  "NFSSharesRoot": "/nfsshares",
  "UUID": "",
  "GPU": false,
  "Hidden": false,
  "NoVTXCheck": false
}

Setting default configuration

It's useful to set default configuration, eg. you always work with AWS EKS, so it comes at certain versioning we could set as a default if not explicitly specified.


List configurable options

$ minikube config 
config modifies minikube config files using subcommands like "minikube config set driver kvm"
Configurable fields: 

 * driver              * log_dir                     * ShowDriverDeprecationNotification
 * vm-driver           * kubernetes-version          * ShowBootstrapperDeprecationNotification
 * container-runtime   * iso-url                     * insecure-registry
 * feature-gates       * WantUpdateNotification      * hyperv-virtual-switch
 * v                   * ReminderWaitPeriodInHours   * disable-driver-mounts
 * cpus                * WantReportError             * cache
 * disk-size           * WantReportErrorPrompt       * embed-certs
 * host-only-cidr      * WantKubectlDownloadMsg      * native-ssh
 * memory              * WantNoneDriverWarning
                       * profile
                       * bootstrapper
Available Commands:
  get         Gets the value of PROPERTY_NAME from the minikube config file
  set         Sets an individual value in a minikube config file
  unset       unsets an individual value in a minikube config file
  view        Display values currently set in the minikube config file

Usage:
  minikube config SUBCOMMAND [flags] [options]


Set default version of Kubernetes
$ minikube config set kubernetes-version v1.14.9
$ minikube config view 
- kubernetes-version: v1.14.9

Loadbalancer - minikube tunnel

A LoadBalancer service is the standard way to expose a service to the internet. With this method, each service gets it’s own IP address. In minikube services of type LoadBalancer can be exposed via the minikube tunnel command. It will run until Ctrl-C is hit.


minikube tunnel runs as a separate daemon, creating a network route on the host to the service CIDR of the cluster using the cluster’s IP address as a gateway. The tunnel command exposes the external IP directly to any program running on the host operating system. The low-level commands used to manage routes are /sbin/ip,/sbin/route that optionally can be added to suedors file to make a life a little easier.

$ minikube profile list
|-----------------|------------|----------------|-----------|--------------------|
|     Profile     | VM Driver  |     NodeIP     | Node Port | Kubernetes Version |
|-----------------|------------|----------------|-----------|--------------------|
| efk             | virtualbox | 192.168.99.113 |      8443 | v1.16.0            |
|-----------------|------------|----------------|-----------|--------------------|

# Before
$ kubectl -n efk get svc # note <pending> EXTERNAL-IP
NAMESPACE NAME                  TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)       
efk       service/elasticsearch LoadBalancer   10.104.149.221  <pending>       9200:30826/TCP
efk       service/kibana        LoadBalancer   10.97.198.117   <pending>       5601:32540/TCP

$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         vodafone.connec 0.0.0.0         UG    600    0        0 wlp4s0
link-local      0.0.0.0         255.255.0.0     U     1000   0        0 wlp4s0
192.168.1.0     0.0.0.0         255.255.255.0   U     600    0        0 wlp4s0
192.168.99.0    0.0.0.0         255.255.255.0   U     0      0        0 vboxnet2

# After - EXTERNAL-IP have been assigned from a pool of 10.96.0.0/12
NAMESPACE NAME                  TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)       
efk       service/elasticsearch LoadBalancer   10.104.149.221  10.104.149.221  9200:30826/TCP
efk       service/kibana        LoadBalancer   10.97.198.117   10.97.198.117   5601:32540/TCP

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         vodafone.connec 0.0.0.0         UG    600    0        0 wlp4s0
10.96.0.0       192.168.99.113  255.240.0.0     UG    0      0        0 vboxnet2 # <- new route to minikube NodeIP created 
link-local      0.0.0.0         255.255.0.0     U     1000   0        0 wlp4s0
192.168.1.0     0.0.0.0         255.255.255.0   U     600    0        0 wlp4s0
192.168.99.0    0.0.0.0         255.255.255.0   U     0      0        0 vboxnet2

$ minikube --profile efk tunnel
Status:
        machine: efk
        pid: 8721
        route: 10.96.0.0/12 -> 192.168.99.113 # route status
        minikube: Running
        services: [elasticsearch, kibana]     # exposed services
    errors: 
                minikube: no errors
                router: no errors
                loadbalancer emulator: no errors
# Verify
$ nc -zv 10.97.198.117 5601 # kibana
Connection to 10.97.198.117 5601 port [tcp/*] succeeded!
$ nc -zv 10.104.149.221 9200 # elasticsearch
Connection to 10.104.149.221 9200 port [tcp/*] succeeded!
$ curl http://10.97.198.117:5601/status


Clean up based on info ~/.minikube/tunnels.json when shuts unexpected

minikube tunnel --cleanup


Avoid to typing password for minikube tunnel, by allowing a group or user to execute ip, route commands without asking for password. Example for Ubuntu 18.04 can be seen below.

$ sudo visudo
...
# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL
%admin ALL=(ALL) NOPASSWD: /sbin/ip,/sbin/route  # <- add this line in this order
...

multi node cluster with kvm

This is now supported natively using --nodes flag:

minikube start --profile minikube-v1.15.11 --nodes=2  --kubernetes-version v1.15.11
minikube ssh   --profile minikube-v1.15.11 --node=m02
minikube       --profile minikube-v1.15.11   node add         # add new node
minikube       --profile minikube-v1.15.11   node delete m03  # delete

$ kubectl get nodes
NAME                    STATUS     ROLES    AGE     VERSION
minikube-v1.15.11       Ready      master   3h25m   v1.15.11
minikube-v1.15.11-m02   NotReady   <none>   3h24m   v1.15.11
minikube-v1.15.11-m03   NotReady   <none>   16m     v1.15.11 # <- add / remove


Nodes names have following naming convention

  • node-1-name: minikube-v1.14.9
  • node-2-name: minikube-v1.14.9-m02

ingress and local DNS (ingress-dns)

TODO:

Add required addons

minikube addons enable ingress --profile minikube-v1.14.9
🌟  The 'ingress' addon is enabled
minikube addons enable ingress-dns --profile minikube-v1.14.9
🌟  The 'ingress-dns' addon is enabled
Add minikube ip as a dns server

Ubuntu 18.04 LTS, configure resolvconf

sudo apt install resolvconf
sudo -p /etc/resolvconf/resolv.conf.d       # possibly created by installation package
sudo vi /etc/resolvconf/resolv.conf.d/base
minikube ip --profile minikube-v1.14.9
192.168.99.126

# Update resolverconf
sudo vi /etc/resolvconf/resolv.conf.d/base
search test
nameserver 192.168.99.126
timeout 5

# Apply changes
sudo resolvconf -u
systemctl disable --now resolvconf.service 

# Test, Add the test ingress
kubectl apply -f https://raw.githubusercontent.com/kubernetes/minikube/master/deploy/addons/ingress-dns/example/example.yaml

# Validate DNS queries are returning A records
nslookup hello-john.test $(minikube ip --profile minikube-v1.14.9)
Server:		192.168.99.126
Address:	192.168.99.126#53

Non-authoritative answer:
Name:	hello-john.test
Address: 192.168.99.126
Name:	hello-john.test
Address: 192.168.99.126 # <- this is ingress IP address

$ kubectl -n kube-system get ingresses.networking.k8s.io example-ingress 
NAME              HOSTS                             ADDRESS          PORTS   AGE
example-ingress   hello-john.test,hello-jane.test   192.168.99.126   80      102s


Note: .local is a reserved TLD, do not use as this is a reserved TLD for mDNS and bind9 DNS servers


Note: .localhost domains will not correctly resolve on chromium since it is used as a loopback address. Instead use .test, .example, or .invalid

References