Difference between revisions of "Kubernetes/minikube"
Jump to navigation
Jump to search
Loadbalancer -
| Line 336: | Line 336: | ||
} </source> | } </source> | ||
|} | |} | ||
= Loadbalancer - <code>minikube tunnel</code> = | |||
A LoadBalancer service is the standard way to expose a service to the internet. With this method, each service gets it’s own IP address. In <tt>minikube</tt> services of type <tt>LoadBalancer</tt> can be exposed via the <code>minikube tunnel</code> command. It will run until <code>Ctrl-C</code> is hit. | |||
<code>minikube tunnel</code> runs as a separate daemon, creating a network route on the host to the service CIDR of the cluster using the cluster’s IP address as a gateway. The tunnel command exposes the external IP directly to any program running on the host operating system. The low-level commands used to manage routes are <code>/sbin/ip,/sbin/route</code> that optionally can be added to <tt>suedors</tt> file to make a life a little easier. | |||
<source lang=bash> | |||
$ minikube profile list | |||
|-----------------|------------|----------------|-----------|--------------------| | |||
| Profile | VM Driver | NodeIP | Node Port | Kubernetes Version | | |||
|-----------------|------------|----------------|-----------|--------------------| | |||
| efk | virtualbox | 192.168.99.113 | 8443 | v1.16.0 | | |||
|-----------------|------------|----------------|-----------|--------------------| | |||
# Before | |||
$ kubectl -n efk get svc # note <pending> EXTERNAL-IP | |||
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) | |||
efk service/elasticsearch LoadBalancer 10.104.149.221 <pending> 9200:30826/TCP | |||
efk service/kibana LoadBalancer 10.97.198.117 <pending> 5601:32540/TCP | |||
$ route | |||
Kernel IP routing table | |||
Destination Gateway Genmask Flags Metric Ref Use Iface | |||
default vodafone.connec 0.0.0.0 UG 600 0 0 wlp4s0 | |||
link-local 0.0.0.0 255.255.0.0 U 1000 0 0 wlp4s0 | |||
192.168.1.0 0.0.0.0 255.255.255.0 U 600 0 0 wlp4s0 | |||
192.168.99.0 0.0.0.0 255.255.255.0 U 0 0 0 vboxnet2 | |||
# After - EXTERNAL-IP have been assigned from a pool of 10.96.0.0/12 | |||
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) | |||
efk service/elasticsearch LoadBalancer 10.104.149.221 10.104.149.221 9200:30826/TCP | |||
efk service/kibana LoadBalancer 10.97.198.117 10.97.198.117 5601:32540/TCP | |||
Kernel IP routing table | |||
Destination Gateway Genmask Flags Metric Ref Use Iface | |||
default vodafone.connec 0.0.0.0 UG 600 0 0 wlp4s0 | |||
10.96.0.0 192.168.99.113 255.240.0.0 UG 0 0 0 vboxnet2 # <- new route to minikube NodeIP created | |||
link-local 0.0.0.0 255.255.0.0 U 1000 0 0 wlp4s0 | |||
192.168.1.0 0.0.0.0 255.255.255.0 U 600 0 0 wlp4s0 | |||
192.168.99.0 0.0.0.0 255.255.255.0 U 0 0 0 vboxnet2 | |||
$ minikube --profile efk tunnel | |||
Status: | |||
machine: efk | |||
pid: 8721 | |||
route: 10.96.0.0/12 -> 192.168.99.113 # route status | |||
minikube: Running | |||
services: [elasticsearch, kibana] | |||
errors: | |||
minikube: no errors | |||
router: no errors | |||
loadbalancer emulator: no errors | |||
</source> | |||
Clean up based on info <code>~/.minikube/tunnels.json</code> when shuts unexpected | |||
<source lang=bash> | |||
minikube tunnel --cleanup | |||
</source> | |||
Avoid to typing password for <code>minikube tunnel</code>, by allowing a group or user to execute <code>ip, route</code> commands without asking for password. Example for Ubuntu 18.04 can be seen below. | |||
<source> | |||
$ sudo visudo | |||
... | |||
# Members of the admin group may gain root privileges | |||
%admin ALL=(ALL) ALL | |||
%admin ALL=(ALL) NOPASSWD: /sbin/ip,/sbin/route # <- add this line in this order | |||
... | |||
</source> | |||
= References = | = References = | ||
Revision as of 21:19, 29 September 2019
Minikube features v1.4.0
- LoadBalancer - using
minikube tunnel - Multi-cluster - using
minikube start -p <name> - NodePorts - using
minikube service - https://minikube.sigs.k8s.io/docs/reference/persistent_volumes/ Persistent Volumes]
- Ingress
- RBAC
- Dashboard -
minikube dashboard - Container runtimes -
start --container-runtime - Configure apiserver and kubelet options via command-line flags
Developer friendly features:
Release notes CHANGELOG.mb
Addons
List addons
$ minikube addons list - addon-manager: enabled - dashboard: enabled - default-storageclass: enabled - efk: disabled - freshpod: disabled - gvisor: disabled - heapster: disabled - ingress: disabled - logviewer: disabled - metrics-server: disabled - nvidia-driver-installer: disabled - nvidia-gpu-device-plugin: disabled - registry: disabled - registry-creds: disabled - storage-provisioner: enabled - storage-provisioner-gluster: disabled
Usage
minikube -p efk start --memory 8192 minikube -p efk addons enable efk # pulling images and setting up can take ~10 min minikube -p efk addons open efk # opens Kibana minikube addons enable heapster #get some insight into CPU
Install/upgrade on Linux
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 \
&& sudo install minikube-linux-amd64 /usr/local/bin/minikube
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 39.8M 100 39.8M 0 0 160k 0 0:04:14 0:04:14 --:--:-- 174k
Start on Windows
Install <syntaxhighlightjs lang="powershell"> choco install minikube kubernetes-cli </syntaxhighlightjs>
HyperV
<syntaxhighlightjs lang="powershell">
- HyperV set config (option 1)
minikube config set vm-driver hyperv minikube config set hyperv-virtual-switch "Default Switch" minikube start
- HyperV single command (option 2)
minikube start --vm-driver hyperv --hyperv-virtual-switch k8s-node-1 --profile k8s-node-1
- --hyperv-virtual-switch :- need to exist
</syntaxhighlightjs>
Virtualbox
<syntaxhighlightjs lang="powershell">
minikube start --profile k8s-node-1
- minikube v1.2.0 on windows (amd64)
- Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
- Configuring environment for Kubernetes v1.15.0 on Docker 18.09.6
- Downloading kubelet v1.15.0
- Downloading kubeadm v1.15.0
- Pulling images ...
- Launching Kubernetes ...
- Verifying: apiserver proxy etcd scheduler controller dns
- Done! kubectl is now configured to use "k8s-node-1"
PS C:\Users\Sylwia> kubectl.exe get nodes -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME minikube Ready master 23m v1.15.0 10.0.2.15 <none> Buildroot 2018.05.3 4.15.0 docker://18.9.6 </syntaxhighlightjs>
Start
Example of start sequence
minikube start 😄 minikube v1.2.0 on linux (amd64) 🔥 Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ... 🐳 Configuring environment for Kubernetes v1.15.0 on Docker 18.09.6 E0728 00:14:31.657125 26453 start.go:403] Error caching images: Caching images for kubeadm: caching images: caching image /home/user1/.minikube/cache/images/k8s.gcr.io/k8s-dns-kube-dns-amd64_1.14.13: stream error: stream ID 25; INTERNAL_ERROR ❌ Unable to load cached images: loading cached images: loading image /home/pipaw/.minikube/cache/images/k8s.gcr.io/k8s-dns-sidecar-amd64_1.14.13: stat /home/user1/.minikube/cache/images/k8s.gcr.io/k8s-dns-sidecar-amd64_1.14.13: no such file or directory 💾 Downloading kubeadm v1.15.0 💾 Downloading kubelet v1.15.0 🚜 Pulling images ... 🚀 Launching Kubernetes ... ⌛ Verifying: apiserver proxy etcd scheduler controller dns 🏄 Done! kubectl is now configured to use "minikube" $ minikube status host: Running kubelet: Running apiserver: Running kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.101 $ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME minikube Ready master 8h v1.15.0 10.0.2.15 <none> Buildroot 2018.05.3 4.15.0 docker://18.9.6 $ kubectl get all -o wide --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system pod/coredns-5c98db65d4-d688k 1/1 Running 0 8h 172.17.0.2 minikube <none> <none> kube-system pod/coredns-5c98db65d4-xdq6d 1/1 Running 0 8h 172.17.0.3 minikube <none> <none> kube-system pod/etcd-minikube 1/1 Running 0 8h 10.0.2.15 minikube <none> <none> kube-system pod/kube-addon-manager-minikube 1/1 Running 0 8h 10.0.2.15 minikube <none> <none> kube-system pod/kube-apiserver-minikube 1/1 Running 0 8h 10.0.2.15 minikube <none> <none> kube-system pod/kube-controller-manager-minikube 1/1 Running 0 8h 10.0.2.15 minikube <none> <none> kube-system pod/kube-proxy-j5f2h 1/1 Running 0 8h 10.0.2.15 minikube <none> <none> kube-system pod/kube-scheduler-minikube 1/1 Running 0 8h 10.0.2.15 minikube <none> <none> kube-system pod/storage-provisioner 1/1 Running 0 8h 10.0.2.15 minikube <none> <none> NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 8h <none> kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 8h k8s-app=kube-dns NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR kube-system daemonset.apps/kube-proxy 1 1 1 1 1 beta.kubernetes.io/os=linux 8h kube-proxy k8s.gcr.io/kube-proxy:v1.15.0 k8s-app=kube-proxy NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR kube-system deployment.apps/coredns 2/2 2 2 8h coredns k8s.gcr.io/coredns:1.3.1 k8s-app=kube-dns NAMESPACE NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR kube-system replicaset.apps/coredns-5c98db65d4 2 2 2 8h coredns k8s.gcr.io/coredns:1.3.1 k8s-app=kube-dns,pod-template-hash=5c98db65d4
Usage
time minikube start #start a cluster by running # Start another local clusters named: --profile can be abbreviated with -p, default is 'minikube' time minikube start --kubernetes-version v1.13.7 --profile k8s-v1.13.7-aws #AWS versions 1.13.7,1.12.6 time minikube start --kubernetes-version v1.15.0 --profile k8s-v1.15.0
Multi-profiles
$ minikube profile list |-----------------|------------|----------------|-----------|--------------------| | Profile | VM Driver | NodeIP | Node Port | Kubernetes Version | |-----------------|------------|----------------|-----------|--------------------| | k8s-v1.13.7-aws | virtualbox | 192.168.99.105 | 8443 | v1.15.0 | | k8s-v1.15.0 | virtualbox | 192.168.99.104 | 8443 | v1.15.0 | |-----------------|------------|----------------|-----------|--------------------|
Get port exposed to host via Virtualbox used by minikube to interact with a cluster
$ sudo ss -ltpn
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=1009,fd=13))
LISTEN 0 10 127.0.0.1:45111 #<-- 0.0.0.0:* users:(("VBoxHeadless",pid=26918,fd=21))
LISTEN 0 5 127.0.0.1:631 0.0.0.0:* users:(("cupsd",pid=27629,fd=7))
LISTEN 0 5 [::1]:631 [::]:* users:(("cupsd",pid=27629,fd=6))
$ vboxmanage showvminfo k8s-v1.15.0 | 'grep port =' vv
NIC 1 Rule(0): name = ssh, protocol = tcp, host ip = 127.0.0.1, host port = 45111, guest ip = , guest port = 22
^^
minikube ssh
|
cat /etc/*rel*
|
|---|---|
$ minikube ssh
_ _
_ _ ( ) ( )
___ ___ (_) ___ (_)| |/') _ _ | |_ __
/' _ ` _ `\| |/' _ `\| || , < ( ) ( )| '_`\ /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )( ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)
$
|
cat /etc/*rel* NAME=Buildroot VERSION=2018.05.3 ID=buildroot VERSION_ID=2018.05.3 PRETTY_NAME="Buildroot 2018.05.3" NAME=Buildroot VERSION=2018.05.3 ID=buildroot VERSION_ID=2018.05.3 PRETTY_NAME="Buildroot 2018.05.3" |
#access Kubernetes Dashboard within Minikube
minikube dashboard
🤔 Verifying dashboard health ...
🚀 Launching proxy ...
🤔 Verifying proxy health ...
🎉 Opening http://127.0.0.1:44835/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ in your default browser...
#Use kubectl to interact with the local cluster
kubectl run --generator=run-pod/v1 hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #correct
kubectl run --generator=deployment/apps.v1 hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #deprecated
kubectl run ^^default^^ hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080 #deprecated
#Expose a service as a NodePort
kubectl expose pod hello-minikube --type=NodePort
kubectl expose deployment hello-minikube --type=NodePort #when pod was created via deployment deprecated option
#Determine the NodePort of the service
kubectl get service <hello-minikube> --output='jsonpath="{.spec.ports[0].nodePort}"'
#curl this endpoint
minikube ssh
curl -s -H 'Cache-Control: no-cache' http://<svc-cluster-ip>:8080 #note 'watch won't work as curl or service proxy
#minikube makes it easy to open this exposed endpoint in your browser
minikube service hello-minikube
minikube stop #stop your local cluster
minikube delete #delete your local cluster
Working with profiles
$ ls -1 ~/.minikube/profiles/ #list all profiles
aws-v1.13.7-cluster
minikube #default profile
#Minikube show current profile
$ minikube profile
aws-v1.13.7-cluster
$ cat ~/.minikube/config/config.json | jq -r .profile #alternative option
#Kubectl show current context
$ kubectl config current-context
$ kubectl config get-contexts #get-clusters
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* aws-v1.13.7-cluster aws-v1.13.7-cluster aws-v1.13.7-cluster
minikube minikube minikube
Switching the Minikube profile does automatically switch the k8s context, and vice versa (v1.2.0 - 2019-06-24)
#Switch minikube profile $ minikube profile minikube #switch to profile: minikube ✅ minikube profile was successfully set to minikube #Switch kubectl context $ kubectl config set-context minikube Context "minikube" modified.
Clean up if needed
minikube delete -p <profile> kubectl config delete-context <context-name> kubectl config delete-context <delete-cluster>
jq .MachineConfig config.json
|
jq .KubernetesConfig config.json
|
|---|---|
{
"KubernetesVersion": "v1.13.7",
"NodeIP": "192.168.99.102",
"NodePort": 8443,
"NodeName": "minikube",
"APIServerName": "minikubeCA",
"APIServerNames": null,
"APIServerIPs": null,
"DNSDomain": "cluster.local",
"ContainerRuntime": "docker",
"CRISocket": "",
"NetworkPlugin": "",
"FeatureGates": "",
"ServiceCIDR": "10.96.0.0/12",
"ImageRepository": "",
"ExtraOptions": null,
"ShouldLoadCachedImages": true,
"EnableDefaultCNI": false
}
|
{
"KeepContext": false,
"MinikubeISO": "https://storage.googleapis.com/minikube/iso/minikube-v1.2.0.iso",
"Memory": 2048,
"CPUs": 2,
"DiskSize": 20000,
"VMDriver": "virtualbox",
"ContainerRuntime": "docker",
"HyperkitVpnKitSock": "",
"HyperkitVSockPorts": [],
"XhyveDiskDriver": "ahci-hd",
"DockerEnv": null,
"InsecureRegistry": null,
"RegistryMirror": null,
"HostOnlyCIDR": "192.168.99.1/24",
"HypervVirtualSwitch": "",
"KvmNetwork": "default",
"DockerOpt": null,
"DisableDriverMounts": false,
"NFSShare": [],
"NFSSharesRoot": "/nfsshares",
"UUID": "",
"GPU": false,
"Hidden": false,
"NoVTXCheck": false
}
|
Loadbalancer - minikube tunnel
A LoadBalancer service is the standard way to expose a service to the internet. With this method, each service gets it’s own IP address. In minikube services of type LoadBalancer can be exposed via the minikube tunnel command. It will run until Ctrl-C is hit.
minikube tunnel runs as a separate daemon, creating a network route on the host to the service CIDR of the cluster using the cluster’s IP address as a gateway. The tunnel command exposes the external IP directly to any program running on the host operating system. The low-level commands used to manage routes are /sbin/ip,/sbin/route that optionally can be added to suedors file to make a life a little easier.
$ minikube profile list
|-----------------|------------|----------------|-----------|--------------------|
| Profile | VM Driver | NodeIP | Node Port | Kubernetes Version |
|-----------------|------------|----------------|-----------|--------------------|
| efk | virtualbox | 192.168.99.113 | 8443 | v1.16.0 |
|-----------------|------------|----------------|-----------|--------------------|
# Before
$ kubectl -n efk get svc # note <pending> EXTERNAL-IP
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
efk service/elasticsearch LoadBalancer 10.104.149.221 <pending> 9200:30826/TCP
efk service/kibana LoadBalancer 10.97.198.117 <pending> 5601:32540/TCP
$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default vodafone.connec 0.0.0.0 UG 600 0 0 wlp4s0
link-local 0.0.0.0 255.255.0.0 U 1000 0 0 wlp4s0
192.168.1.0 0.0.0.0 255.255.255.0 U 600 0 0 wlp4s0
192.168.99.0 0.0.0.0 255.255.255.0 U 0 0 0 vboxnet2
# After - EXTERNAL-IP have been assigned from a pool of 10.96.0.0/12
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
efk service/elasticsearch LoadBalancer 10.104.149.221 10.104.149.221 9200:30826/TCP
efk service/kibana LoadBalancer 10.97.198.117 10.97.198.117 5601:32540/TCP
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default vodafone.connec 0.0.0.0 UG 600 0 0 wlp4s0
10.96.0.0 192.168.99.113 255.240.0.0 UG 0 0 0 vboxnet2 # <- new route to minikube NodeIP created
link-local 0.0.0.0 255.255.0.0 U 1000 0 0 wlp4s0
192.168.1.0 0.0.0.0 255.255.255.0 U 600 0 0 wlp4s0
192.168.99.0 0.0.0.0 255.255.255.0 U 0 0 0 vboxnet2
$ minikube --profile efk tunnel
Status:
machine: efk
pid: 8721
route: 10.96.0.0/12 -> 192.168.99.113 # route status
minikube: Running
services: [elasticsearch, kibana]
errors:
minikube: no errors
router: no errors
loadbalancer emulator: no errors
Clean up based on info ~/.minikube/tunnels.json when shuts unexpected
minikube tunnel --cleanup
Avoid to typing password for minikube tunnel, by allowing a group or user to execute ip, route commands without asking for password. Example for Ubuntu 18.04 can be seen below.
$ sudo visudo ... # Members of the admin group may gain root privileges %admin ALL=(ALL) ALL %admin ALL=(ALL) NOPASSWD: /sbin/ip,/sbin/route # <- add this line in this order ...
References
- minikube Github
- CHANGELOG.md Github
- Networking.md Github