Difference between revisions of "Kubernetes/Helm"
(3 intermediate revisions by the same user not shown) | |||
Line 11: | Line 11: | ||
tar xzvf helm-${LATEST}-linux-amd64.tar.gz | tar xzvf helm-${LATEST}-linux-amd64.tar.gz | ||
sudo install linux-amd64/helm /usr/local/bin/helm | sudo install linux-amd64/helm /usr/local/bin/helm | ||
helm version | |||
</source> | </source> | ||
Line 68: | Line 69: | ||
<source lang=bash> | <source lang=bash> | ||
# | # 'Helm plugin' install | ||
## Option 1. | |||
helm plugin install https://github.com/databus23/helm-diff --version master # or tag ie v3.8.1 | |||
## Option 2. | |||
VERSION=$(curl --silent "https://api.github.com/repos/databus23/helm-diff/releases/latest" | jq -r .tag_name); echo $VERSION | VERSION=$(curl --silent "https://api.github.com/repos/databus23/helm-diff/releases/latest" | jq -r .tag_name); echo $VERSION | ||
helm plugin install https://github.com/databus23/helm-diff/releases/download/$VERSION/helm-diff-linux.tgz | helm plugin install https://github.com/databus23/helm-diff/releases/download/$VERSION/helm-diff-linux-amd64.tgz | ||
## Update with helm plugin | |||
helm plugin list | |||
helm plugin update diff | |||
# Install using tarball, simply extract to '$HELM_PLUGINS' | # Install using tarball, simply extract to '$HELM_PLUGINS' | ||
eval $(helm env | grep PLUGINS) # eg. -> HELM_PLUGINS="/home/vagrant/.local/share/helm/plugins" | eval $(helm env | grep PLUGINS) # eg. -> HELM_PLUGINS="/home/vagrant/.local/share/helm/plugins" | ||
VERSION=$(curl --silent "https://api.github.com/repos/databus23/helm-diff/releases/latest" | jq -r .tag_name); echo $VERSION | VERSION=$(curl --silent "https://api.github.com/repos/databus23/helm-diff/releases/latest" | jq -r .tag_name); echo $VERSION | ||
wget https://github.com/databus23/helm-diff/releases/download/$VERSION/helm-diff-linux.tgz | wget https://github.com/databus23/helm-diff/releases/download/$VERSION/helm-diff-linux-amd64.tgz | ||
tar -xzf helm-diff-linux.tgz -C $HELM_PLUGINS | tar -xzf helm-diff-linux-amd64.tgz -C $HELM_PLUGINS | ||
# Usage | # Usage |
Latest revision as of 15:45, 24 November 2023
Note: This content refers to helm3 as authored in April, 2020.
Install Helm
Note: Helm2 development is still active so, the latest tag may point actually to helm2 instead to helm3
# latest version of helm2 LATEST=$(curl --silent "https://api.github.com/repos/helm/helm/releases/latest" | jq -r .tag_name); echo $LATEST # latest version of helm3 LATEST=$(curl --silent "https://api.github.com/repos/helm/helm/releases" | jq -r .[].tag_name | sort -V | grep -v rc | tail -n -1); echo $LATEST curl -LO https://get.helm.sh/helm-${LATEST}-linux-amd64.tar.gz tar xzvf helm-${LATEST}-linux-amd64.tar.gz sudo install linux-amd64/helm /usr/local/bin/helm helm version
Helm-docs
REPO=norwoodj/helm-docs LATEST=$(curl --silent "https://api.github.com/repos/${REPO}/releases" | jq -r .[].tag_name | sort -V | grep -v rc | tail -n -1 | tr -d 'v'); echo $LATEST curl -LO https://github.com/norwoodj/helm-docs/releases/download/v${LATEST}/helm-docs_${LATEST}_Linux_x86_64.tar.gz tar xzvf helm-docs_${LATEST}_Linux_x86_64.tar.gz sudo install helm-docs /usr/local/bin/
Note: For bravehearts check this out https://github.com/roboll/helmfile, declarative way to install helm charts
Charts repository
- Official Helm3
- GitHub Helm2 [Depricated]
Example below refers how to install metrics-server
# Search repo $ helm search hub metrics-server # searches 'hub' https://hub.helm.sh/ URL CHART VERSION APP VERSION DESCRIPTION https://hub.helm.sh/charts/bitnami/metrics-server 4.2.0 0.3.7 Metrics Server... https://hub.helm.sh/charts/stable/metrics-server 2.11.1 0.3.6 Metrics Server... # Add repo $ helm repo add bitnami https://charts.bitnami.com/bitnami $ helm repo list NAME URL gitlab https://charts.gitlab.io bitnami https://charts.bitnami.com/bitnami $ helm search repo bitnami/metrics NAME CHART VERSION APP VERSION DESCRIPTION bitnami/metrics-server 4.2.0 0.3.7 Metrics Server is ... # Install helm install metrics-server bitnami/metrics-server -n metrics-server --dry-run # my-release is 'metrics-server' # install 'metrics.k8s.io/v1beta1' k get apiservices.apiregistration.k8s.io | grep metrics # if return nothing do below helm upgrade --install metrics-server bitnami/metrics-server -n metrics-server --set apiService.create=true # Test k get apiservices.apiregistration.k8s.io | grep metrics NAME SERVICE AVAILABLE AGE v1beta1.metrics.k8s.io metrics-server/metrics-server True 32s kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" k top nodes
Diff plugin
Limitations and caveats:
- since helm3, 3-way merge patch strategy is used, this is not reflected in helm diff that was created in the helm2 times, this means any manual changes to objects created by a chart won't be detected, but any chart changes will
- Show diff for actual state vs desired state Issue-176
# 'Helm plugin' install ## Option 1. helm plugin install https://github.com/databus23/helm-diff --version master # or tag ie v3.8.1 ## Option 2. VERSION=$(curl --silent "https://api.github.com/repos/databus23/helm-diff/releases/latest" | jq -r .tag_name); echo $VERSION helm plugin install https://github.com/databus23/helm-diff/releases/download/$VERSION/helm-diff-linux-amd64.tgz ## Update with helm plugin helm plugin list helm plugin update diff # Install using tarball, simply extract to '$HELM_PLUGINS' eval $(helm env | grep PLUGINS) # eg. -> HELM_PLUGINS="/home/vagrant/.local/share/helm/plugins" VERSION=$(curl --silent "https://api.github.com/repos/databus23/helm-diff/releases/latest" | jq -r .tag_name); echo $VERSION wget https://github.com/databus23/helm-diff/releases/download/$VERSION/helm-diff-linux-amd64.tgz tar -xzf helm-diff-linux-amd64.tgz -C $HELM_PLUGINS # Usage $> helm diff [command] Available Commands: release Shows diff between release manifests revision Shows diff between revision manifests rollback Show a diff explaining what a helm rollback could perform upgrade Show a diff explaining what a helm upgrade would change. version Show version of the helm diff plugin # Examples $> helm diff revision [flags] RELEASE REVISION1 REVISION2 --context 5 # --context output NUM lines of context around changes (default -1) $> helm diff upgrade RELEASE . -C 5 $> helm diff rollback RELEASE REVISION1 -C 5 # Compare the manifests details of a different releases created from the same chart $> helm diff release RELEASE1 RELEASE2 -C 5
Operations
Helm command
helm version --short # --> v3.1.3+g0a9a9a8 helm [command] -h # get help create|env|get|history|install|lint|list|plugin|rollback|show|status|template|uninstall|upgrade # Flags: # --kube-context string -name of the kubeconfig context to use # --kubeconfig string -path to the kubeconfig file # -n, --namespace string
Operations
# Add repo helm repo add stable https://kubernetes-charts.storage.googleapis.com/ helm repo update # Make sure we get the latest list of charts helm search repo stable/jenkins # Set kubectl context # Helm3 bug:does may not respect '--namespace' flag, so set default namespace kubectl create ns jenkins kubectl config set-context $(kubectl config current-context) --namespace=jenkins # Install release helm install [NAME] [CHART] [flags] helm install jenkins-ci stable/jenkins # release name: jenkins-ci helm install jenkins-ci ./jenkins-2.192.1.tgz --set service.type=NodePort # install from a tarball package helm install stable/mysql --generate-name # release name will be generated # Get information about releases helm ls -A # show a list of all deployed releases -A --all-namespaces NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION jenkins-ci jenkins 1 2020-04-25 15:.. deployed jenkins-1.17.2 lts helm get all -n jenkins jenkins-ci # all|hooks|manifest|notes|values(user defined values) # Uninstall release helm uninstall jenkins-ci --namespace jenkins
- Rollback
$> helm rollback <RELEASE> [REVISION] [flags] $> helm rollback diy 2 --dry-run # <RELEASE> = diy $> time helm rollback diy 2 --wait # --wait until all Pods, PVCs... are in a ready state before marking the release as successful. # It will wait for as long as --timeout # Note: Each rollback increments REVISION with +1 $> helm history diy REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Mon Apr 27 10:39:38 2020 superseded diy-0.1.0 6.4.2 Install complete 2 Mon Apr 27 12:24:54 2020 superseded diy-0.1.0 6.4.2 Upgrade complete 3 Mon Apr 27 16:22:34 2020 superseded diy-0.1.0 6.4.2 Upgrade complete 4 Mon Apr 27 16:35:31 2020 deployed diy-0.1.0 6.4.2 Rollback to 2
- Jenkins chart post install instructions or
get notes
commands
helm get notes -n jenkins jenkins-ci
NAME: jenkins-ci LAST DEPLOYED: Sat Apr 25 15:33:36 2020 NAMESPACE: jenkins STATUS: deployed REVISION: 1 NOTES: 1. Get your 'admin' user password by running: printf $(kubectl get secret --namespace jenkins jenkins-ci -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo 2. Get the Jenkins URL to visit by running these commands in the same shell: export POD_NAME=$(kubectl get pods --namespace jenkins -l "app.kubernetes.io/component=jenkins-master" -l "app.kubernetes.io/instance=jenkins-ci" -o jsonpath="{.items[0].metadata.name}") echo http://127.0.0.1:8080 kubectl --namespace jenkins port-forward $POD_NAME 8080:8080 3. Login with the password from step 1 and the username: admin For more information on running Jenkins on Kubernetes, visit: https://cloud.google.com/solutions/jenkins-on-container-engine
Subcharts
Subcharts and overriding variables
Subchart is a chart that lives in charts/
directory. It gets templated each time with the parent chart but can also be tested and deployed separately.
- A subchart is considered "stand-alone", which means a subchart can never explicitly depend on its parent chart.
- For that reason, a subchart cannot access the values of its parent.
- A parent chart can override values for subcharts.
- Helm has a concept of global values that can be accessed by all charts.
- Overrides
If subchart has a variable variable1
this can be overriden from parent chart like below
chartname: variable1: foo
Dependencies
Dependencies in Helm3 are driven by dependencies:
block in Chart.yaml
file.
dependencies: - name: nfs-server version: "0.1.0" condition: deployNfsServer.enabled # (optional) A yaml path that resolves to a boolean, used for enabling/disabling charts (e.g. subchart1.enabled ) repository: "file://../nfs-server" # if 'repository' is unspecified assumes it exists in the charts directory enabled: true
At first we need to build the dependency package snapshot that will be used to build application. The snapshot will be packaged and placed in charts/
directory. Each time we do helm dependency update
package get updated with any changes done to the referenced chart.
# Update and build dependencies helm dependency update # creates Chart.lock file helm dependency build # builds a package and places in charts/ directory eg. nfs-server-0.1.0.tgz # --- step 1 - Chart.lock cat Chart.lock dependencies: - name: nfs-server repository: file://../nfs-server version: 0.1.0 digest: sha256:e04f412832d56d97cfd153d2b2dd9eb5d6dad2c0fef65d3f92293672a2a6a043 generated: "2020-05-20T17:01:48.039203425Z" # --- step 2 - package the dependency chart # | $ tree mft/charts/ # | mft/charts/ # | └── nfs-server-0.1.0.tgz
Now on each install the dependent chart will be installed along with the main chart. All manifests are applied in order:
- aggregrated into a single set; then
- sorted by type followed by name; and then
- created/updated in that order.
helm upgrade --install mychart ./mychart --dry-run
Create a chart aka init
Note: Check The chart file structure as the helm create NAME
does not include all optional files you may use
helm version --short # -> v3.1.3+g0a9a9a8 helm create diy # diy = [NAME] # this creates standard file tree not all optional files are included $> tree diy diy ├── charts # any charts upon which this chart depends ├── Chart.yaml ├── templates # no rigid naming pattern, recommended .yaml for YAML files and .tpl for helper | | # all files here will be sent through the template engine │ ├── deployment.yaml │ ├── _helpers.tpl # template helpers that you can re-use throughout the chart │ ├── ingress.yaml │ ├── NOTES.txt # Optional. “help text” for your chart, displayed after installation │ ├── serviceaccount.yaml │ ├── service.yaml │ └── tests │ └── test-connection.yaml └── values.yaml # default configuration values for this chart helm template diy | bat -l yaml --style=plain # display rendered yaml manifests kubectl create ns diy helm install ./diy --generate-name -n diy # Generated name # ____NAME______ # podName : pod/diy-1587926286-866d97bb75-vskns # serviceName : service/diy-1587926286 # deploymentName: deployment.extensions/diy-1587926286 # NAME TYPE # secret/sh.helm.release.v1.diy-1587926286.v1 helm.sh/release.v1 helm install foo ./diy -n diy # Custom name # _NAME__ # podName : pod/foo-diy-6fd79d9bbd-nh5hl # serviceName : service/foo-diy # deploymentName: deployment.extensions/foo-diy # NAME TYPE # secret/sh.helm.release.v1.foo.v1 helm.sh/release.v1 helm list -n diy NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION diy-1587926286 diy 1 2020-04-26 19:38... deployed diy-0.1.0 1.16.0 foo diy 1 2020-04-26 20:02... deployed diy-0.1.0 1.16.0
Template
Everything in templates/
goes through templating engine before being rendered. Basic template might look like below:
# TEMPLATE ---> RENDER ---> DEPLOYED # # helm install mychart ./mychart -n mychart # cat templates/configmap.yaml # helm template mychart/ # kubectl -n mychart get cm mychart-configmap -oyaml | apiVersion: v1 | apiVersion: v1 | apiVersion: v1 | kind: ConfigMap | kind: ConfigMap | kind: ConfigMap | metadata: -> | metadata: --> | metadata: | name: {{ .Release.Name }}-configmap | name: RELEASE-NAME-configmap | name: mychart-configmap | data: | data: | data: | myvalue: "Hello World Piotrek" | myvalue: "Hello World Piotrek" | myvalue: Hello World Piotrek
Note: In kubectl get cm
above creationTimestamp:
and selfLink:
data yaml keys have been not included for clarity of side-by-side comparison.
- References
- Buildin objects helm.sh
- Best Practicies helm.sh
- Math Functions gohugo.io
- Go Template package golang.org
- Good concepts, advanced usage
Text and spaces
Note: Details about whitespace control you can find in Flow Control in Controlling Whitespace section.
If an action's left delimiter (by default "{{") is followed immediately by a minus sign and ASCII space character ("{{- "), all trailing white space is trimmed from the immediately preceding text. Similarly, if the right delimiter ("}}") is preceded by a space and minus sign (" -}}"), all leading white space is trimmed from the immediately following text. In these trim markers, the ASCII space must be present; "{{-3}}" parses as an action containing the number -3.
For instance, when executing the template whose source is
"{{23 -}} < {{- 45}}" // the generated output would be "23<45"
For this trimming, the definition of white space characters is the same as in Go: space, horizontal tab, carriage return, and newline.
Flow Control
- Flow Control Medium
Comments
# Inline, there is no spaces in between the `double-curly-brackets` and `forward-slash` VariableFoo: {{ .Values.VariableFoo | quote }}{{/* FooComment __1 */}} # Multiline_1 {{- /* data: {{- $files := .Files }} {{- range tuple "files/sso.yaml" }} {{ . }}: |- {{ $files.Get . }} {{- end }} */ -}} # Multiline_2, note this can cause some rendering issues sometimes {{- /* stringData: - not base64 encoded data: - base64 encoded, eg 'extra: YmFyCg==' */ -}}
Check if API Resource Exists
Using Capabilities.APIVerions.Has will allow you to check for the existence of a specific API version or resource.
{{- if .Capabilities.APIVersions.Has "networking.k8s.io/v1beta1/Ingress" }} apiVersion: networking.k8s.io/v1beta1 kind: Ingress ... {{- end }}
Package and push charts to OCI registry
- push-helm-charts-to-amazon-ecr
- Giving your Charts a Home in Docker Registry
- [ECR: Allow for alternate mediaTypes] AWS ECR
Known workarounds
Invalid spec selector after upgrading helm template
This is caused by dynamic labels in selector
as of Kubernetes version X field is immutable.
- Invalid spec selector after upgrading helm template StackOverflow
Steps to before upgrade without need of uninstalling charts:
$ kubectl delete statefulsets.apps --cascade=false my-release # Edit the deployment: kubectl patch deployments my-release --type=json -p='[{"op": "remove", "path": "/spec/selector/matchLabels/chart"}]'
Variables and Spring functions
Variables are normally not "global". They are scoped to the block in which they are declared.
{{- range $key, $val := .Values.favorite }} {{ $key }}: {{ $val | quote }} {{- end }}
variables $key
and $val
will only be in scope inside of the range
block.
$
- variable that is always global, points to the root context. It's useful eg. when you are looping in a range and you need to know the chart's release name.
{{- range .Values.tlsSecrets }} apiVersion: v1 kind: Secret metadata: name: {{ .name }} labels: # Many helm templates would use `.` below, but that will not work, # however `$` will work here app.kubernetes.io/name: {{ template "fullname" $ }} # I cannot reference .Chart.Name, but I can do $.Chart.Name helm.sh/chart: "{{ $.Chart.Name }}-{{ $.Chart.Version }}" ... {{- end }}
Templates in helm cannot access variables. However, the context handed to the template is a dictionary. The spring library is accessible from within the Go templates and can be used.
Use function "set" to add a new key/value pair to a dictionary.
$_ := set $myDict "name4" "value4"
Note that set returns the dictionary (a requirement of Go template functions), so you may need to trap the value as done above with the $_
assignment.