Difference between revisions of "Kubernetes/Kustomize"

From Ever changing code
Jump to navigation Jump to search
 
(30 intermediate revisions by the same user not shown)
Line 1: Line 1:
= [https://kustomize.io/ Kustomize] =
= [https://kustomize.io/ Kustomize] =
* [https://kubectl.docs.kubernetes.io/ kubectl+kustomize] SIG CLI
kustomize lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is.
kustomize lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is.
= Embedded versions =
;FluxCD (v2): runs kustomize-controller and helm-controller, use the following methods to find out versions of the embedded components:
* kustomize-controller versions [https://github.com/fluxcd/kustomize-controller/blob/main/CHANGELOG.md#100 CHANGELOG]:
** 0.27.0 - Kustomize v4.5.7
** 1.0.0 - Kustomize v5.0.3 (introduced in 1.0.0-rc.4 release)
** 1.2.0 - Kustomize v5.3.0, SOPS v3.8.1
* helm-controller versions [https://github.com/fluxcd/helm-controller/blob/main/CHANGELOG.md#0370 CHANGELOG]:
** v0.37.0 - helm v3.13.2, post-renderer kustomize v5.3.0
;ArgoCD
* v2.10.6+d504d2b
** kustomize: v5.2.1 2023-10-19T20:13:51Z
** Helm: v3.14.3+gf03cc04
** kubectl: v0.26.11


= Install =
= Install =
Line 8: Line 26:


# Install on Linux - option2
# Install on Linux - option2
VERSION=v3.8.7
VERSION=v4.1.2
VERSION=$(curl --silent "https://api.github.com/repos/kubernetes-sigs/kustomize/releases/latest" | jq -r .tag_name | cut -d"/" -f2); echo $VERSION
VERSION=$(curl --silent "https://api.github.com/repos/kubernetes-sigs/kustomize/releases" | jq -r '.[].tag_name | select(. | contains("kustomize"))' | sort | tail -1 | cut -d"/" -f2); echo $VERSION
curl -L https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2F${VERSION}/kustomize_${VERSION}_linux_amd64.tar.gz -o kustomize_${VERSION}_linux_amd64.tar.gz
curl -L https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2F${VERSION}/kustomize_${VERSION}_linux_amd64.tar.gz -o kustomize_${VERSION}_linux_amd64.tar.gz
tar xzvf kustomize_${VERSION}_linux_amd64.tar.gz
tar xzvf kustomize_${VERSION}_linux_amd64.tar.gz
sudo install ./kustomize /usr/local/bin/kustomize
sudo install ./kustomize /usr/local/bin/kustomize
sudo install ./kustomize /usr/local/bin/kustomize_${VERSION}
kustomize version --short
{kustomize/v4.1.2  2021-04-15T20:38:06Z  }


$ kustomize version --short
kustomize version
{kustomize/v3.8.7  2020-11-11T23:14:14Z  }
v5.3.0
</source>
</source>


= Kustomize build workflow =
= Kustomize build workflow =
* [https://github.com/kubernetes-sigs/kustomize/issues/2052 kustomize vars] - use <code>envsubst</code> instead
<source>$ kustomize build ~/target</source>
<source>$ kustomize build ~/target</source>
# load universal k8s object descriptions
# load universal k8s object descriptions
Line 27: Line 50:
# fix name references
# fix name references
# emit yaml
# emit yaml
= <code>kustomize.yaml</code>
A build stage first
* processes resources,
* then it processes generators, adding to the resource list under consideration,
* then it processes transformers to modify the list,
* and finally runs validators to check the list for whatever error.
<source lang=yaml>
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- {pathOrUrl}
- what resources you want to customize
# cross-cutting fields
namespace: custom
namePrefix: dev-
nameSuffix: "-svc"
commonLabels:
  app: web
commonAnnotations:
  value: app
generators:
- {pathOrUrl}
- what new resources should be created.
generatorOptions:
  disableNameSuffixHash: true
  labels:
    env: prod
  annotations:
    app: custom
transformers:
- {pathOrUrl}
- what to transform in above mentioned resources
validators:
- {pathOrUrl}
- ...
</source>
== Patches ==
;patchStrategicMerge: Kubernetes supports a customized version of JSON merge patch called strategic merge patch. This patch format is used by <code>kubectl apply</code>, <code>kubectl edit</code> and <code>kubectl patch</code>, and contains specialized directives to control how specific fields are merged.


= Example 101 =
= Example 101 =
{{Note|Bases have been deprecated in v2.1.0 [https://kubernetes-sigs.github.io/kustomize/blog/2019/06/18/v2.1.0/#resources-expanded-bases-deprecated resources-expanded-bases-deprecated]}}
* [https://kustomize.io/tutorial Kustomize builder] note that it operates on the 1st yaml document
{| class="wikitable"
{| class="wikitable"
|+ TableHeadline
|+ Example 101 - environment type overrides
|-  
|-  
! base/kustomization.yaml
! base/kustomization.yaml
Line 70: Line 143:
│   ├── kustomization.yaml
│   ├── kustomization.yaml
│   └── virtual-service.yaml
│   └── virtual-service.yaml
└── overlays
└── overlays # more contextual this directory could be called 'environments'
     ├── dev
     ├── dev
     │   ├── gateway_patch.yaml
     │   ├── gateway_patch.yaml
Line 98: Line 171:
So, what happens
So, what happens
<source lang=bash>
<source lang=bash>
# Applying patch, overlays/dev/gateway_patch.yaml  
# Applying the patch: overlays/dev/gateway_patch.yaml  
apiVersion: networking.istio.io/v1beta1
apiVersion: networking.istio.io/v1beta1
kind: Gateway
kind: Gateway
Line 111: Line 184:
     hosts:
     hosts:
     - sonarqube-dev.acme.com # <- override
     - sonarqube-dev.acme.com # <- override
# |  
#   |  
# | over the base
#   | over the base
# v  
#   v  
 
# base/gateway.yaml
# base/gateway.yaml
apiVersion: networking.istio.io/v1beta1
apiVersion: networking.istio.io/v1beta1
Line 131: Line 205:
       number: 443
       number: 443
       protocol: HTTP
       protocol: HTTP
# |  
#   |  
# | results with
#   | results with
# v  
#   v  
 
apiVersion: networking.istio.io/v1beta1
apiVersion: networking.istio.io/v1beta1
kind: Gateway
kind: Gateway
Line 161: Line 236:
:[[File:ClipCapIt-200910-010734.PNG]]
:[[File:ClipCapIt-200910-010734.PNG]]


= Examples =
= Cheatsheet =
* [https://github.com/kubernetes-sigs/kustomize/blob/master/examples/chart.md Helm charts - last mile]
* [https://github.com/kubernetes-sigs/kustomize/blob/master/examples/chart.md Helm charts - last mile]
= Patch [https://github.com/kubernetes-sigs/kustomize/blob/master/examples/patchMultipleObjects.md multiple objects] =
<syntaxhighlight lang="yaml">
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base
patches:
  - path: patch.json
    target:
      kind: PersistentVolume
      version: v1
      group: ""
      name: volume-(data|master)-\d # regex match
      labelSelector: |
        app.kubernetes.io/component=storage,
        app.kubernetes.io/name=elasticsearch
</syntaxhighlight>
Component example
<syntaxhighlight lang="yaml">
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
patches:
- target:
    kind: HelmRelease
    version: v2beta1
    group: helm.toolkit.fluxcd.io
    name: external-dns-.+$ # [1]
  patch: |-
    apiVersion: helm.toolkit.fluxcd.io/v2beta1
    kind: HelmRelease
    metadata:
      name: ALL            # [2]
      namespace: flux-system
    spec:
      values:
        tolerations:
          - key: "components.gke.io/gke-managed-components"
            operator: Exists
        affinity:
          nodeAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 100
                preference:
                  matchExpressions:
                  - key: "predictx/workload"
                    operator: In
                    values:
                    - "infra"
# [1] Regex match
# [2] The name is replaced by a name from the matched object. Any value ie 'ALL' is required.
</syntaxhighlight>
= Delete an object from the base =
''Strategic Merge Patch'' provides some patch options like replace, merge, and delete. The simple 'patch' can only patch (manipulate) it.
<syntaxhighlight lang="yaml">
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base
patchesStrategicMerge:
- |-
  apiVersion: v1
  kind: Namespace
  metadata:
    name: unwanted-namespace
  $patch: delete
</syntaxhighlight>
= secretGenerator =
Secrets can be generated from environment variables. Within a template file there is a list of variables, where the variable will become a key and it's value the value.
Environment variable secret template
<source lang=bash>
GIT_USERNAME
GIT_PASSWORD
GIT_CREDENTIALS
</source>
Kustomization
<source lang=yaml>
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
secretGenerator:
  - name: argocd-git-secret
    envs:
      - git.env
    options:
      disableNameSuffixHash: true
</source>
= Patch - add an item to a list =
* https://stackoverflow.com/questions/71622419/adding-items-to-a-list-with-kubectl-kustomize
* [https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/strategic-merge-patch.md#strategic-merge-patch Strategic merge patch docs]
In the standard JSON merge patch, JSON objects are always merged but lists are always replaced. Often that isn't what we want. To solve this problem, Strategic Merge Patch uses the go struct tag of the API objects to determine what lists should be merged and which ones should not. Read more at [https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/strategic-merge-patch.md#strategic-merge-patch strategic merge patch docs].
<source lang=yaml>
patchesJson6902:
  - patch: |-
      - op: add
        path: /spec/valuesFrom/-
        value: # below map will be added as an item to the list, pay attention to `-` sign at the end of path
          kind: ConfigMap
          name: values-1-yaml
    target:
      group: helm.toolkit.fluxcd.io
      kind: HelmRelease
      name: kube-prometheus-stack
      version: v2beta1
</source>
= [https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/replacements/ Replacements] =
* https://stackoverflow.com/questions/71358674/kustomize-how-to-reference-a-value-from-a-configmap-in-another-resource-overlay
Use of vars is getting deprecated please use replacements.
= Known issues =
= Known issues =
* [https://github.com/kubernetes-sigs/kustomize/issues/2034 commonLabels altering podSelector.matchLabels] and [https://github.com/kubernetes-sigs/kustomize/issues/157 Allow excluding some label selectors from commonLabels]
* [https://github.com/kubernetes-sigs/kustomize/issues/2034 commonLabels altering podSelector.matchLabels] and [https://github.com/kubernetes-sigs/kustomize/issues/157 Allow excluding some label selectors from commonLabels]
Line 186: Line 387:


= Resources =
= Resources =
* [https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/replacements/ Replacement transform is deprecating Vars]
* [https://github.com/kubernetes-sigs/kustomize Kustomize sig]
* [https://github.com/kubernetes-sigs/kustomize Kustomize sig]
* [https://kubectl.docs.kubernetes.io/guides/config_management/components/ v3.7.0+ Components]
* [https://github.com/kubernetes-sigs/kustomize/blob/master/docs/glossary.md#kustomization Glossary]
* [https://github.com/kubernetes-sigs/kustomize/blob/master/docs/glossary.md#kustomization Glossary]
* [https://github.com/kubernetes-sigs/kustomize/blob/master/docs/fields.md Kustomization File Fields]
* [https://kubectl.docs.kubernetes.io/references/kustomize/ Kustomization File Fields]
* [https://kubectl.docs.kubernetes.io/pages/examples/kustomize.html Kustomize - examples] kubectl.docs.kubernetes.io
* [https://kubectl.docs.kubernetes.io/pages/examples/kustomize.html Kustomize - examples] kubectl.docs.kubernetes.io
* [https://kubectl.docs.kubernetes.io/pages/app_composition_and_deployment/structure_directories.html Kustomize structure_directories]
* [https://kubectl.docs.kubernetes.io/pages/app_composition_and_deployment/structure_directories.html Kustomize structure_directories]
* [https://kubectl.docs.kubernetes.io/pages/reference/kustomize.html reference] Good!
* [https://kubectl.docs.kubernetes.io/pages/reference/kustomize.html reference] Good!
* [https://github.com/kubernetes-sigs/kustomize/blob/master/examples/inlinePatch.md inlinePatch]
* [https://github.com/kubernetes-sigs/kustomize/blob/master/examples/inlinePatch.md inlinePatch]
* [https://github.com/kubernetes-sigs/kustomize/blob/master/examples/chart.md kustomization of a helm chart]
* [https://github.com/kubernetes-sigs/kustomize/blob/master/examples/configureBuiltinPlugin.md#using-the-commonlabels-and-commonannotations-fields Customize Kustomize] annotation and label buildin transformer

Latest revision as of 23:33, 6 August 2024

Kustomize

kustomize lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is.

Embedded versions

FluxCD (v2)
runs kustomize-controller and helm-controller, use the following methods to find out versions of the embedded components:
  • kustomize-controller versions CHANGELOG:
    • 0.27.0 - Kustomize v4.5.7
    • 1.0.0 - Kustomize v5.0.3 (introduced in 1.0.0-rc.4 release)
    • 1.2.0 - Kustomize v5.3.0, SOPS v3.8.1
  • helm-controller versions CHANGELOG:
    • v0.37.0 - helm v3.13.2, post-renderer kustomize v5.3.0
ArgoCD
  • v2.10.6+d504d2b
    • kustomize: v5.2.1 2023-10-19T20:13:51Z
    • Helm: v3.14.3+gf03cc04
    • kubectl: v0.26.11

Install

# Detects your OS and downloads kustomize binary to cwd
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash

# Install on Linux - option2
VERSION=v4.1.2
VERSION=$(curl --silent "https://api.github.com/repos/kubernetes-sigs/kustomize/releases" | jq -r '.[].tag_name | select(. | contains("kustomize"))' | sort | tail -1 | cut -d"/" -f2); echo $VERSION
curl -L https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2F${VERSION}/kustomize_${VERSION}_linux_amd64.tar.gz -o kustomize_${VERSION}_linux_amd64.tar.gz
tar xzvf kustomize_${VERSION}_linux_amd64.tar.gz
sudo install ./kustomize /usr/local/bin/kustomize
sudo install ./kustomize /usr/local/bin/kustomize_${VERSION}

kustomize version --short
{kustomize/v4.1.2  2021-04-15T20:38:06Z  }

kustomize version
v5.3.0

Kustomize build workflow

$ kustomize build ~/target
  1. load universal k8s object descriptions
  2. read kustomization.yaml from target
  3. kustomize bases (recurse 2-5)
  4. load and/or generate resources
  5. apply target's kustomization operations
  6. fix name references
  7. emit yaml

= kustomize.yaml A build stage first

  • processes resources,
  • then it processes generators, adding to the resource list under consideration,
  • then it processes transformers to modify the list,
  • and finally runs validators to check the list for whatever error.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- {pathOrUrl}
- what resources you want to customize 

# cross-cutting fields
namespace: custom
namePrefix: dev-
nameSuffix: "-svc"
commonLabels:
  app: web
commonAnnotations:
  value: app

generators:
- {pathOrUrl}
- what new resources should be created.
generatorOptions:
  disableNameSuffixHash: true
  labels:
    env: prod
  annotations:
    app: custom

transformers:
- {pathOrUrl}
- what to transform in above mentioned resources

validators:
- {pathOrUrl}
- ...

Patches

patchStrategicMerge
Kubernetes supports a customized version of JSON merge patch called strategic merge patch. This patch format is used by kubectl apply, kubectl edit and kubectl patch, and contains specialized directives to control how specific fields are merged.

Example 101

Note: Bases have been deprecated in v2.1.0 resources-expanded-bases-deprecated


Example 101 - environment type overrides
base/kustomization.yaml overlays/dev/kustomization.yaml overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
commonLabels:
  app: sonarqube
resources:
- gateway.yaml
- virtual-service.yaml
apiVersion: ...
kind: Kustomization
patches:
- gateway_patch.yaml
- virtual-service_patch.yaml
resources:
- ../../base
apiVersion: ...
kind: Kustomization
patches:
- gateway_patch.yaml
- virtual-service_patch.yaml
resources:
- ../../base
.
├── base
│   ├── gateway.yaml
│   ├── kustomization.yaml
│   └── virtual-service.yaml
└── overlays # more contextual this directory could be called 'environments'
    ├── dev
    │   ├── gateway_patch.yaml
    │   ├── kustomization.yaml
    │   └── virtual-service_patch.yaml
    └── prod
        ├── gateway_patch.yaml
        ├── kustomization.yaml
        └── virtual-service_patch.yaml

# Build kuctomized output
kustomize version --short # -> {kustomize/v3.8.2  2020-08-29T17:44:01Z  }
kustomize build overlays/dev # apply patches
kustomize build base         # run common functions (as described in base/kustomize.yaml) against the whole code base


What happens?

  1. kustomize build overlays/dev finds kustomization.yaml, that describes:
    1. patches: [gateway_patch.yaml, virtual-service_patch.yaml] to be used over the base resources: [../../base]. There are 3 type of patches: patches, patchesStrategicMerge, patchesJson6902 to choose from
  2. overlays/dev/kustomization.yaml cascades to the base (source of manifests to be changed) via directive resources: ["../../base"]
  3. The base directory contains and runs its own kustomization.yaml file.
  4. The base/kustomization.yaml contains common operations, eg. commonLabels, namePrefix functions to be applied to whole code base.
  5. Then patch file(s) are applied eg. gateway_patch.yaml contains enough information to identify a resource/object and apply changes.


So, what happens

# Applying the patch: overlays/dev/gateway_patch.yaml 
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: sonarqube 
spec:
  servers:
  - port:
      number: 443
      name: http
      protocol: HTTP
    hosts:
     - sonarqube-dev.acme.com # <- override
#   | 
#   | over the base
#   v 

# base/gateway.yaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  labels:
    app: sonarqube
  name: sonarqube
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - sonarqube.acme.com
    port:
      name: http
      number: 443
      protocol: HTTP
#   | 
#   | results with
#   v 

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  labels:
    app: sonarqube
    owner: piotr # <- label added by base kustomize.yaml fn
  name: sonarqube
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - sonarqube-dev.acme.com # <- patch override
    port:
      name: http
      number: 443
      protocol: HTTP


Check yourselves

#         __unchanged manifest_    _base kustomization_    ___patch overlay____________
vimdiff <(cat base/gateway.yaml) <(kustomize build base) <(kustomize build overlays/dev)
ClipCapIt-200910-010734.PNG

Cheatsheet

Patch multiple objects

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base

patches:
  - path: patch.json
    target:
      kind: PersistentVolume
      version: v1
      group: ""
      name: volume-(data|master)-\d # regex match
      labelSelector: |
        app.kubernetes.io/component=storage,
        app.kubernetes.io/name=elasticsearch


Component example

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

patches:
- target: 
    kind: HelmRelease
    version: v2beta1
    group: helm.toolkit.fluxcd.io
    name: external-dns-.+$ # [1]
  patch: |-
    apiVersion: helm.toolkit.fluxcd.io/v2beta1
    kind: HelmRelease
    metadata:
      name: ALL            # [2]
      namespace: flux-system
    spec:
      values:
        tolerations:
          - key: "components.gke.io/gke-managed-components"
            operator: Exists
        affinity:
          nodeAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 100
                preference:
                  matchExpressions:
                  - key: "predictx/workload"
                    operator: In
                    values:
                    - "infra"

# [1] Regex match
# [2] The name is replaced by a name from the matched object. Any value ie 'ALL' is required.

Delete an object from the base

Strategic Merge Patch provides some patch options like replace, merge, and delete. The simple 'patch' can only patch (manipulate) it.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base

patchesStrategicMerge:
- |-
  apiVersion: v1
  kind: Namespace
  metadata:
    name: unwanted-namespace
  $patch: delete

secretGenerator

Secrets can be generated from environment variables. Within a template file there is a list of variables, where the variable will become a key and it's value the value.


Environment variable secret template

GIT_USERNAME
GIT_PASSWORD
GIT_CREDENTIALS


Kustomization

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

secretGenerator:
  - name: argocd-git-secret
    envs:
      - git.env
    options:
      disableNameSuffixHash: true

Patch - add an item to a list

In the standard JSON merge patch, JSON objects are always merged but lists are always replaced. Often that isn't what we want. To solve this problem, Strategic Merge Patch uses the go struct tag of the API objects to determine what lists should be merged and which ones should not. Read more at strategic merge patch docs.


patchesJson6902:
  - patch: |-
      - op: add
        path: /spec/valuesFrom/-
        value: # below map will be added as an item to the list, pay attention to `-` sign at the end of path
          kind: ConfigMap
          name: values-1-yaml
    target:
      group: helm.toolkit.fluxcd.io
      kind: HelmRelease
      name: kube-prometheus-stack
      version: v2beta1

Replacements

Use of vars is getting deprecated please use replacements.

Known issues

In some settings it makes sense for commonLabels to be included in selectors, and in some settings it doers not make sense to include them in selectors. Kustomize includes by default, and there is no way to opt out. As workaround, you can convert matchLabels to matchExpressions and Kustomize won't touch them. API docs

- podSelector:
      matchLabels:
        app: mongodb-backup


is equivalent with

- podSelector:
      matchExpressions:
        - key: app
        operator: In
        values:
          - mongodb-backup

and Kustomize will keep its hands off.

Resources