Actions

Kubernetes/Security

From Ever changing code

< Kubernetes

API Server and Role Base Access Control

Once the API server has determined who you are (whether a pod or a user), the authorization is handled by RBAC.


To prevent unauthorized users from modifying the cluster state, RBAC is used by defining roles and role bindings for a user. A service account resource is created for a pod to determine what control has over the cluster state. For example, the default service account will not allow you to list the services in a namespace.


The Kubernetes API server provides CRUD actions (Create, Read, Update, Delete) interface for interacting with cluster state over a RESTful API. API calls can come only from 2 sources:

  • kubectl
  • POD

There is 4 stage process

  1. Authentication
  2. Authorization
  3. Admission
  4. Writing the configuration state CRUD actions to persistent store etcd database
ClipCapIt-190706-211859.PNG

Example plugins:

  • serviceaccount plugin applies default serviceaccount to pods that don't explicitly specify


RBAC is managed by 4 resources, divided over 2 groups

RBAC resources
Group-1 namespace resources Group-2 cluster level resources resources type
roles cluster roles defines what can be done
role bindings cluster role bindings defines who can do it


When deploying a pod a default serviceaccount is assigned if not specified in the pod manifest. The serviceaccount represents an identity of an app running on a pod. Token file holds authentication token. Let's create a namespace and create a test pod to try to list available services.

kubectl create ns rbac
kubectl run apitest --image=nginx -n rbac #create test container, to run API call test from


Each pod has serviceaccount, the API authentication token is on a pod. When a pod makes API call uses the token, this allows to assumes the serviceaccount, so it gets identity. You can preview the token on the pod.

kubectl -n rbac1 exec -it apitest-<UID> -- /bin/sh  #connect to the container shell

#display token and namespace that allows to connect to API server from this pod
root$ cat /var/run/secrets/kubernetes.io/serviceaccount/{token,namespace} 

#call API server to list K8s services in 'rbac' namespace
root$ curl localhost:8001/api/v1/namespaces/rbac/services


List all serviceaccounts. Serviceaccounts can only be used within the same namespace.

kubectl get serviceaccounts -n rbac
kubectl get secrets
NAME                  TYPE                                  DATA   AGE
default-token-qqzc7   kubernetes.io/service-account-token   3      39h
kubectl get secrets default-token-qqzc7 -o yaml #display secrets

ServiceAccount

The API server is first evaluating if the request is coming from a service account or a normal user /or normal user account meeting, a private key, a user store or even a file with a list of user names and passwords. Kubernetes doesn't have objects that represent normal user accounts, and normal users cannot be added to the cluster through.

kubectl get    serviceaccounts #or 'sa' in short
kubectl create serviceaccount  jenkins

kubectl get    serviceaccounts jenkins -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: "2019-08-05T07:10:40Z"
  name: jenkins
  namespace: default
  resourceVersion: "678"
  selfLink: /api/v1/namespaces/default/serviceaccounts/jenkins
  uid: 21cba4bb-b750-11e9-86b3-0800274143a9
secrets:
- name: jenkins-token-cspjm

kubectl get secret [secret_name]


Assign ServiceAccoubt to a pod

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  serviceAccountName: jenkins #<-- ServiceAccount
  containers:
  - image: busybox:1.28.4
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox
  restartPolicy: Always

#Verify
kubectl.exe get pods -o yaml | sls serviceAccount

        {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"busybox","namespace":"default"},"spec":{"c
ontainers":[{"command":["sleep","3600"],"image":"busybox:1.28.4","imagePullPolicy":"IfNotPresent","name":"busybox"}],"r
estartPolicy":"Always","serviceAccountName":"jenkins"}}
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    serviceAccount: jenkins
    serviceAccountName: jenkins

Create Administrative account

This is a process of setting up a new remote administrator.

kubectl.exe config set-credentials piotr --username=piotr --password=password

#new section in ~/.kube/config has been added:
users:
- name: user1
...
- name: piotr
  user:
    password: password
    username: piotr

#create clusterrolebinding, this is for authonomus users not-recommended
kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous
clusterrolebinding.rbac.authorization.k8s.io/cluster-system-anonymous created

#copy server ca.crt
laptop$ scp ubuntu@k8s-cluster.acme.com:/etc/kubernetes/pki/ca.crt .

#set kubeconfig 
kubectl config set-cluster kubernetes --server=https://k8s-cluster.acme.com:6443 --certificate-authority=ca.crt --embed-certs=true

#Create context
kubectl config set-context kubernetes --cluster=kubernetes --user=piotr --namespace=default

#Use contect to current
kubectl config use-context kubernetes

Create a role (namespaced permissions)

The role describes what actions can be performed. This role allows to list services from a web namespace.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: web #this need to be created beforehand
  name: service-reader
rules:
- apiGroups: [""]
  verbs: ["get", "list"]
  resources: ["services"]


The role does not specify who can do it. Thus we create a roleBinding with a user, serviceAccount or group. The roleBinding can only reference a single role, but can bind to multi: users, serviceAccounts or groups

kubectl create rolebinding roleBinding-test --role=service-reader --serviceaccount=web:default -n web

# Verify access has been granted
curl localhost:8001/api/v1/namespaces/web/services

Create a clusterrole (cluster-wide permissions)

In this example we create ClusterRole that can access persitenvolumes APIs, then we will create ClusterRolebinding (pv-test) with a default ServiceAccount (name: default) in 'web' namespace. The SA is a account that pod assumes/uses by default when getting Authenticated by API-server. When we then attach to the container and try to list cluster-wide resource - persitenvolumes , this will be allowed because of ClusterRole, that the pod has assumed.

# Create a ClusterRole to access PersistentVolumes:
kubectl create clusterrole pv-reader --verb=get,list --resource=persistentvolumes

# Create a ClusterRoleBinding for the cluster role:
kubectl create clusterrolebinding pv-test --clusterrole=pv-reader --serviceaccount=web:default


The YAML for a pod that includes a curl and proxy container:

apiVersion: v1
kind: Pod
metadata:
  name: curlpod
  namespace: web
spec:
  containers:
  - image: tutum/curl
    command: ["sleep", "9999999"]
    name: main
  - image: linuxacademycontent/kubectl-proxy
    name: proxy
  restartPolicy: Always


Create the pod that will allow you to curl directly from the container:

kubectl apply -f curlpod.yaml
kubectl get pods -n web               # Get the pods in the web namespace
kubectl exec -it curlpod -n web -- sh # Open a shell to the container:

#Verify you can access PersistentVolumes (cluster-level) from the pod
/ # curl localhost:8001/api/v1/persistentvolumes
{
  "kind": "PersistentVolumeList",
  "apiVersion": "v1",
  "metadata": {
    "selfLink": "/api/v1/persistentvolumes",
    "resourceVersion": "7173"
  },
  "items": []
}/ #

List all API resources

PS C:> kubectl.exe proxy
Starting to serve on 127.0.0.1:8001
List all API resources
Text Web browser
PS C:> curl http://localhost:8001

StatusCode        : 200
StatusDescription : OK
Content           : {
                      "paths": [
                        "/api",
                        "/api/v1",
                        "/apis",
                        "/apis/",
                        "/apis/admissionregistration.k8s.io",
                        "/apis/admissionregistration.k8s.io/v1beta1",
                        "/apis/apiextensions.k8s.io",
                        "...
RawContent        : HTTP/1.1 200 OK
                    Content-Length: 2738
                    Content-Type: application/json
                    Date: Wed, 14 Aug 2019 07:21:12 GMT

                    {
                      "paths": [
                        "/api",
                        "/api/v1",
                        "/apis",
                        "/apis/",
                        "/apis/admissionr...
Forms             : {}
Headers           : {[Content-Length, 2738], [Content-Type, application/json],        
                    [Date, Wed, 14 Aug 2019 07:21:12 GMT]}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        : System.__ComObject
RawContentLength  : 2738
 :ClipCapIt-190813-082554.PNG

Network policies

Network policies allow you to specify which pods can talk to other pods. The example Calico's plugin allows for securing communication by:

  • applying network policy based on:
    • pod label-selector
    • namespace label-selector
    • CIDR block range
  • securing communication (who can access pods) by setting up:
    • ingress rules
    • egress rules


POSTing any NetworkPolicy manifest to the API server will have no effect unless your chosen networking solution supports network policy. Network Policy is just an API resource that defines a set of rules for Pod access. However, to enable a network policy, we need a network plugin that supports it. We have a few options:

  • Calico, Cilium, Kube-router, Romana, Weave Net

minikube

If you plan to use Minikube with its default settings, the NetworkPolicy resources will have no effect due to the absence of a network plugin and you’ll have to start it with --network-plugin=cni.

minikube start --network-plugin=cni --memory=4096

Install Calico network policies

What is the Canal? Tigera and CoreOS’s was a project to integrate Calico and flannel, read more...


Install Calico =<v3.5 canal network policies plugin:

wget -O canal.yaml https://docs.projectcalico.org/v3.5/getting-started/kubernetes/installation/hosted/canal/canal.yaml
curl https://docs.projectcalico.org/v3.8/manifests/canal.yaml -O
curl https://docs.projectcalico.org/v3.8/manifests/calico-policy-only.yaml -O 

# Update Pod IPs to '--cluster-cidr', changing this value after installation has no affect
## Get Pod's cidr
kubectl cluster-info dump | grep -m 1 service-cluster-ip-range
kubectl cluster-info dump | grep -m 1 cluster-cidr

## Minikube cidr can change, so exec to Pod is best option eventually you can check in minikube config
grep ~/.minikube/profiles/minikube/config.json | grep ServiceCIDR

## Update config
POD_CIDR="<your-pod-cidr>" sed -i -e "s?10.244.0.0/16?$POD_CIDR?g" canal.yaml

kubectl apply -f canal.yaml

Cilium - networkPolicies

Cilium DaemonSet will place one Pod per node. Each Pod then will enforce network policies on the traffic using Berkeley Packet Filter (BPF).

minikube start --network-plugin=cni --memory=4096 #--kubernetes-version=1.13
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.5/examples/kubernetes/1.14/cilium-minikube.yaml


Create NetworkPolicy

NetworkPolicy describes what network traffic is allowed for a set of Pods, if Pods are selected but there is no rules then traffic is denied.


Create a 'default' isolation policy for a namespace by creating a NetworkPolicy that selects all pods but does not allow any ingress traffic to those pods. The example we will run in dev namespace.


Create a namespace

kubectl create ns dev
kubectl explain networkpolicy.spec.ingress #get yaml schema fields


Default policies manifests
Default deny all ingress traffic Default allow all ingress traffic
cat > deny-all-ingress.yaml << EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
# namespace: dev
spec:
  podSelector: {} # select all pods in a ns
  ingress:        # allowing rules,
                  # if empty no traffic allowed
  policyTypes:    
  - Ingress 
EOF
cat > allow-all-ingress.yaml << EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-ingress
# namespace: dev
spec:
  podSelector: {}
  ingress:  # rules to apply to selected nodes
  - {}      # all allowed rule
  policyTypes:
  - Ingress # rule types that the NetworkPolicy relates to
EOF


Create NetworkPolicy

$ kubectl apply -f deny-all-ingress.yaml
networkpolicy.networking.k8s.io/deny-all-ingress created

$ kubectl get networkPolicy -A
NAMESPACE   NAME               POD-SELECTOR   AGE
default     deny-all-ingress   <none>         6s

$ kubectl describe networkPolicy deny-all-ingress
Name:         deny-all-ingress
Namespace:    dev
Created on:   2019-08-20 22:31:30 +0100 BST
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"networking.k8s.io/v1","kind":"NetworkPolicy","metadata":{"annotations":{},"name":"deny-all-ingress","namespace":"dev"},"spe...
Spec:
  PodSelector:     <none> (Allowing the specific traffic to all pods in this namespace)
  Allowing ingress traffic:
    <none> (Selected pods are isolated for ingress connectivity)
  Allowing egress traffic:
    <none> (Selected pods are isolated for egress connectivity)
  Policy Types: Ingress


Run test pod

# Single Pods in -n --namespace
kubectl -n dev run --generator=run-pod/v1 busybox1 --image=busybox -- sleep 3600
kubectl -n dev run --generator=run-pod/v1 busybox2 --image=busybox -- sleep 3600
# default labels: --labels="run=busybox1"

kubectl -n dev run --generator=run-pod/v1 busybox3 --image=busybox --labels="app=A" -- sleep 3600
kubectl -n dev run --generator=run-pod/v1 busybox4 --image=busybox --labels="app=B" -- sleep 3600

# Get pods with labels
kubectl -n dev get pods -owide --show-labels
NAMESPACE NAME          READY   STATUS    RESTARTS   AGE   IP              NODE       NOMINATED NODE READINESS GATES LABELS
dev       pod/busybox1  1/1     Running   0          34m   10.15.185.121   minikube   <none>         <none>          run=busybox1
dev       pod/busybox2  1/1     Running   0          34m   10.15.171.50    minikube   <none>         <none>          run=busybox2
dev       pod/busybox3  1/1     Running   0          31m   10.15.140.94    minikube   <none>         <none>          app=A
dev       pod/busybox4  1/1     Running   0          18s   10.15.174.22    minikube   <none>         <none>          app=B
dev       pod/nginx1    1/1     Running   0          27m   10.15.19.151    minikube   <none>         <none>          run=nginx1
dev       pod/nginx2    1/1     Running   0          27m   10.15.104.113   minikube   <none>         <none>          run=nginx2

# Ping, should should timeout because NetworkPolicy in place
kubectl exec -ti busybox1 -- ping -c3 10.15.171.50 #<busybox2-ip>

PING 10.15.171.50 (10.15.171.50): 56 data bytes
--- 10.15.171.50 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
Note
If you wish to use dns names eg. busybox2 it requires to create a service, without you can't find names:
kubectl -n dev exec -ti busybox1 -- nslookup busybox2
Server:         10.96.0.10
Address:        10.96.0.10:53

** server can't find busybox2.dev.svc.cluster.local: NXDOMAIN

*** Can't find busybox2.svc.cluster.local: No answer
*** Can't find busybox2.cluster.local: No answer
*** Can't find busybox2.dev.svc.cluster.local: No answer
*** Can't find busybox2.svc.cluster.local: No answer
*** Can't find busybox2.cluster.local: No answer

# Try nginx, but the container does not have ping,curl just whet --spider <dns|ip>
kubectl -n dev run --generator=run-pod/v1 nginx1 --image=nginx
kubectl -n dev run --generator=run-pod/v1 nginx2 --image=nginx
kubectl -n dev expose pod nginx1 --port=80

kubectl -n dev get services
NAMESPACE NAME           TYPE      CLUSTER-IP   EXTERNAL-IP PORT(S) AGE SELECTOR   LABELS
dev       service/nginx1 ClusterIP 10.97.69.143 <none>      80/TCP  22m run=nginx1 run=nginx1

# Call by dns, fqdn: nginx1.dev.svc.cluster.local
kubectl -n dev exec -ti busybox1 -- /bin/wget --spider http://nginx1

Connecting to nginx1 (10.97.69.143:80)
remote file exists


Create allow-A-to-B.yaml policy

cat > allow-A-to-B.yaml << EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-a-to-b #must be lowercase req.DNS-1123
# namespace: dev
spec:
  podSelector: {}
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: A
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: B
  policyTypes:
  - Ingress
  - Egress
EOF

kubectl -n dev create -f allow-A-to-B.yaml
networkpolicy.networking.k8s.io/allow-out-to-in created


Test allow policy

kubectl -n dev exec -ti busybox1 -- ping -c3 10.15.171.50 # fail
kubectl -n dev exec -ti busybox3 -- ping -c3 10.15.174.22 # success

# Apply labels from: A to: B
kubectl -n dev label pod busybox1 app=A
kubectl -n dev label pod busybox2 app=B
kubectl -n dev exec -ti busybox1 -- ping -c3 10.15.171.50 # success

# Tidy up
kubectl -n dev delete networkpolicy --all


Deployment (optional test)
kubectl create deployment nginx --image=nginx          # create a deployment
kubectl scale rs nginx-554b9c67f9 --replicas=3         # scale deployment
#kubectl run nginx --image=nginx --replicas=3 #deprecated command 
kubectl expose deployment nginx --port=80

# Try accessing a service from another pod
kubectl run --generator=run-pod/v1 busybox --image=busybox -- sleep 3600
kubectl exec busybox -it -- /bin/sh #this often crashes
/ # wget --spider --timeout=1 nginx #this should timeout
     #--spider does not download just browses

kubectl exec -ti busybox -- wget --spider --timeout=1 nginx


Create NetworkPolicy that allows ingress port 5432 from pods with 'web' label

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-netpolicy
spec:
  podSelector:   # apply policy
    matchLabels: # to these pods
      app: web
  ingress:
  - from:
    - podSelector:     # allow traffic
        matchLabels:   # from these pods
          app: busybox
    ports:
    - port: 80


Label a pod to get the NetworkPolicy:

kubectl label pods [pod_name] app=db
kubectl run busybox --rm -it --image=busybox /bin/sh
#wget --spider --timeout=1 nginx #this should timeout


NetowrkPolicy examples
namespace NetworkPolicy IP block NetworkPolicy egress NetworkPolicy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ns-netpolicy
spec:
  podSelector:
    matchLabels:
      app: db
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          tenant: web
    ports:
    - port: 5432
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ipblock-netpolicy
spec:
  podSelector:
    matchLabels:
      app: db
  ingress:
  - from:
    - ipBlock:
        cidr: 192.168.1.0/24
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: egress-netpol
spec:
  podSelector:
    matchLabels:
      app: web
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: db
    ports:
    - port: 5432

TLS Certificates

Pods communicate with API server using mutual TLS. The ca-bundle is automatically mounted in a pod /var/run/secrets/kubernetes.io/serviceaccount as ca.crt file using default service account.

.   API Server <-------[crt]- Pod[crt signed]
   trusts certs                     by CA
   signed by CA


Show automatically installed ca-bundle

kubectl exec busybox1 -- ls /var/run/secrets/kubernetes.io/serviceaccount
ca.crt     namespace  token


Generate a new certificate

Kubernetes has build-in api to generate custom certificates.


Create a namespace and a pod that will use the new certificate while calling the server-api.

kubectl create ns my-namespace
kubectl -n my-namespace run --generator=run-pod/v1 my-pod --labels="app=busybox" --image=busybox -- sleep 3600
# add create a service 'my-svc'


Install SSL toolkit used, to generate certs CFSSL - CloudFlare's


Manifest to create csr

Create csr and signing CertificateSigningRequest object
csr CertificateSigningRequest
cat <<EOF | cfssl genkey - | cfssljson -bare server
{
  "hosts": [
    "my-svc.my-namespace.svc.cluster.local",
    "my-pod.my-namespace.pod.cluster.local",
    "172.168.0.24",
    "10.0.34.2"
  ],
  "CN": "my-pod.my-namespace.pod.cluster.local",
  "key": {
    "algo": "ecdsa",
    "size": 256
  }
}
EOF
2019/08/22 21:23:24 [INFO] generate received request
2019/08/22 21:23:24 [INFO] received CSR
2019/08/22 21:23:24 [INFO] generating key: ecdsa-256
2019/08/22 21:23:24 [INFO] encoded CSR
$ ls -l
...
-rw-r--r-- 1 ubuntu ubuntu 558 Aug 22 21:23 server.csr
-rw------- 1 ubuntu ubuntu 227 Aug 22 21:23 server-key.pem
cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: pod-csr.web
spec:
  groups:
  - system:authenticated
  request: $(cat server.csr | base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

# this only work with here-docs as kubectl cannot interpret 
# bash when passing -f .yaml file


Sign the request

$ kubectl.exe -n dev get csr
NAME         AGE     REQUESTOR       CONDITION
my-pod-csr   9m51s   minikube-user   Pending  #<- status

$ kubectl.exe -n dev describe csr
Name:               my-pod-csr
Labels:             <none>
Annotations:        <none>
CreationTimestamp:  Thu, 22 Aug 2019 21:46:28 +0100
Requesting User:    minikube-user
Status:             Pending #<- changes to Approved,Issued once approved
Subject:
  Common Name:    my-pod.my-namespace.pod.cluster.local
  Serial Number:
Subject Alternative Names:
         DNS Names:     my-svc.my-namespace.svc.cluster.local
                        my-pod.my-namespace.pod.cluster.local
         IP Addresses:  172.168.0.24
                        10.0.34.2
Events:  <none>

# Approve
$ kubectl.exe -n dev certificate approve my-pod-csr
certificatesigningrequest.certificates.k8s.io/my-pod-csr approved

$ kubectl.exe -n dev get csr
NAME         AGE   REQUESTOR       CONDITION
my-pod-csr   14m   minikube-user   Approved,Issued

# Preview the certificate, it's part of 'CertificateSigningRequest' object
$ kubectl.exe -n dev get csr my-pod-csr -o yaml
apiVersion: v1
items:
- apiVersion: certificates.k8s.io/v1beta1
  kind: CertificateSigningRequest
  metadata:
    creationTimestamp: "2019-08-22T20:46:28Z"
    name: my-pod-csr
    resourceVersion: "8027"
    selfLink: /apis/certificates.k8s.io/v1beta1/certificatesigningrequests/my-pod-csr
    uid: 409e7164-c06d-4a5f-896a-30b7329c5156
  spec:
    groups:
    - system:masters
    - system:authenticated
    request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0...SRVFVRVNULS0tLS0K
    usages:
    - digital signature
    - key encipherment
    - server auth
    username: minikube-user
  status:
    certificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN3ekNDQWF1Z0F3...DQVRFLS0tLS0K
    conditions:
    - lastUpdateTime: "2019-08-22T21:00:45Z"
      message: This CSR was approved by kubectl certificate approve.
      reason: KubectlApprove
      type: Approved
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

# Extract certificate
kubectl get csr my-pod-csr -o jsonpath='{.status.certificate}' | base64 --decode > server.crt

Security context aka runas

This feature allows to assign security context at container or pod level (applies to all containers), defining permissions and a user of running processes. Elsewhere it'd run as root if not defined. Help:

kubectl explain pod.spec.containers.securityContext


Show default security context

kubectl -n dev run --generator=run-pod/v1 alpine-with-defaults --image=alpine --restart Never -- /bin/sleep 99999
kubectl -n dev exec alpine-with-defaults -- id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)


Different security contexts

Headline
Container context Add/drop capabilities Read only filesystem
apiVersion: v1
kind: Pod
metadata:
  name: alpine-nonroot
spec:
  containers:
  - name: alpine-main
    image: alpine
    command: ["/bin/sleep", "9999"]
    securityContext:
      runAsUser: 1111    # [1]
      runAsNonRoot: true # [2]

# [1] will 'entrypoint' AsUser even if the user does not exist
# [2] this will look into USER variable in Dockerfile
apiVersion: v1
kind: Pod
metadata:
  name: alpine-changedate
spec:
  containers:
  - name: alpine-main
    image: alpine
    command: ["/bin/sleep", "9999"]
    securityContext:
      capabilities:
        add:        # allow date change
        - SYS_TIME  # the date will change back in ~5s
        drop:
        - CHOWN     # won't allow change owner
apiVersion: v1
kind: Pod
metadata:
  name: alpine-ro-filesystem
spec:
  containers:
  - name: alpine-main
    image: alpine
    command: ["/bin/sleep", "9999"]
    securityContext:
      readOnlyRootFilesystem: true
    volumeMounts:
    - name: rw-volume
      mountPath: /volume
      readOnly: false
  volumes:
  - name: rw-volume
    emptyDir:

#sucess
kubectl exec -it alpine-ro-filesystem -- touch /volume/aaa 
#fails as on rootFS
kubectl exec -it alpine-ro-filesystem -- touch aaa


Pod security context
kubectl explain pod.spec.securityContext


Manifest

apiVersion: v1
kind: Pod
metadata:
  name: alpine-group-context
spec:
  securityContext:
    fsGroup: 555 # special supplemental group that applies to all containers in a pod
    supplementalGroups: [666, 777]
  containers:
  - name: first
    image: alpine
    command: ["/bin/sleep", "9999"]
    securityContext:
      runAsUser: 1111
    volumeMounts:
    - name: shared-volume
      mountPath: /volume
      readOnly: false
  - name: second
    image: alpine
    command: ["/bin/sleep", "9999"]
    securityContext:
      runAsUser: 2222
    volumeMounts:
    - name: shared-volume
      mountPath: /volume
      readOnly: false
  volumes:
  - name: shared-volume
    emptyDir:


Verification commands:

$ kubectl exec -it group-context -c first -- id
uid=1111 gid=0(root) groups=555,666,777          # user 1111
$ kubectl exec -it group-context -c second -- id
uid=2222 gid=0(root) groups=555,666,777          # user 2222

$ kubectl exec -it group-context -c first -- touch /tmp/file0
$ kubectl exec -it group-context -c first -- ls -l /tmp/file0
-rw-r--r--    1 1111     root             0 Aug 26 16:08 /tmp/file0    # default group

$ kubectl exec -it group-context -c first -- touch /volume/file1
$ kubectl exec -it group-context -c first -- ls -l /volume/file1
-rw-r--r--    1 1111     555              0 Aug 26 16:06 /volume/file1 # group specified at Pod level

Secrets, secure KV pairs

Secrets are maps that hold key value pairs, they persist beyond pod lifecycle. They can be passed to a container as environemnt variable but mounting secrets as volume is recommended way. This way, secrets never get written to disk because are stored in an in-memory filesystem (tmpfs) and mounted as a volume. Applications will read secrets as files from the mounted path.


Each pod has default secret (holds service-account token) volume attached to it.

kubectl get secrets -A # get all secrets

# Start a pod to see mounted secret
kubectl -n dev run --generator=run-pod/v1 alpine-with-defaults-1 --image=alpine:3.7 --restart Never -- /bin/sleep 99999
kubectl -n dev describe pod alpine-with-defaults-1 | grep -A4 -e Volumes -e Mounts
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-sk6hd (ro)
Conditions:
  Type              Status
  Initialized       True 
--
Volumes:
  default-token-sk6hd:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-sk6hd
    Optional:    false

# Describe the secret
kubectl -n dev describe secrets default-token-sk6hd
Name:         default-token-sk6hd
Namespace:    dev
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: default
              kubernetes.io/service-account.uid: c3ed7359-d815-4ecb-9862-0d25996b6ebe

Type:  kubernetes.io/service-account-token
Data
====
token:      eyJhbGciOiJSUzI1NiIs**********1qxmToQLwCUdhUqSYEag
ca.crt:     1066 bytes
namespace:  3 bytes

# Preview volume mounts of secrets
kubectl -n dev exec -it alpine-with-defaults-1 -- mount | grep kubernetes
tmpfs on /run/secrets/kubernetes.io/serviceaccount type tmpfs (ro,relatime)

# Preview the secret value from a container
kubectl -n dev exec -it alpine-with-defaults-1 -- cat /var/run/secrets/kubernetes.io/serviceaccount/token
eyJhbGciOiJSUzI1NiIs**********1qxmToQLwCUdhUqSYEag


Store certificates in secrets
openssl genrsa -out https.key 2048
openssl rand -hex 10 > ~/.rnd # avoid ERR: 140468465254848:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/home/<user>/.rnd 
openssl req -new -x509 -key https.key -out https.cert -days 365 -subj /CN=www.example.com
touch file
kubectl -n dev create secret generic https-cert --from-file=https.key --from-file=https.cert --from-file=file

kubectl -n dev describe secrets https-cert 
Name:         https-cert
Namespace:    dev
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
file:        0 bytes
https.cert:  1131 bytes
https.key:   1675 bytes

# Decoded version in base64 format
$ kubectl -n dev get secrets https-cert -oyaml
apiVersion: v1
data:
  file: ""
  https.cert: LS0tLS1CRUdJTiBDRVJUSUZJQ0F******lDQVRFLS0tLS0K
  https.key: LS0tLS1CRUdJTi*****IFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
kind: Secret
metadata:
  creationTimestamp: "2019-08-27T07:12:59Z"
  name: https-cert
  namespace: dev
  resourceVersion: "21864"
  selfLink: /api/v1/namespaces/dev/secrets/https-cert
  uid: e9e7941c-46ac-4996-99d2-f53bc8e2a1ba
type: Opaque


Use the secrets with nginx now

Headline
nginx-https manifest https-cert secret manifest
apiVersion: v1
kind: Pod
metadata:
  name: nginx-https
spec:
  containers:
  - image: nginx
    name: html-web
    env:
    - name: INTERVAL
      valueFrom:
        configMapKeyRef:
          name: config
          key: sleep-interval
    volumeMounts:
    - name: html
      mountPath: /var/htdocs
  - image: nginx:alpine
    name: web-server
    volumeMounts:
    - name: html
      mountPath: /usr/share/nginx/html
      readOnly: true
    - name: config
      mountPath: /etc/nginx/conf.d
      readOnly: true
    - name: certs
      mountPath: /etc/nginx/certs/
      readOnly: true
    ports:
    - containerPort: 80
    - containerPort: 443
  volumes:
  - name: html
    emptyDir: {}
  - name: config
    configMap:
      name: config
      items:
      - key: nginx-config.conf
        path: https.conf
  - name: certs
    secret:
      secretName: https-cert
apiVersion: v1
kind: ConfigMap
metadata:
  name: config
data:
  nginx-config.conf: |
    server {
        listen              80;
        listen              443 ssl;
        server_name         www.example.com;
        ssl_certificate     certs/https.cert;
        ssl_certificate_key certs/https.key;
        ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers         HIGH:!aNULL:!MD5;

        location / {
            root   /usr/share/nginx/html;
            index  index.html index.htm;
        }

    }
  sleep-interval: |
    25
kubectl port-forward example-https 8443:443 &
curl -k https://localhost:8443

Resources