Post k8s installation config

After k8s installation, you need to install other tools such as: - helm - reverse proxy

I consider cni plugin (e.g. calico, flannel, etc.) is a part of basic component of k8s cluster.

1. Install helm

The official release page of helm can be found here

You can use the below bash script to install the latest version.

vim install_helm.bash

# put the below content and run it
bash install_helm.bash

#!/bin/bash

set -euo pipefail

HELM_VERSION="v3.18.4"
HELM_TAR="helm-${HELM_VERSION}-linux-amd64.tar.gz"
HELM_URL="https://get.helm.sh/${HELM_TAR}"
TMP_DIR="/tmp/helm-install"
HELM_DIR="linux-amd64"

mkdir -p "$TMP_DIR"
cd "$TMP_DIR"

echo "Downloading Helm ${HELM_VERSION} to ${TMP_DIR}..."
wget -q "${HELM_URL}"

echo "Extracting Helm..."
tar -xzf "${HELM_TAR}"

echo "Setting execution permission..."
chmod a+x "${HELM_DIR}/helm"

echo "Moving Helm binary to /usr/local/bin..."
sudo mv "${HELM_DIR}/helm" /usr/local/bin/helm

echo "Cleaning up..."
rm -rf "$TMP_DIR"

echo "Verifying Helm installation..."
if command -v helm >/dev/null 2>&1; then
    helm version
    echo "Helm installed successfully."
else
    echo "Helm installation failed."
    exit 1
fi

You should see the helm version and the success message. You need to change version and target system architecture if you use another OS other than debian 11.

2. Set up a reverse proxy

A reverse proxy is essential for a k8s cluster. Otherwise, the applications deployed in the cluster are not accessible from the outside world. There are many possible reverse proxy solutions such as: - Kong (commercial alternative): https://konghq.com/ - Traefik (commercial alternative): https://traefik.io/

In this tutorial, we choose ingress-nginx.

2.1 The ingress nginx controller mode

There are three modes to set up the proxy and reverse proxy for a k8s cluster: - host - load balancer - nodePort

2.1.1 Host mode

In host mode, the Ingress controller it uses the host's network namespace. This means that the Ingress controller binds directly to the host's network interfaces and ports.

The advantage of the host mode is that it can achieve higher performance compared to other modes, as it eliminates the overhead of the kube-proxy layer.

The disadvantage is that you cannot run multiple instances of the Ingress controller on the same host with the same ports, as there would be port conflicts.

To view the detailed configuration of host mode, check the section of hostNetwork: true section in the values.yaml template.

2.1.2 Load balancer mode

In the load balancer mode, the Ingress controller typically runs as a service, and an external load balancer (normally provided by the cloud provider) is provisioned to distribute incoming traffic to the Ingress controller service.

This mode is suitable for cloud environments where a load balancer service can be provisioned dynamically (e.g., AWS ELB, GCP Load Balancer). The external load balancer takes care of distributing traffic to the nodes running the Ingress controller service.

To view the detailed configuration, check the section of appProtocol:True section

2.1.3 nodePort mode

In NodePort mode, the Ingress controller service is exposed on a static port on each node in the cluster. This port is accessible from outside the cluster, and the traffic is then forwarded to the Ingress controller.

This mode is often used in on-premises or bare-metal environments where cloud load balancers are not available or in development/testing scenarios.

While it provides external access, it might not be as suitable for production environments due to potential challenges in scaling and managing external access.

3. A real example

In this example, we choose the host mode. So the ingress-nginx listens to the network interface of the host server. As a result, only one ingress nginx pod can be deployed on each node. And we don't want to have more than one ingress. So we added a node selector on the ingress nginx controller service is only deployed on a specific node.

3.1 Select which node to deploy the ingress nginx controller

To deploy ingress nginx controller service on a specific node: - Label a worker node with a specific label (e.g. ingress-node) - Add a node selector on the ingress nginx controller service

We label only one node, because we need to set up a dns resolver entry, so all the incoming querier can be redirected to the node which contains the ingress controller.

Even we have two pods of Ingress controller. The second one that is not in the DNS will never be used.

# our k8s cluster 
- Master Node (k8s-master) – 10.50.5.67
- Worker Node 1 (k8s-worker1) – 10.50.5.68
- Worker Node 2 (k8s-worker2) – 10.50.5.69

# Here we choose worker 1 to host ingress
# FQDN for k8s Ingress controller
10.50.5.68   *.casd.local

3.2 Label the node

The below commands show you how to label a node with a specific label

# get all available nodes
kubectl get nodes

# output example
NAME            STATUS   ROLES           AGE   VERSION
onyxia-master   Ready    control-plane   19h   v1.33.2
onyxia-w01      Ready    <none>          19h   v1.33.2
onyxia-w02      Ready    <none>          19h   v1.33.2

# general form to label a node, 
kubectl label node <nodename> <label-key>=<label-value>

# example
kubectl label node onyxia-w01 ingress-node=true

# to un-label a node, you can use the below command
kubectl label node <nodename> <labelname>-

# example
kubectl label node worker2 public-

# after labeling, you will see new pod of nginx gets created.
kubectl get all -n ingress-nginx -w
kubectl get pods -n ingress-nginx -w

3.3 Deploy the ingress nginx controller service

You can use the below commands to deploy the ingress nginx controller service

# we want it to run is the namespace ingress-nginx, so we create a namespace
kubectl create namespace ingress-nginx

# add ingress-nginx helm repo
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

# update the repo content
helm repo update

# list available release 
helm search repo

# output example
NAME                            CHART VERSION   APP VERSION     DESCRIPTION
ingress-nginx/ingress-nginx     4.13.0          1.13.0          Ingress controller for Kubernetes using NGINX a

3.3.1 Configure the ingress-nginx controller

With the above ingress-nginx repo, we can install an ingress-nginx controller service in our cluster.

You can find the full doc https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/

You can find the values.yaml template here https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml

3.3.2 A minimum config example

Note the below ingress_values.yaml is an example of how our cluster configures the ingress-nginx controller.

controller:
  watchIngressWithoutClass: true
  allowSnippetAnnotations: false
  config:
    error-log-level: "info"
    ignore-invalid-headers: "false"
    proxy-request-buffering: "off"
    proxy-body-size: "0"
    large-client-header-buffers: "4 16k"

  hostNetwork: true
  extraEnvs:
    - name: MY_POD_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP
  kind: DaemonSet
  service:
    enabled: true
    type: ClusterIP
  ingressClassResource:
    name: nginx
    enabled: true
    default: true
    controllerValue: "k8s.io/ingress-nginx"

rbac:
  create: true
podSecurityPolicy:
  enabled: false

With the above configuration, you can have a minimum running ingress controller

3.3.3 Deploy the ingress service

# we deploy the ingress service with above
# here the version is the helm chart version.
helm install ingress-nginx ingress-nginx/ingress-nginx -f ingress_values.yaml -n ingress-nginx --version v4.13.0

# get all components of the ingress-nginx
kubectl get all -n ingress-nginx

# output example
NAME                                 READY   STATUS    RESTARTS   AGE
pod/ingress-nginx-controller-lnd4s   1/1     Running   0          81s

NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
service/ingress-nginx-controller             ClusterIP   10.96.43.173     <none>        80/TCP,443/TCP   81s
service/ingress-nginx-controller-admission   ClusterIP   10.111.211.231   <none>        443/TCP          81s

NAME                                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                              AGE
daemonset.apps/ingress-nginx-controller   1         1         1       1            1           ingress-node=true,kubernetes.io/os=linux   81s

After the pod of ingress service is created, you can try to send a request to the ip of service/ingress-nginx-controller.

# In our example, the ip address of the service is 10.96.43.173, you can try below command
curl 10.96.43.173 

# if you see below output, it means ingress-nginx is running and answering request
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

ingress nginx cve-2025-1974: https://kubernetes.io/blog/2025/03/24/ingress-nginx-cve-2025-1974/ Avoid the versions which are affected by this CVE.

3.3.4 Test ingress with an application

You can try to deploy the mario app and check the certificate.

The full manifest can be found here

# copy the three yaml files in a folder, then run
kubectl apply -f .

If your dns is configured to redirect queries to the ingress service, then you should be able to use the url to access the service.

You will notice, ingress assigns a fake certificat. Because we have not configured a valid certificate for ingress.

We need to replace this fake certificate with a valid certificate.

4. Configure nginx with a custom certificate for all services

In this tutorial, we suppose you only have: - a self-signed root CA certificate. - a wildcard certificate signed by the root CA.

The objectives are: - Ingress controller trusts the root CA, so it can validate certificates signed by it (for TLS termination). - All apps use the wildcard certificate (e.g., *.casd.local) signed by that internal root CA. - TLS is terminated at the ingress level, and the root CA is the trusted anchor.

4.1 Check your certificates

You need to have the below certificates: - root ca: - wildcard certificate signed by ca - wildcard certificate private key

4.2 Create a TLS secret with the wildcard certificate

Create a secret to host the certificate and private key. we name the secret as casd-wildcard-certificate, you can use the below command

# general form
kubectl create secret tls <secret-name> --namespace <namespace-name> --key=pathTo/ingress-tls.key --cert=pathTo/ingress-tls.crt -o yaml

# example
kubectl create secret tls casd-wildcard-certificate --key=wildcard-casd.key --cert=wildcard-casd.crt -o yaml -n ingress-nginx 

# view the content of the secret, the certificate and private value is in base64, you need to decode it to view the
# value. No encryption at all, so we need to pay attention on who can view this secret.
kubectl get secret casd-wildcard-certificate -o jsonpath='{.data}' -n ingress-nginx 

# you can also edit the value directly
kubectl edit secret casd-wildcard-certificate -n ingress-nginx 

4.3 Create a secret for root ca

# Create a Secret for the root CA
kubectl create secret generic casd-root-ca \
  --from-file=ca.crt \
  -n ingress-nginx

# check the created secret 
kubectl get secret casd-root-ca -o jsonpath='{.data}' -n ingress-nginx 

4.4 Mount Root CA and wildcard certificate into NGINX controller

To tell the ingress to use the given certificate, you need to use extraArgs.default-ssl-certificate config. Below is a full example. Then you need to update the ingress controller with new configuration

controller:
  # the ingress controller will process all Ingress resources that do not have an ingressClassName field.
  watchIngressWithoutClass: true
  allowSnippetAnnotations: false

  # use node selector to install nginx on a specific node
  # all nodes that have label ingress-node:"true" will have a replicas of the nginx 
  nodeSelector:
    ingress-node: "true"

  config:
    error-log-level: "info"
    ignore-invalid-headers: "false"
    proxy-request-buffering: "off"
    proxy-body-size: "0"
    large-client-header-buffers: "4 16k"
    # This tells NGINX to verify the TLS certificate presented by the upstream (backend) service.
    # default value is off
    proxy-ssl-verify: "on"
    # Specifies the CA certificate NGINX should use to verify the backend service’s TLS certificate
    proxy-ssl-trusted-ca-file: "/etc/nginx/certs/ca.crt"
    # tells NGINX which CA to use when verifying client certificates, i.e., when a client presents a certificate to authenticate itself.
    ssl-trusted-ca-file: "/etc/nginx/certs/ca.crt"

  hostNetwork: true

  extraEnvs:
    - name: MY_POD_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP

  kind: DaemonSet

  service:
    enabled: true
    type: ClusterIP

  ingressClassResource:
    name: nginx
    enabled: true
    default: true
    controllerValue: "k8s.io/ingress-nginx"
    # Parameters is a link to a custom resource containing additional
    # configuration for the controller. This is optional if the controller
    # does not require extra parameters.
    parameters: {}

  # Set global default TLS certificate (wildcard)
  # no need to use  `- secretName: casd-test-tls-secret` in ingress.yaml
  # to specify a custom certificate
  # the default certificate should be a wildcard which covers your domain
  extraArgs:
    default-ssl-certificate: "ingress-nginx/casd-wildcard-certificate"

  # Mount internal CA certificate
  extraVolumeMounts:
    - name: root-ca
      mountPath: /etc/nginx/certs
      readOnly: true

  extraVolumes:
    - name: root-ca
      secret:
        secretName: casd-root-ca

rbac:
  create: true

podSecurityPolicy:
  enabled: false

Make sure you have the wildcard and root-ca certificate secret in ingress-nginx name space.

4.5 Update existing ingress-nginx deployment

The best way to update a deployment (deployed via helm chart) is to modify the values.yaml. Then call the below command

# general form
helm upgrade <deployment-name> <chart-name> -f <config-file> -n <namespace>

# example
helm upgrade ingress-nginx ingress-nginx/ingress-nginx -f ingress_values.yaml -n ingress-nginx

# to delete 
helm delete ingress-nginx -n ingress-nginx