Skip to content

Install and config containerd

You can find the official doc for installing containerd here.

1. Prerequisites

To make containerd run correctly, we need to do the following config - add two moduls into the linux kernel(e.g. overlay, br_netfilter) - if containerd needs to work with k8s cri, you need to reconfigure some kernel parameters

1.1 Add modules to linux kernel

The below command will create a file containerd.conf in /etc/modules-load.d/, and add two lines overlay, and br_netfilter in the containerd.conf file.

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

The file specifies the kernel modules overlay, and br_netfilter should be automatically loaded at boot.

  • overlay: This module is required for containerd and Docker. It enables the OverlayFS filesystem, which helps efficiently store container layers.
  • br_netfilter: This module ensures that bridged traffic is correctly processed by iptables. It's an essential module for Kubernetes networking (iptables-based rules)

To load the module without rebooting, you can load the module manually with the below command

sudo modprobe overlay
sudo modprobe br_netfilter

To check if the module is loaded correctly or not, you can use the below command.

lsmod | grep -E 'overlay|br_netfilter'

# the expected output
br_netfilter           32768  0
bridge                262144  1 br_netfilter
overlay               147456  0

1.2 Reconfigure kernel parameters (If containerd is installed for k8s cluster)

We need to modify the value of the below kernel parameters:

  • net.bridge.bridge-nf-call-iptables = 1: this parameter ensures that iptables processes traffic from bridged network interfaces. It's required for Kubernetes networking (especially CNI plugins like Flannel, Calico).
  • net.ipv4.ip_forward = 1: this parameter enables IP forwarding, allowing the machine to route packets. It's required for Kubernetes pod-to-pod communication.
  • net.bridge.bridge-nf-call-ip6tables = 1: this parameter enables ip6tables processes IPv6 bridged traffic. It's useful if your cluster supports IPv6 networking.

IP forwarding is a feature in the Linux kernel that allows a machine to act as a router, forwarding network packets from one interface to another. By default, Linux does not forward packets between network interfaces unless explicitly enabled.

As all kernel parameters are stored in /etc/sysctl.d/, by convention, we create a config file 9-kubernetes-cri.conf, and put the above three lines in it. You can use the below command to do it.

cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

To apply the change you can use

# The below command will trigger the system to read and apply all settings from:
# /etc/sysctl.conf
# /run/sysctl.d/*.conf
# /etc/sysctl.d/*.conf
# /usr/lib/sysctl.d/*.conf
sudo sysctl --system

# test the updated value
sudo sysctl net.bridge.bridge-nf-call-iptables
sudo sysctl net.ipv4.ip_forward
sudo sysctl net.bridge.bridge-nf-call-ip6tables

# or you can test all with oneliner
sudo sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

2. Install containerd (via apt)

The containerd.io packages in DEB and RPM formats are distributed by Docker (not by the containerd project). The Docker documentation for how to set up apt-get to install containerd.io packages:

2.1 Remove old versions if exist

Your Linux distribution may provide unofficial Docker packages, which may conflict with the official packages provided by Docker. You must uninstall these packages before you install the official version of Docker Engine.

The unofficial packages to uninstall are:

  • docker.io
  • docker-compose
  • docker-doc
  • podman-docker

Run the following command to uninstall all conflicting packages:

# remove all conflicting packages
for pkg in docker.io docker-doc docker-compose podman-docker containerd runc; do sudo apt-get remove $pkg; done

# Remove the installed default config file
rm /etc/containerd/config.toml

2.2 Configure the official repo

# install required packages
sudo apt-get install \
    ca-certificates \
    curl \
    gnupg

# add gpg key for the containerd.io repo
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

# add docker.io repo to source.list
echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

2.3 Install containerd binary and set up config

Install the containerd binary via new repo

sudo apt update
sudo apt install containerd.io

# check the installed version
containerd --version

# expected output
containerd containerd.io 1.7.25 bcc810d6b9066471b0b6fa75f557a15a1cbf31bb

3. Configure containerd after installation

There is no default configuration file after installation. You need to create one by yourself.

3.1 Generate a default config

# generate a default config
containerd config default | sudo tee /etc/containerd/config.toml

The default config file needs to be modified, below are some points you need to pay attention to: - cgroup driver setting: containerd requires cgroup(linux kernel) to setup resources(e.g. cpu, memory, etc.) and security policies. It uses a cgroup driver to communicate with the cgroup. You need to adapt this value based on your system setting - sandbox image url: In k8s, all pods have a sandbox container that uses sandbox image. - custom image registry: If you want to use custom image registry, you need to change the default config too.

3.2 Change the cgroup driver to systemd

Find the following section [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] in /etc/containerd/config.toml, then add this line SystemdCgroup = true under it.

Or you can run the below command to change it, note it only works if the generated config contains SystemdCgroup = false

sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

After the above command, you should see the below lines in etc/containerd/config.toml

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

3.3 change sandbox image pull url

All pods in k8s require a sandbox image, the containerd default config has a default sandbox image pulling image. Based on the k8s version, you need to adapt this value.

You should find the below line, and change the image url to your k8s required image url. 

sandbox_image = "k8s.lixx.cn/pause:3.10"

Every Kubernetes Pod has a Pause container(sandbox container) that holds the network namespace and acts as the parent of all other containers in the Pod. It has two main goals: - Keeps Pod Resources Active – It ensures that networking and IPC resources remain stable even if app containers restart. - Efficient Namespace Sharing – Other containers in the Pod share its PID, network, and IPC namespaces.

3.4 Custom image registry

containerd allows us to use custom image registry. Suppose our image registry runs at https://reg.casd.local.

You need to edit the containerd config file /etc/containerd/config.toml

sudo vim /etc/containerd/config.toml

# find the  plugins."io.containerd.grpc.v1.cri".registry section in the config.tom
# add the below line, you need to change the url value to your url
[plugins."io.containerd.grpc.v1.cri".registry]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
           [plugins."io.containerd.grpc.v1.cri".registry.mirrors."reg.casd.local"]
               endpoint = ["https://reg.casd.local"]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."reg.casd.local".tls]
           insecure_skip_verify = true
        [plugins."io.containerd.grpc.v1.cri".registry.configs."reg.casd.local".auth]
           username = "toto"
           password = "chagneMe"
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]

# now we need to reload the daemon
sudo systemctl daemon-reload

# restart containerd
sudo systemctl restart containerd

# try to pull an image from the private image registry
sudo crictl pull reg.casd.local/casd/redis

DEPRECATION: The mirrors property of [plugins."io.containerd.grpc.v1.cri".registry] is deprecated since containerd v1.5 and will be removed in containerd v2.1. Use config_path instead. For now, we don't migrate to config_path, because it does not provide all features(for example .auth).

3.5 Restart and enable containerd service on all the nodes

$ sudo systemctl restart containerd
$ sudo systemctl enable containerd

# you can check the service status
$ sudo systemctl status containerd

# check the version
containerd --version

# you can use the default containerd client to test
ctr version

4. Install and setup containerd client

Below is a list of containerd major clients: - ctr: default - crictl: client compatbile with k8s

4.1 ctr

containerd provides a CLI called ctr. It's a low-level CLI tool designed for direct interaction with containerd. It supports the complete life-cycle of a container(e.g. pull image, create container, etc.) But it does not support the Kubernetes CRI (Container Runtime Interface).

# list containers
ctr containers list

# pull image
ctr images pull docker.io/library/nginx:latest

# list images
sudo ctr images ls

# run a container
ctr run --rm -t docker.io/library/nginx:latest my-nginx

All the cri pluging configurations in /etc/containerd/config.toml are ignored by ctr, because it does not support CRI. That's we recommend you to use crictl for testing.

For more details of ctr , you can visit this page

4.2 crictl

The official docs of crictl can be found here.

The latest release version can be found here.

Follow the below steps to install and config crictl

# install crictl bin
VERSION="v1.32.0"
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
sudo tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
rm -f crictl-$VERSION-linux-amd64.tar.gz

# config crictl.yaml
cat <<EOF | sudo tee /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 2
debug: false
pull-image-on-create: false
EOF

# list all images
sudo crictl images

# pull images
sudo crictl pull reg.casd.local/casd/redis

# list pods
sudo crictl pods

# list containers
sudo crictl ps -a

# get logs of a container
sudo crictl logs <conainer_id>

Appendix: cgroup and systemd

Control Groups (cgroups) and systemd are closely related because systemd is responsible for managing system services and processes, and it heavily relies on cgroups to do so.

  • systemd organizes processes using cgroups to track and manage resource usage;
  • Every service, user session, or scope started by systemd gets its own cgroup. For example, the nginx.service runs in /sys/fs/cgroup/system.slice/nginx.service/
  • This allows fine-grained control over CPU, memory, I/O, and other resources.
# test cgroup version
mount | grep cgroup

# expected output
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)


# we can set max cpu and memory for a service
systemctl set-property nginx.service CPUQuota=50%
systemctl set-property nginx.service MemoryMax=500M