Docker compose¶

In previous tutorial, we have seen how to use docker to run single container. Imagine that we have a list of container to run for one service, and they need to be coordinated. For this kind of situation, we can use docker compose.

Docker compose is used to manage applications and increase efficiency in container development. Configurations are defined in a single YAML file, making applications easy to build and scale.

In short, Docker Compose uses a single docker-compose.yml configuration file to create a list of services(i.e. containers).

1. Requirements¶

To run docker compose, you need both Docker Engine and Docker Compose binaries. There are two ways:

Install standalone binaries of Docker Engine and Docker Compose.
Install Docker Desktop, It contains the Development environment with graphical user interface including Docker Engine and Docker Compose.

2. Installation¶

Check the first tutorial 01.Install_docker_dockerCompose.md.

3. Important terms¶

There are three important component in the docker-compose.yml file: - services - volumes - networks

A simple example of docker-compose.yml

version: "3.7"
services:
  ...
volumes:
  ...
networks:
  ...

3.1 Services¶

services refer to the containers’ configuration.

For example, let’s take a dockerized web application consisting of a front end, a back end, and a database. We’d likely split these components into three images, and define them as three different services in the configuration:

services:
  frontend:
    image: my-vue-app
    ...
  backend:
    image: my-springboot-app
    ...
  db:
    image: postgres
    ...

3.2 Volumes¶

Volumes, are physical areas of disk space shared between the host and a container, or even between containers. In other words, a volume is a shared directory in the host, visible from some or all containers.

3.3 Networks¶

Networks define the communication rules between containers, and between a container and the host. Common network zones will make the containers’ services discoverable by each other, while private zones will segregate them in virtual sandboxes.

4. More about services¶

A service contains the below parts: - image - network - volume - Dependencies

4.1 Getting the image¶

There are two possibility: - pull image from docker registry (e.g docker hub, etc.): - build locally from a docker file

Pull image from docker registry¶

You need to configure the docker registry url (docker hub by default). Then you need to specify the image name and tag.

Below is an example, which pulls an image ubuntu with tag latest

services:
   my-service:
      image: ubuntu:latest

Build image from docker file¶

We will use the keyword build, and a docker file.

Below is an example where the docker file is hosted locally

services: 
  my-custom-app:
    build: /path/to/dockerfile/
    ...

The docker file can be hosted remotely too. Below is an example where the docker file is hosted on github

services: 
  my-custom-app:
    build: https://github.com/my-repo/my-project.git
    ...

If you want to share the build image with others, you can add another line (image:<image-name>) like in below example

services: 
  my-custom-app:
    build: /path/to/dockerfile/
    image: my-project-image
    ...

This will create an image on your local image registry after the build process

4.2 Configuring the network¶

There are two types of communication: - comm between host and containers - comm between containers

Communication between host and containers¶

** To reach a container from the host, the ports must be exposed declaratively through the ports keyword**. It will match the container exposed port with the host port. The first value is the host port, the second value is the container exposing port

In below example, we have three services: - helloworld: expose the container port 80, and match it with the host port 80 - myapp1: expose the container port 3000, and match it with the host port 8080 - myapp2: expose the container port 3000, and match it with the host port 8081

So if you type - localhost:80, you will reach the helloworld service - localhost:8080, you will reach the myapp1 service - localhost:8081, you will reach the myapp2 service

services:
  network-example-service:
    image: helloworld:latest
    ports:
      - "80:80"
    ...
  my-custom-app:
    image: myapp1:latest
    ports:
      - "8080:3000"
    ...
  my-custom-app-replica:
    image: myapp2:latest
    ports:
      - "8081:3000"
    ...

Communication between containers¶

Docker containers communicate between themselves in networks created, implicitly or explicily. By default, all containers in the same services share the same default network. A service can communicate with another service on the same network by simply referencing it by using : (e.g. container1:80). We can expose a container port by using the expose keyword.

The below example expose port 80 of the service app 1. Other services inside the same network can access it by using app1:80

services:
  network-example-service:
    image: app1:latest
    expose:
      - "80"

Custom networks¶

If we don't want to use the default network set up, we can use custom network configuration. We can use networks keyword to define virtual networks to segregate containers. In below example, we create two virtual network: - public-network - private-network

The pub-service1 and pub-service2 are in the public-network. so they can communicate between them. The private-service is the only container in the private-network, so it can't communicate with pub-service1 and pub-service2

services:
  pub-service1:
    image: alpine:latest
    networks: 
      - public-network
    ...
  pub-service2:
    image: alpine:latest
    networks: 
      - public-network
    ...
  private-service:
    image: alpine:latest
    networks: 
      - private-network
    ...
networks:
  public-network: {}
  private-network: {}

4.3 Configure Volumes¶

There are three types of volumes: - anonymous - named - host

Anonymous and named volumes are managed by the Docker engine, Docker will generate the directories and store data in the host. These volumes are automatically mounted when the container is started.

The good practice is to use the named volume. Below are the commands of named volume management

# create the volume
docker volume create <volume-name>

# list existing volume
docker volume list

# mount volume on the container
docker run --name <container-name> -v <volume-name>:<container-mount-path> <image-name>

# remove volume
docker volume remove <volume-name>

Host volumes allow us to specify an existing folder in the host and mount it with a specific path on the container.

In the below example, we have two services: - app1: has three volumes, the two first volumes are host volumes which matches existing host directory with container directory. The last one uses a named volume. You can also notice, we need to declare the named volume first in the upper level volumes specs.

To mount a volume in read-only mode by appending :ro to the volume declaration. For example /home:/my-volumes/readonly-host-volume:ro specifies that the /home folder is read only. (we don’t want a Docker container erasing our users by mistake).

services:
  app1:
    image: alpine:latest
    volumes: 
      - /tmp:/my-volumes/host-volume
      - /home:/my-volumes/readonly-host-volume:ro
      - my-named-volume:/my-volumes/named-global-volume
    ...
  app2:
    image: alpine:latest
    volumes:
      - my-named-volume:/another-path/the-same-named-global-volume
    ...
volumes:
  my-named-volume:

4.4 Container dependencies¶

We need to create a dependency chain between our services so that some services get loaded before (and unloaded after) other ones. We can achieve this result through the depends_on keyword:

Below examples specifies that service kafka needs zookeeper to run first.

services:
  kafka:
    image: kafka
    depends_on:
      - zookeeper
    ...
  zookeeper:
    image: zookeeper
    ...

We should be aware, however, that Compose won’t wait for the zookeeper service to finish loading before starting the kafka service; it’ll simply wait for it to start. If we need a service to be fully loaded before starting another service, we need to get deeper control of the startup and shutdown order in Compose.

5. Managing Environment Variables¶

Working with environment variables is easy in Compose. We can define static environment variables, as well as dynamic variables, with the ${} notation:

To define the environment values, we have the following approaches: 1. Compose file 2. Shell environment variables 3. Environment file 4. Dockerfile 5. Variable not defined.

We can mix the above approaches, but let’s keep in mind that Compose uses the priority order (1 has the highest order), overwriting the value of less important approaches with the higher priorities approaches

Once you have declared the Environment Variables, you can use them in your docker-compose file. Below is an example on how to use env var in the docker-compose file.

services:
  database: 
    image: "postgres:${POSTGRES_VERSION}"
    environment:
      DB: mydb
      USER: "${USER}"

5.1 Declare env var in docker compose file¶

You can set environment variables directly in your Compose file. This option has many limitation. The value is visible which makes it hard to version your compose file.

services:
  webapp:
    image: my-webapp-image
    environment:
      DB: mydb
      USER: toto

You can also use the -e option in the docker run/compose command. For example

docker run -e "[variable-name]=[new-value]"
docker run -e "DEBUG=1"

docker compose -e "[variable-name]=[new-value]"
docker compose -e "DEBUG=1"

5.2 Declare env var in Shell environment variables¶

# we declare the env var before calling the docker compose command
export POSTGRES_VERSION=alpine
export USER=foo

docker-compose -f docker-compose-file.yaml up

5.3 Declare env var in Environment file¶

An .env file in Docker Compose is a text file used to define environment variables that should be made available to Docker containers when running docker compose up. This file typically contains key-value pairs of environment variables, and it allows you to centralize and manage configuration in one place.

The .env file is the default method for setting environment variables in your containers. It is very useful if you have multiple environment variables you need to store.

The .env file should be placed at the root of the project directory next to your compose.yaml file. For more information on formatting an environment file, see Syntax for environment files.

When you run docker compose up, all the env var inside the compose file will be replaced by the values of the env file and generate the final config file. You can verify the generated config file by using below command:

docker compose config

For example If you define an environment variable DEBUG=1 in your .env file, and your compose.yml file looks like this:

 services:
    webapp:
      image: my-webapp-image
      environment:
        - DEBUG=${DEBUG}

Docker Compose replaces ${DEBUG} with the value 1 from the .env file.

Multiple env file¶

You can use multiple .env files in your compose.yml with the env_file attribute, and Docker Compose reads them in the order specified. If the same variable is defined in multiple files, the last definition takes precedence:

services:
  webapp:
    image: my-webapp-image
    env_file:
      - path: ./default.env
        required: true # default value
      - path: ./override.env
        required: false # this env file is Optional

You can also use the --env-file to add custom env file while running the docker compose command

5.4 Declare env var in Docker file¶

You can define as many env var in the Docker file as want use ENV VAR1=$TEST1

6. Scaling and Replicas¶

The docker-compose scale command. Newer versions deprecated it, and replaced it with the scale option.

Below is an example how to use Docker swarn(a cluster of Docker engines) to autoscale our containers.

The deploy section is effective only when deploying to docker swarn.

services:
  worker:
    image: my-webapp-image
    networks:
      - frontend
      - backend
    deploy:
      mode: replicated
      replicas: 6
      resources:
        limits:
          cpus: '0.50'
          memory: 50M
        reservations:
          cpus: '0.25'
          memory: 20M
      ...

7. Lifecycle management¶

The general docker compose command can be found Here

Note the newer version, the command is no longer docker-compose, but docker compose

The docker container lifecycle can be described as: - Create - Run - Pause - Stop - Delete

The below image shows the commands to change the state of the container

7.1 Service(List of containers) creation¶

The docker compose up command builds, (re)creates, starts, and attaches to containers for a service.

Unless they are already running, this command also starts any linked services.

docker compose [-f <arg>...] [options] [COMMAND] [ARGS...]

# container creation, If the config file has a different name than the default one (docker-compose.yml), we must use
# the option -f to specify the config file path
# the -d option makes the compose process run in the background
docker compose -f <docker-compose-file-path> up -d

7.2 Running the services¶

Starts existing containers for a service

# run container, if the containers are already created 
docker compose -f <docker-compose-file-path> start

7.3 Pause/Unpause the services¶

Pause/Unpause a running service

docker compose -f <docker-compose-file-path> pause/unpasue

7.4 Stop the services¶

There are different level of stop.

# Stops running containers without removing them. They can be started again with docker compose start.
docker compose stop

# Stops containers and removes containers, networks, volumes, and images created by up.
docker compose down

An application example¶

You can follow this tutorial to have a first idea how a docker compose service runs.

You can find the source file in src/composetest