Deploy a minio cluster in bare_metal mode¶

In this tutorial, we will deploy a minio cluster in bare_metal mode with multi-Node and multi-drive.

You can find the official doc here

1. Prerequisites¶

1.1 Networking and Firewalls¶

1.1.1 Communication between nodes¶

Each node should have full bidirectional network access to every other node in the cluster

For example, the following command explicitly opens the default MinIO server API port 9000 for servers running firewalld :

firewall-cmd --permanent --zone=public --add-port=9000/tcp
firewall-cmd --reload

1.1.2 User interface¶

If you set a static MinIO Console port (e.g. :9001) you must also grant access to that port to ensure connectivity from external clients.

MinIO strongly recomends using a load balancer to manage connectivity to the cluster. The Load Balancer should use a Least Connections algorithm for routing requests to the MinIO deployment, since any MinIO node in the deployment can receive, route, or process client requests.

The following load balancers are known to work well with MinIO:

NGINX
HAProxy

1.2 Sequential Hostnames¶

MinIO requires using expansion notation {x...y} to denote a sequential series of MinIO hosts when creating a server pool. MinIO supports using either a sequential series of hostnames or IP addresses to represent each minio server process in the deployment.

This procedure assumes use of sequential hostnames due to the lower overhead of management, especially in larger distributed clusters.

Create the necessary DNS hostname mappings prior to starting this procedure. For example, the following hostnames would support a 4-node distributed deployment:

minio-01.casd.local
minio-02.casd.local
minio-03.casd.local
minio-04.casd.local

1.3 Local JBOD Storage with Sequential Mounts¶

MinIO strongly recommends direct-attached JBOD(just a bunch of disks) arrays with XFS-formatted disks for best performance.

Important point to consider:

Direct-Attached Storage (DAS) has significant performance and consistency advantages over networked storage (NAS, SAN, NFS).
Use XFS as file system, other file system have lower performance while exhibiting unexpected or undesired behavior.
RAID or similar technologies do not provide additional resilience or availability benefits when used with distributed MinIO deployments, and typically reduce system performance.
Ensure all nodes in the cluster use the same type (NVMe, SSD, or HDD) of drive with identical capacity (e.g. N TB) . MinIO does not distinguish drive types and does not benefit from mixed storage types. Additionally. MinIO limits the size used per drive to the smallest drive in the deployment. For example, if the deployment has 15 10TB drives and 1 1TB drive, MinIO limits the per-drive capacity to 1TB.

1.3.1 Mount disk to nodes¶

MinIO requires using expansion notation {x...y} to denote a sequential series of drives when creating the new deployment, where all nodes in the deployment have an identical set of mounted drives. MinIO also requires that the ordering of physical drives remain constant across restarts, such that a given mount point always points to the same formatted drive. MinIO therefore strongly recommends using /etc/fstab or a similar file-based mount configuration to ensure that drive ordering cannot change after a reboot. For example:

$ mkfs.xfs /dev/sdb -L DISK1
$ mkfs.xfs /dev/sdc -L DISK2
$ mkfs.xfs /dev/sdd -L DISK3
$ mkfs.xfs /dev/sde -L DISK4

$ nano /etc/fstab

  # <file system>  <mount point>  <type>  <options>         <dump>  <pass>
  LABEL=DISK1      /mnt/disk1     xfs     defaults,noatime  0       2
  LABEL=DISK2      /mnt/disk2     xfs     defaults,noatime  0       2
  LABEL=DISK3      /mnt/disk3     xfs     defaults,noatime  0       2
  LABEL=DISK4      /mnt/disk4     xfs     defaults,noatime  0       2

noatime is for optimizing system performance

Trouble shoot¶

if you can't install xfs, you may need to use following command to install the required package

Distribution	Command
Debian	apt-get install xfsprogs
Ubuntu	apt-get install xfsprogs
Alpine	apk add xfsprogs
Arch Linux	pacman -S xfsprogs
Kali Linux	apt-get install xfsprogs
CentOS	yum install xfsprogs
Fedora	dnf install xfsprogs
Raspbian	apt-get install xfsprogs

1.4 Time synchronization¶

Multi-node systems must maintain synchronized time and date to maintain stable internode operations and interactions. Make sure all nodes sync to the same time-server regularly. Check the doc of time_sync.

Redundancy¶

MinIO erasure coding is a data redundancy and availability feature that allows MinIO deployments to automatically reconstruct objects on-the-fly despite the loss of multiple drives or nodes in the cluster. Erasure Coding provides object-level healing with less overhead than adjacent technologies such as RAID or replication. Distributed deployments implicitly enable and rely on erasure coding for core functionality.

Erasure Coding splits objects into data and parity blocks, where parity blocks support reconstruction of missing or corrupted data blocks. The number of parity blocks in a deployment controls the deployment’s relative data redundancy. Higher levels of parity allow for higher tolerance of drive loss at the cost of total available storage.

The default erasure coding config of MinIO is set to EC:4 (4 parity blocks per erasure set). You can set a custom parity level by setting the appropriate MinIO Storage Class environment variable.

Consider using the MinIO Erasure Code Calculator for guidance in selecting the appropriate erasure code parity level for your cluster.

Less erasure code partiy increase storage efficiency but decrease the disk failure tolerance. Choose it well for your use case.

1.5 Recommended Operating Systems¶

The official recommended OS is here: https://min.io/docs/minio/linux/operations/installation.html#minio-installation-platform-support

But in this tutorial, we use debian 11.

1.6 Clean existing Data¶

When starting a new MinIO server in a distributed environment, the storage devices must not have existing data.

Once you start the MinIO server, all interactions with the data must be done through the S3 API. Use the - MinIO Client: https://min.io/docs/minio/linux/reference/minio-mc.html#minio-client - MinIO Console: https://min.io/docs/minio/linux/administration/minio-console.html#minio-console - one of the MinIO Software Development Kits to work with the buckets and objects.

1.7 Create user account and group¶

By default, the minio.service runs as the minio-user User and Group. In this tutorial, we decide to use minio. You can create the user and group using the groupadd and useradd commands. The following example creates the user, group, and sets permissions to access the folder paths intended for use by MinIO. These commands typically require root (sudo) permissions.

sudo groupadd -r minio
sudo useradd -M -r -g minio minio

# change the owner of the data folder
sudo chown minio:minio /mnt/data

1.8 Create¶

2. Deploy the minio service on mode single node¶

2.1 Install the binary¶

# get the source
wget https://dl.min.io/server/minio/release/linux-amd64/archive/minio_20231120224007.0.0_amd64.deb -O minio.deb

# install the package
sudo dpkg -i minio.deb

2.2 Configure the systemd service file¶

The .deb or .rpm packages install the following systemd service file to /usr/lib/systemd/system/minio.service. For binary installations, create this file manually on all MinIO hosts.

The below file is an example, as we changed the user and group to minio. So we need to modify the User and Group.

[Unit]
Description=MinIO
Documentation=https://min.io/docs/minio/linux/index.html
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio

[Service]
WorkingDirectory=/usr/local

User=minio-user
Group=minio-user
ProtectProc=invisible

EnvironmentFile=-/etc/default/minio
ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/default/minio\"; exit 1; fi"
ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES

# MinIO RELEASE.2023-05-04T21-44-30Z adds support for Type=notify (https://www.freedesktop.org/software/systemd/man/systemd.service.html#Type=)
# This may improve systemctl setups where other services use `After=minio.server`
# Uncomment the line to enable the functionality
# Type=notify

# Let systemd restart this service always
Restart=always

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536

# Specifies the maximum number of threads this process can create
TasksMax=infinity

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=infinity
SendSIGKILL=no

[Install]
WantedBy=multi-user.target

# Built for ${project.name}-${project.version} (${project.name})

2.3 Configure MinIO’s environment file¶

Minio service uses env var to configure the runtime. By default, we put all environment variable that minio needs in the /etc/default/minio.

For the single node deployment, we only have one server

# hostnames
minio.casd.local
# all host has only one mount point
/mnt/data

3. Deploy the minio cluster on mode multi node¶

As we described before, our cluster has a single server pool consisting 3 servers with the below config:

# hostnames
minio-1.casd.local
minio-2.casd.local
minio-3.casd.local

# all host has only one mount point
/mnt/data

# The deployment has a load balancer running at https://datalake.casd.local that manages connections across all MinIO hosts.

Based on the above cluster information, we can write a file

# Set the hosts and volumes MinIO uses at startup
# The command uses MinIO expansion notation {x...y} to denote a
# sequential series.
#
# The following example covers four MinIO hosts
# with 4 drives each at the specified hostname and drive locations.
# The command includes the port that each MinIO server listens on
# (default 9000)

MINIO_VOLUMES="http://minio-{1...3}.casd.local:9000/mnt/data"

# Set all MinIO server options
#
# The following explicitly sets the MinIO Console listen address to
# port 9001 on all network interfaces. The default behavior is dynamic
# port selection.

MINIO_OPTS="--console-address :9001"

# Set the root username. This user has unrestricted permissions to
# perform S3 and administrative API operations on any resource in the
# deployment.
#
# Defer to your organizations requirements for superadmin user name.

MINIO_ROOT_USER=minioadmin

# Set the root password
#
# Use a long, random, unique string that meets your organizations
# requirements for passwords.

MINIO_ROOT_PASSWORD=minio-secret-key-CHANGE-ME

# Set to the URL of the load balancer for the MinIO deployment
# This value *must* match across all MinIO servers. If you do
# not have a load balancer, set this value to to any *one* of the
# MinIO hosts in the deployment as a temporary measure.
MINIO_SERVER_URL="https://datalake.casd.local:9000"

3.1 Run the minio server process¶

In cluster mode, you need to start the minio service on all nodes. The minio service detects that MINIO_VOLUMES has multiple nodes and multiple drives, it will then coordinate the communication between the nodes automatically.

sudo systemctl start minio.service

# you can check the output of the minio systemd with below command
sudo journalctl -f -u minio.service

3.2 Trouble shoot¶

If the minio service exit with error, you can debug the cluster based on the output of the minio service.

Check user and file access permission¶

As we mentioned in the systemd config, the user and group of the minio service is minio:minio. We must make sure that the permission of the mount point of the drives is correct. For example, run the below command if the permission and owner is not correct.

chown -R minio:minio /mnt/data

Check the hostname¶

As the minio cluster communication is through http. The hostname and /etc/hosts must be correctly set, so the nodes can communicate with each other

Empty the drive¶

If there are data inside the drive, the minio service can't format the disk. As a result, the minio service will not start

4. Test the cluster¶

If the minio service runs correctly. You can test the cluster

4.1 Access via minio console (web app)¶

You will notice that all minio nodes provides a web GU interface. You can access them via any browser. There is no notion of master and slaves. These nodes are all have the same roles.

http://10.50.5.73:9001/
http://10.50.5.74:9001/
http://10.50.5.75:9001/

4.2 Access via minio client (CLI)¶

You can follow this guide to install the minio client (mc).

After installation, you can use below command to create an alias

# general form
mc alias set <ALIAS> <HOSTNAME> <ACCESS_KEY> <SECRET_KEY>

# example
mc alias set myminio http://10.50.5.73:9000 admin changeMe

# check the connection
mc admin info myminio

5. Setup a load balancer¶

As minio provide an api access at port 9000 and a web access(minio-console) at port 9001. The nginx conf needs to two setup a load balancer for t

server {
    listen 80;
    server_name datalake-console.casd.local;

    # redirect http request to https
    return 301 https://$host$request_uri;

}


server {
    listen 443 ssl;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA';

    ssl_prefer_server_ciphers on;
    charset utf-8;
    # To allow special characters in headers
    ignore_invalid_headers off;

    # Allow any size file to be uploaded.
    # Set to a value such as 1000m; to restrict file size to a specific value
    client_max_body_size 0;
    # To disable buffering
    proxy_buffering off;

    ssl_certificate /etc/nginx/ssl/wildcard_casd.crt;
    ssl_certificate_key /etc/nginx/ssl/wildcard_casd.key;

    server_name datalake-console.casd.local;

    access_log /var/log/nginx/minio_acc.log;
    error_log /var/log/nginx/minio_err.log;

    location / {
                  charset utf-8;
                  proxy_set_header X-Real-IP $remote_addr;
                  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                  proxy_set_header X-Forwarded-Proto $scheme;
                  proxy_set_header Host $http_host;

                  proxy_connect_timeout 300;
                  # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
                  proxy_http_version 1.1;
                  proxy_set_header Connection "";
                  chunked_transfer_encoding off;

                  proxy_pass http://minio1.casd.local:9001;
}
}

setup a openldap¶

# general form
mc idp ldap policy attach <alias> <role> --user='<user-dn>'

# example
mc idp ldap policy attach minio consoleAdmin --user='uid=pengfei,ou=people,dc=casd,dc=org'