Install aptly

1. System Package repository Introduction

Before we start, we need to understand what is a linux system package repository. In Linux, the system package repository is a storage location hosted on remote servers from which the system retrieves and installs software and updates.

Each linux distribution has its own repository and package manager. For example, for Red Hat distribution, they uses yum/dnf as package manager, the default repo url is shown below

[base]
name=CentOS-7 - Base
mirrorlist=http://mirrorlist.centos.org/?release=7&arch=$basearch&repo=os&infra=$infra
#baseurl=http://mirror.centos.org/centos/7/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#released updates
[updates]
name=CentOS-7 - Updates
mirrorlist=http://mirrorlist.centos.org/?release=7&arch=$basearch&repo=updates&infra=$infra
#baseurl=http://mirror.centos.org/centos/7/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#additional packages that may be useful
[extras]
name=CentOS-7 - Extras
mirrorlist=http://mirrorlist.centos.org/?release=7&arch=$basearch&repo=extras&infra=$infra
#baseurl=http://mirror.centos.org/centos/7/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-7 - Plus
mirrorlist=http://mirrorlist.centos.org/?release=7&arch=$basearch&repo=centosplus&infra=$infra
#baseurl=http://mirror.centos.org/centos/7/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

For debian distribution, it uses apt-get/apt as package manager, and the default/standard repo url are:

deb http://deb.debian.org/debian bullseye main
deb-src http://deb.debian.org/debian bullseye main
deb http://security.debian.org/debian-security bullseye-security main contrib
deb-src http://security.debian.org/debian-security bullseye-security main contrib
deb http://deb.debian.org/debian/ bullseye-updates main contrib
deb-src http://deb.debian.org/debian/ bullseye-updates main contrib

You can find the repo list are in /etc/apt/sources.lits file and files under the /etc/apt/sources.list.d directory.

1.1 Repo list format

Below figure shows the format of each row in the source list.

Type

The term deb indicates that it is the repository of binaries, which are pre-compiled fiels. The term deb-src indicates that it is the repository of packages in source file format, which requires compilation in order to use it in the system.

Repository URL

The URL (HTTP, HTTPS, or FTP) represents the location of the repository from which you can download the packages.

Distribution

The next term is the short codename (i.e. Buster, Wheezy, Lenny, Jessie, Bullseye, etc.) of the release of your current system. The repo server may support multiple release, so we need to specify which release we want.

Component

The final term represents the categories of the Debian package. The available categories of the Debian distribution are : - main: This category contains packages that are released under a free license (BSD, GPL, etc.) and that meet the DFSG (Debian Free Software Guidelines). These packages also contain the source code within them, which can be modified and redistributed. - contrib: This category contains the packages that meet the DFSG (Debian Free Software Guidelines. The packages in the Contrib category are open-source packages, but depend on non-free packages to work. - non-free: This category contains the packages that do not meet the DFSG (Debian Free Software Guidelines). These packages have some strict license conditions that restrict the usage and redistribution of the software.

1.2 Add a repo by using the sources.list file

In below example, we add the virtualbox repo to the repo list. You can follow below steps to add a custom repository:

1. Download and import GPG keys of vbox. You can use below command. You may need to install gpg first

   wget -O- -q https://www.virtualbox.org/download/oracle_vbox_2016.asc | sudo gpg --dearmour -o /usr/share/keyrings/oracle_vbox_2016.gpg
   

2. Open the /etc/apt/sources.list file and add below line at the end of the file. deb [arch=amd64 signed-by=/usr/share/keyrings/oracle_vbox_2016.gpg] http://download.virtualbox.org/virtualbox/debian bullseye contrib. Or you can run below command

   echo "deb [arch=amd64 signed-by=/usr/share/keyrings/oracle_vbox_2016.gpg] http://download.virtualbox.org/virtualbox/debian bullseye contrib" | sudo tee /etc/apt/sources.list.d/virtualbox.list
   

3. Save and close
4. Update apt (sudo apt update)
5. Search the virtual box package with apt search virtualbox. You should see below output

   Sorting... Done
   Full Text Search... Done
   libvirt-daemon-driver-vbox/stable 7.0.0-3 amd64
     Virtualization daemon VirtualBox connection driver
   
   virtualbox-6.1/unknown 6.1.42-155177~Debian~bullseye amd64
     Oracle VM VirtualBox
   
   virtualbox-7.0/unknown 7.0.6-155176~Debian~bullseye amd64
     Oracle VM VirtualBox
   

2. Installing Aptly

You must not run below command with root user.

2.1 Install Aptly dependencies

Atply requrires below package: - bzip2: compression - gnupg: the gnu privacy guard is a complete and free implementation of the OpenPGP standard - gpgv: gpgv is actually a stripped-down version of gpg which is only able to check signatures. - xz-utils: is a general-purpose data compression software with a high compression ratio

sudo apt install bzip2 gnupg gpgv

2.2 Install Aptly

The package repo for aptly can be found here

Step1 : Download the public gpg key of Aptly

# New key per 2022-03-15 !
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys A0546A43624A8331

Step2 : Add aptly repo to sources.list

# -a means append which is important, without it the original content will be removed
echo "deb http://repo.aptly.info/ squeeze main" | sudo tee -a /etc/apt/sources.list

Don't worry about squeeze part in the repo name. I have tested it, it works for the debian 11 (bullseye).

Step3 : Update repo and install

sudo apt update
sudo apt install aptly

Step4 : Check your aptly install

aptly version

# you should see below output
aptly version: '1.5.0'

2.3 Create GPG pair

We will use a GPG key pair to sign the published repositories. If you don't have one, use below command to generate one.

# call the gpg key gen client
gpg --full-generate-key

# it will prompt below lines, just fill it with name, email
Real name: casd-debian
Email address: service@casd.eu
You selected this USER-ID:
    "casd-onyxia <casd-support@casd.eu>"

Change (N)ame, (E)mail, or (O)kay/(Q)uit? O

# then it will ask you to enter a passphrase to secure the private key

After you entered all the information, a folder .gnupg will be generated in the current folder. For more information about the GPG key management, you can check this page. Below are some commonly used command

# list secret keys
gpg --list-secret-keys

# list public keys
gpg --list-keys

# delet keys, to remove keys you need to delete the secret key first, then the public key
gpg --delete-secret-key
gpg --delete-key

Troubleshooting

The gpg agent is very bugy. And you may receive many error message. The most common solution is to restart the gpg agent by using below command

# kill the gpg-agent process will trigger the daemon to create a new one
gpgconf --kill gpg-agent

Verify the passphrase of the secret key

The simplest way is to use the change password feature of the secret key. It asks

gpg --passwd <secret-key-id>

3. Creating repo mirrors

Now we have all the necessary parts to create a local mirror of an official repo. Suppose we need to mirror bullseye (current stable 2023) for architecture amd64. Only main component is required and this repository is targeted for servers, not desktops. You can find local debian repo mirror per Country in this page. In below command, we choose the server in France.

# command 1
aptly mirror create -architectures=amd64 -filter='Priority (required) | Priority (important) | Priority (standard)' bullseye-main http://ftp.fr.debian.org/debian/ bullseye main

You should see this error ERROR: unable to fetch mirror: verification of detached signature failed: exit status 2 when you run above command. So we need to add the default debian keyring as trusted.

# command 2
gpg --no-default-keyring --keyring /usr/share/keyrings/debian-archive-keyring.gpg --export | gpg --no-default-keyring --keyring trustedkeys.gpg --import

Now rerun the command 1, you should see below output

Downloading http://ftp.fr.debian.org/debian/dists/bullseye/InRelease...
Success downloading http://ftp.fr.debian.org/debian/dists/bullseye/InRelease
gpgv: can't allocate lock for '/home/coder/.gnupg/trustedkeys.gpg'
gpgv: Signature made Sat 17 Dec 2022 10:15:20 AM UTC
gpgv:                using RSA key 0146DC6D4A0B2914BDED34DB648ACFD622F3D138
gpgv: Good signature from "Debian Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>"
gpgv: Signature made Sat 17 Dec 2022 10:15:21 AM UTC
gpgv:                using RSA key A7236886F3CCCAAD148A27F80E98404D386FA1D9
gpgv: Good signature from "Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>"
gpgv: Signature made Sat 17 Dec 2022 10:20:04 AM UTC
gpgv:                using RSA key A4285295FC7B1A81600062A9605C66F00D6C9793
gpgv:                issuer "debian-release@lists.debian.org"
gpgv: Good signature from "Debian Stable Release Key (11/bullseye) <debian-release@lists.debian.org>"

Mirror [bullseye-main]: http://ftp.fr.debian.org/debian/ bullseye successfully added.
You can run 'aptly mirror update bullseye-main' to download repository contents.

3.1 The filter flag

The flag -filter= allows us to cut down number of packages to download. First part, Priority (required) | Priority (important) | Priority (standard) is essential “base” Debian system.

We can also specify individual packages explicitly. For example, with below command, the mirror will add package such as: nginx, postgresql, etc.

aptly mirror create -architectures=amd64 -filter='Priority (required) | Priority (important) | Priority (standard) | nginx | postgresql' -filter-with-deps bullseye-main http://ftp.fr.debian.org/debian/ bullseye main

# we create also mirror for bullseye-updates and bullseye-security 
# you can notice the repo url is the same, but the distribution is `bullseye-updates`
# and the component is `main` 

aptly mirror create -architectures=amd64 -filter='Priority (required) | Priority (important) | Priority (standard) | nginx | postgresql' -filter-with-deps bullseye-updates http://ftp.fr.debian.org/debian/ bullseye-updates main

Flag -filter-with-deps instructs aptly to include dependencies of matching packages as well.

If filter is not specified, all packages would be included in the mirror and that would require more space and download size would be bigger.

In below example, we will mirror all the package of the debian security repo, because we have 0 filter in it.

# without filter, the mirror will take 20GiB 
aptly mirror create -architectures=amd64 bullseye-security http://security.debian.org/debian-security bullseye-security/updates main contrib non-free

# with filter 
aptly mirror create -architectures=amd64 -filter='Priority (required) | Priority (important) | Priority (standard) | nginx | postgresql' -filter-with-deps bullseye-security http://security.debian.org/debian-security bullseye-security/updates main contrib non-free

3.2 Check the created mirror

# bullseye-update mirror
aptly mirror create -architectures=amd64 bullseye-updates http://ftp.fr.debian.org/debian/ bullseye-updates main

# bullseye-security mirror
aptly mirror create -architectures=amd64 bullseye-security http://security.debian.org/debian-security bullseye-security/updates main contrib non-free


# get the full list of existing mirrors
aptly mirror list

# you should see below output
List of mirrors:
 * [bullseye-main]: http://ftp.fr.debian.org/debian/ bullseye

To get more information about mirror, run `aptly mirror show <name>`.

# get the detail of bullseye-main mirror
 aptly mirror show bullseye-main

# you should see below output
Name: bullseye-main
Archive Root URL: http://ftp.fr.debian.org/debian/
Distribution: bullseye
Components: main
Architectures: amd64
Download Sources: no
Download .udebs: no
Filter: Priority (required) | Priority (important) | Priority (standard)
Filter With Deps: no
Last update: never

Information from release file:
Acquire-By-Hash: yes
Architectures: all amd64 arm64 armel armhf i386 mips64el mipsel ppc64el s390x
Changelogs: https://metadata.ftp-master.debian.org/changelogs/@CHANGEPATH@_changelog
Codename: bullseye
Components: main contrib non-free
Date: Sat, 17 Dec 2022 10:14:37 UTC
Description:  Debian 11.6 Released 17 December 2022

Label: Debian
No-Support-For-Architecture-All: Packages
Origin: Debian
Suite: stable
Version: 11.6

3.3 Edit existing mirror

When you want to modify an existing mirror configuraiton, you need to run below command. You can find more information here

# note with below command, the old filter will be replaced, if you want to append, you need to do it
# manually 
aptly mirror edit -filter='nginx | postgresql' -filter-with-deps bullseye-main

# append example
aptly mirror edit -filter='Priority (required) | Priority (important) | Priority (standard) | nginx | postgresql' -filter-with-deps bullseye-main

3.4 Update(Synchronize) the mirror

Now we are ready to synchronize the local mirror with the official repo.

The doc and command aptly called it update, I found this is ambigue, so I use the word synchronize.

# this command will download all the filtered package to local server
# in ~/.aptly/pool/
aptly mirror update bullseye-main

# you should see below output
Downloading http://ftp.fr.debian.org/debian/dists/bullseye/InRelease...
Success downloading http://ftp.fr.debian.org/debian/dists/bullseye/InRelease
gpgv: can't allocate lock for '/home/coder/.gnupg/trustedkeys.gpg'
gpgv: Signature made Sat 17 Dec 2022 10:15:20 AM UTC
gpgv:                using RSA key 0146DC6D4A0B2914BDED34DB648ACFD622F3D138
gpgv: Good signature from "Debian Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>"
gpgv: Signature made Sat 17 Dec 2022 10:15:21 AM UTC
gpgv:                using RSA key A7236886F3CCCAAD148A27F80E98404D386FA1D9
gpgv: Good signature from "Debian Archive Automatic Signing Key (11/bullseye) <ftpmaster@debian.org>"
gpgv: Signature made Sat 17 Dec 2022 10:20:04 AM UTC
gpgv:                using RSA key A4285295FC7B1A81600062A9605C66F00D6C9793
gpgv:                issuer "debian-release@lists.debian.org"
gpgv: Good signature from "Debian Stable Release Key (11/bullseye) <debian-release@lists.debian.org>"
Downloading & parsing package files...
Downloading http://ftp.fr.debian.org/debian/dists/bullseye/main/binary-amd64/Packages.gz...
Success downloading http://ftp.fr.debian.org/debian/dists/bullseye/main/binary-amd64/Packages.gz
Applying filter...
Packages filtered: 58640 -> 325.
Building download queue...
Download queue: 325 items (168.48 MiB)

Current mirror contents are stored in the package database. You can update mirror at any moment as required.

4 Use snapshots

As you know, the package in the official repo can be updated anytime, but our mirror can't synchronize each modification in real time. We can use snapshot to release packages in a state

4.1 Create snapshots

It’s time take snapshots of the mirrors to preserve exact current mirror state. I will label snapshots after mirror name, applying version for the main bullseye mirror (i.e. Debian 11.6 -> bullseye ) and current date for frequently updated security and updates mirrors:

# the general form
aptly snapshot create <snapshot-name> from mirror <mirror-name>

# if you don't know the existing mirror name, use below command to get all available mirrors
aptly mirror list

# create a snapshot for mirror bullseye-main
aptly snapshot create bullseye-main-11.6 from mirror bullseye-main

# create a snapshot for mirror bullseye-updates
aptly snapshot create bullseye-updates-20230206 from mirror bullseye-updates

# create a snapshot for bullseye-security
aptly snapshot create bullseye-security-20230206 from mirror bullseye-security

4.2 Merging snapshots

Releasing the snapshop one by one is not very productive. We can merge the three snapshoot into one by using below command.

# general form
# -latest option chooses merge strategy: package with latest version “wins”.
# it can take as many sub snapshot as possible
aptly snapshot merge -latest <snapshot-name> <sub-snapshot-name1> <sub-snapshot-name2> ...

# if you don't know your snapshot name, you can use below command to get them
aptly snapshot list

# our example
aptly snapshot merge -latest bullseye-stable-20230206 bullseye-main-11.6 bullseye-security-20230206 bullseye-updates-20230206

Let's check the merged snapshot, for example the latest version of package nginx should came from the security mirror, because the there is a security update latly.

We can use below command to track the package origin

aptly package show -with-references 'Name (nginx)'

# you should see below output
Package: nginx
Priority: optional
Section: httpd
Installed-Size: 102
Maintainer: Debian Nginx Maintainers <pkg-nginx-maintainers@alioth-lists.debian.net>
Architecture: all
Version: 1.18.0-6.1+deb11u3
Depends: nginx-core (<< 1.18.0-6.1+deb11u3.1~) | nginx-full (<< 1.18.0-6.1+deb11u3.1~) | nginx-light (<< 1.18.0-6.1+deb11u3.1~) | nginx-extras (<< 1.18.0-6.1+deb11u3.1~), nginx-core (>= 1.18.0-6.1+deb11u3) | nginx-full (>= 1.18.0-6.1+deb11u3) | nginx-light (>= 1.18.0-6.1+deb11u3) | nginx-extras (>= 1.18.0-6.1+deb11u3)
Filename: nginx_1.18.0-6.1+deb11u3_all.deb
Size: 92936
MD5sum: 47deabf24cd33a9440782b61f30bc2e9
SHA1: 320d46db1c47ef6f7ccc1c7d3685403615127b33
SHA256: 795f27bbd556a60e132e6f779b78105969009d129e727ab60f826e1bf4320365
SHA512: 47e69866c8e829f60b05d7074c28c89a21c09a3d3e6527192882f2c1f63abfafd8a3ee2f45ab1bb381508eb7f17e5663656b05da4efdfdb3ed129c467aa112ca
Description: small, powerful, scalable web/proxy server
Description-Md5: 902443ddbee17249123a068e7ca7c6d8
Homepage: https://nginx.net

References to package:
  mirror [bullseye-security]: http://security.debian.org/debian-security/ bullseye-security/updates
  mirror [bullseye-main]: http://ftp.fr.debian.org/debian/ bullseye
  snapshot [bullseye-main-11.6]: Snapshot from mirror [bullseye-main]: http://ftp.fr.debian.org/debian/ bullseye
  snapshot [bullseye-stable-20230206]: Merged from sources: 'bullseye-main-11.6', 'bullseye-security-20230206', 'bullseye-updates-20230206'
  snapshot [bullseye-security-20230206]: Snapshot from mirror [bullseye-security]: http://security.debian.org/debian-security/ bullseye-security/updates

in the references to package block, you can notice the package origin is from snapshot bullseye-security-20230206 (last row).

5 Publish Repository

Now we can publish our snapshot bullseye-stable-20230206 as Debian repository, which will be consumed by apt-get client.

Use below command to publish the snapshot

# general form
# the -distribution option: specifies the distribution name of the repo, which is mandatory
aptly publish snapshot -distribution=<distribution-name> <snapshot-name>


# our example
aptly publish snapshot -distribution=bullseye bullseye-stable-20230206

The above command will publish the repository to ~/.aptly/public/ directory. You can start any HTTP server to serve this directory as static files, or use aptly built-in webserver for testing.

# run aptly built-in webserver (for testing only, not recommended for prod)
aptly serve

5.1 Check published snapshot dependencies

Below command will generate a graph which shows the dependencies of the published snapshot

# this command requires graphviz
# -output="": specify output filename, default is to open result in viewer
# -format="png": graph output format, could be anything graphviz supports, e.g. png, pdf, svg, …
aptly graph -output="/tmp/package_build_history.png" -format="png"

# install graphviz
sudo apt install graphviz

5.2 Serve the published via a web server

For testing purpose, the built-in webserver is enough. But For production, we need to install a real webserver. In this tutorial, we recommend nginx, because it has better performence for service static content and consum less resources compare to Apache2.

You can check this tutorial to install nginx

After the installation of nginx, you need to create a virtual host. To do so, create a conf file under /etc/nginx/sites-available/. In our tutorial, I named the conf file as deb-repo.conf

sudo vim /etc/nginx/sites-available/deb-repo.conf

# add below content in it
# you need to modify the server name and the root path 
server {
    server_name deb.casd.local;

    root /package-repo/aptly/.aptly/public;

    location / {
        autoindex on;
        try_files $uri $uri/ =404;
    }
}

# check the correctness of your conf syntax
sudo nginx -t

To enable this conf, you need to create a sym link in /etc/nginx/sites-enabled/

sudo ln -s /etc/nginx/sites-available/deb-repo.conf /etc/nginx/sites-enabled/

# you may need to restart the nginx server

sudo systemctl restart nginx

We do not recommend to add ssl to transfer http to https. Because the server identity and package integrity is ensured by the apt GPG protocol. And the package is public data, no need to ensure confidentiality. The ssl will only add some performence overhead without any utility.

6. Using the private Repository

6.1 Prepare the GPG (GNU privacy guard) key

First I need to import public part of the GPG key that was used to sign the repository into trusted apt keyring on target machine.

The file extenion .asc indicates this public key is ASCII armored key. The extension .gpg indicates the key is in binary OpenPGP format. It is very important for the clients when they import the key as trusted gpg key into their key store.

# run from the server which has aptly 
gpg --export --armor > repo_gpg_key.asc

# the generated file should look like this
-----BEGIN PGP PUBLIC KEY BLOCK-----

mQGNBGPX2GABDACnY9HpMwewenBppirK3vZ4/JiKtzevZCwDMlx6OjKGcnfDKMVU
/jWWoncIgZ3TxH8nEy3AgV109wy9ztX5kBa1IUgwdnLwpWepvGIKlWCCk9g2dEOM
VJFlNcyl3pVUxbjov7jvcx/epB/79RO8IkRjjtLhDbiH5Wh07+gSfbIdDNXqT7av
bnt1ZJNOo7+zRVJ5bgxbEF0q+sp4ESUtIxD8wfgV6N3dlzIgrA5IgLS4qt/0OWoC
IdSf9Kq5LD+FXYJ9Q0WrbDC4G0zBQuYLf/eaAMkPnGBDTHLSWdGXL02jQQdZb9bq
H/Zdunm+akngiOc1vAozXMECtZMVvdZ0GopJldZ0oofVRVSiRSimctpGlQ95BGU8
+/cDV174mtTvnw5RIsi9lsyif/ab11mt+dthJFt4/wcppEEhMLKqF/UTTshpSnes
GAsEabdlv9kJu2gwJ79ftUgBDdbFp9d4unJoo2ui4ejuyJi1CXQdcKFaTIWRNolu
cRWFdlJWvmKi++0AEQEAAbQiY2FzZC1vbnl4aWEgPGNhc2Qtc3VwcG9ydEBjYXNk
LmV1PokB1AQTAQoAPhYhBF4/+09JAtKGdNdhUtXMNPl62Y49BQJj19hgAhsDBQkD
wmcABQsJCAcCBhUKCQgLAgQWAgMBAh4BAheAAAoJENXMNPl62Y49I1IL/Rs0lmsu
LktQ3fEwGjFYTBtU4a659Fb4QG6l65msiRtLH3jKOdng3B/CwRE71yzbv3MgQntF
Myp7PKQhdD32hTKM0d8n8Fbj05qkq+SjoRmhnKH8gVpUv+fduEORSMpGqCYh+EI2
9pw9Vkl5BWZFrx7KuXzatBShA1ZAF1v+kIRe8fFtr63laoyKhxM3rvtHJk/wFQe2
b8Vy2ZyI1ez48WjoYJTo/S9VeX6x46FcA4KEvAbxcO2XcmOdQqmGl7d234HpYOns
UDM/bmwqJgV+4J4Dq2a4WSkpObMUIMns99mTFCCJ6J43YftjMZ12AFjxmAgMx5g4
FbdAsFhtD3WXsHcAMrbjbEdom00LiCkCBSzrHrPoT+mFM7D4tX7tbKKBg7oV+na7
fICb5eIfqpAQaHjLlVelWS4K4vLT7LSDAgJH0WrjtOiz1idZK6Gv6DBLc0lOmimB
xmcpVoHCEWs7RScjaMNnDdJsUx7yGA081AHsoo5Xaig1C37N9SYCBFMAELkBjQRj
19hgAQwAqZ1LN0bjGGGaQq9Cgvffa+ROP75Ss0KjET1w++Ng2MkmxZCWdLv84qp9
u6MEM6wxtL3wKi2Vwk5QGvfVi5AvOgRo0D15CtI8JKDZlnVGYiJ9f5Xl1aKubV0t
ZJSM2yb/J5vg/MJ2TTKHYeNgnvFANkWrDeyhxCEfh6qUcY1nTuxkbmVTpw4YAFpI
5WQuqtuI/94tJ6kd5P5AWCuVrZ88/Je43EGUhfaAA9353oHsz+r12FH1Bj1V+B9X
dOSxemTZdb9zkpcKtlXSiGwfJWaPXABJmcKZ3T1LIRC0dunTLaEeoN7AlfQEuU5c
prsi35SPBoo0sIKgUgzqGN7L89fBx2yQ4kHSzmhQUSI/F9sqdkMHxKWddyM+qX8O
8uBILun+SG76NVZRoC+u3DsSwkfdvpVq/SOEOIaVWlWLj1yVIf7/aP1fYLCWs8B4
x1YuMuDd36mswOINjceC+n0Khewfc9iB6VO/CZaKfoC2BegbMtsQ3FX010rPKgb5
gIHIwleNABEBAAGJAbwEGAEKACYWIQReP/tPSQLShnTXYVLVzDT5etmOPQUCY9fY
YAIbDAUJA8JnAAAKCRDVzDT5etmOPUYsC/9x/qfOQamOlB/ShmrajEDCB7KaR3/n
Lb8ihC9GDXx0wJVEshpd9F3mQN+0/P05hRVj6AXJNHfyu0WcBjqtP1YCQKycK9FM
I3xhvWsfzQQihw0r+a2DjXF4zWZwO9bQincbypHGNzDpiKC2FCR+Ciuijqm9EhIP
knqbSvPHMQzuxiFdr+jVF+jpq3TEKeGVZzBIJhQJZzCrRjTn3op65nqcoGodJgOg
vO4tHEHHeEG9ws9PR1L8FUO362C9/5xs/CziN6UppBN1U7MjLTuMMk2DLnCX8Q4N
/fV3q/MbYYCHyzVCoaOoZuCHl+uN4cbS3JulnZjEC9XPtUtQgVpIS3Xpak+c8Vku
F5+rdWftPzc53+DgFqOVHO1gci2B6Xpc6pMzaQ7UJF1SxZQid0VbsmBK88K41oQz
Jzfk329lUq4D5cYFt//hJ2Fk6CS3AHkyrV2zGoZZ65BZ8OnYIavjGs2A7FOYZxo6
AvQWBa75d4ySlyqQHh9wHJ0V5Ba7X/kePzM=
=s9av
-----END PGP PUBLIC KEY BLOCK-----

6.2 Copy the public key to target machine

There are two solutions for adding the the gpg key of the private repo: Solution 1 : Use apt-key tool Solution 2 : Without tool

Solution 1:

This solution is deprecated, so we don't recommend this solution.

# down the public key from the deb server
wget http://deb.casd.local/casd_gpg_key.asc

# apt-key requires gnupg package, if you don't have it you need to install it
sudo apt install gnupg 

# run from the target server which consume apt repository
# add our GPG as trusted key
sudo apt-key add casd_gpg_key.asc

Solution 2:

We recommand this solution, and the client does not need to install any extra package.

# download the key and add it to the key store
# -q activate quiet mode to hide output
# -O- means output to std out.
wget -qO- http://deb.casd.local/casd_gpg_key.asc | sudo tee /etc/apt/trusted.gpg.d/casd_gpg_key.asc

Change apt source repo list

Now we need to edit the /etc/apt/sources.list file and add our repo as a source repo. Here suppose our repo server url is http://deb.casd.local, your source.list should look like below file

# comment the default sources
# deb http://deb.debian.org/debian bullseye main
# deb http://security.debian.org/debian-security bullseye-security main
# deb http://deb.debian.org/debian bullseye-updates main

# add our repo
deb http://deb.casd.local/ bullseye main

Save and exit, then you need to update the apt-get repo cache

# update repo cache
sudo apt-get update

# update the installed packages
sudo apt-get upgrade

As this repository has been published from snapshot, it would never change until it is update to new snapshot. Good thing is that I can setup all my machine to use this repo and get identical set of packages installed. Bad thing is that I need to maintain and update my repo as updates are coming, but if I have many machines, the advantage of predictable upgrades outweighs the maintenance costs.

7. Upgrading repository

Several days after you published your repository, when you update your mirrors again. And you discover there are new security updates. You need to follow below steps to build a new release

# update mirror
aptly mirror update bullseye-main
aptly mirror update bullseye-updates
aptly mirror update bullseye-security 

# create new snapshot
aptly snapshot create bullseye-main-11.6 from mirror bullseye-main
aptly snapshot create bullseye-updates-<date> from mirror bullseye-updates
aptly snapshot create bullseye-security-<date> from mirror bullseye-security

# merge the snapshot
aptly snapshot merge -latest bullseye-stable-<date> bullseye-main-11.6 bullseye-updates-<date> bullseye-security-<date> 

# publish the new release
# when there is already a release in use, you need to add switch to replace the old release
aptly publish switch bullseye bullseye-stable-<date>

# generate the build graph
aptly graph -output="/pathToAptly/package_build_history_<date>.png" -format="png"

# if you don't need the old release anymore, you can drop them
aptly db cleanup

8. Serve the packages

Although you can run aptly serve to publish the packages via port 8080 for test purpose. It's recommended to set up a web server such as Nginx or apache2

9. Cron

There is a cron job script which try to update the mirror and provide release. https://github.com/gvogets/aptly-scripts/blob/master/cronjob

We find the cron job is not adapted for our need. So we right our own cron job scripts: - A release cron job: publish aptly snapshots automatically - A clean cron job :

9.1 Aptly release cron job

The below cron job aptly_build_release.bash is designed to publish aptly snapshots automatically

#!/bin/bash

# if we put this script in cron.weekly, this script will be executed with root privilege. And the aptly db data is user specific.
# So run as root can not get the information about the mirror and snapshoot. So for the aptly command, we need to use sudo -u aptly
# to run the command as user `aptly`.
# script config
uid=changeMe
dist_name=bullseye
root_path=/path/to/.aptly
log_file_path="${root_path}/log/aptly_log.out"
gpg_pwd_file_path="${root_path}/creds/pwd"
repo_path="${root_path}/public"
export_repo_path="${root_path}/export"
release_date="$(date '+%Y%m%d')"
snapshot_merge_list=()

# set bash script abort on the first command fails, if 
set -eo pipefail

# Saves file descriptors so they can be restored to whatever they were before 
#redirection or used themselves to output to whatever they were before the following redirect.
exec 3>&1 4>&2
# Restore file descriptors for particular signals. Not generally necessary since they 
# should be restored when the sub-shell exits.
trap 'exec 2>&4 1>&3' 0 1 2 3
# Redirect stdout to file log.out then redirect stderr to stdout. Note that the order is 
# important when you want them going to the same file. stdout must be redirected 
# before stderr is redirected to stdout.
# By default, the log will be appended to the log file. If you only want to keep the log
# of the last run, replace >> by >
exec 1>>"${log_file_path}" 2>&1
# Everything below will go to the file 'aptly_log.out':

# clean the old graph
rm -f ${repo_path}/*.png

# clean last release zip symoblic link
rm -f ${repo_path}/*.zip


# read the gpg private key password from a file
gpg_pwd=$(<"${gpg_pwd_file_path}")

# update all existing mirror
sudo -u $uid aptly mirror list --raw | xargs -n1 sudo -u $uid aptly mirror update


# create a list of mirror
REPOS=$(sudo -u $uid aptly mirror list --raw | cut -d '#' -f1 | sort | uniq | xargs)

# loop through the mirror list, create new snapshot for each mirror
# build a snapshot merge list
for REPO in $REPOS; do
    sudo -u $uid aptly snapshot create "${REPO}-${release_date}" from mirror "${REPO}"
    snapshot_merge_list+=("${REPO}-${release_date}")
done

echo "${snapshot_merge_list}"


# merge the snapshot
sudo -u $uid aptly snapshot merge -latest "${dist_name}-stable-${release_date}" "${snapshot_merge_list[@]}"

# publish the new release
# when there is already a release in use, you need to add switch to replace the old release
# we need to put -batch=true when run this script in cron, otherwise, aptly and gpg will use the same /dev/tty. And this will cause 
# the -passphrase option fail, and a promt will show up. 
sudo -u $uid aptly publish switch -batch=true -passphrase="${gpg_pwd}" "${dist_name}" "${dist_name}-stable-${release_date}"

# generate the build graph
sudo -u $uid aptly graph -output="${repo_path}/package_build_history_${release_date}.png" -format="png"

# if you don't need the old release anymore, you can drop them
sudo -u $uid aptly db cleanup

# zip the release and put it to the export repo
sudo -u $uid zip -r "${export_repo_path}/${dist_name}-${release_date}.zip" "${repo_path}"

# create a symbolic link
sudo -u $uid ln -s "${export_repo_path}/${dist_name}-${release_date}.zip" "${repo_path}/${dist_name}-${release_date}.zip"

9.1.1 Script configuration

The first section of this script is the configuration.

• uid: The uid of the user which you want to run the aptly script
• dist_name: The name of the Debian distribution which you want to publish
• root_path: The root folder path of the aptly app. This path will be different if you follow an other installation guide. If you follow my guide. It should be located at ~/.aptly. By default, it should contain three folders db, pool and public. To make the corn script work, you must create three new folders(i.e. log, creds, and public) under the root_path.
• log_file_path: The path of the log file of the cron job
• gpg_pwd_file_path: The path of the file which contains the password of the GPG private key
• repo_path: This folder will host the source of the deb package, and it will be used by the webserver to publish the package
• export_repo_path: This folder contains the zip file of all the publish deb packages. Each zip file can be used as an export of a release.

The aptly root path folder should have the following shape:

.aptly/
├── creds
├── db
├── export
├── log
├── pool
└── public

• In creds folder, it contains the passphrase of the GPG private key. This key is used to sign all the packages that are published by you. The default file name is pwd. If you change the file name, you need to change the config too in the cron script.
• In db folder, it contains the database of the aptly which stores the state of the aptly (e.g. mirror list, snapshot list, published package list, etc.)
• In export folder, it contains a zip file of all published packages. This zip file can be copied and deployed on another web server. The web server can act as a deb repo afterward.
• In log folder, it contains a log file of the cron script execution
• In pool folder, it contains all the packages of the available mirrors
• In public folder, it contains all the publish packages of the current deb repo.

9.1.2 Handling New emerging mirrors

By default, the script will consider all mirrors available in the aptly, update it, create a snapshot, build a merge of all snapshots and publish the merged snapshot.

If you want to add a new mirror and enable the mirror update via the corn job, you only need to create a mirror and update the mirror. You don't need to modify the above script

For example, we want to add two more mirrors: - docker: - k8s

Create Mirror for k8s

# add gpg key of k8s repo as trusted key
gpg --no-default-keyring --keyring trustedkeys.gpg --keyserver keyserver.ubuntu.com --recv-keys B53DC80D13EDEF05

# create mirror
aptly mirror create -architectures=amd64 k8s-main  http://apt.kubernetes.io/ kubernetes-xenial main

# update mirror
aptly mirror update k8s-main

Create Mirror for docker

# add gpg key for the containerd.io repo
gpg --no-default-keyring --keyring trustedkeys.gpg --keyserver keyserver.ubuntu.com --recv-keys 7EA0A9C3F273FCD8

# create mirror
aptly mirror create -architectures=amd64 docker-main http://download.docker.com/linux/debian/ bullseye stable

# update mirror
aptly mirror update docker-main

9.2 Cron job for cleaning old release

The snapshot and the generated zip files will not be cleaned automatically. As a result, every week we will have at least 4 more snapshot in the db and a zip file which cost 100GB. So we need to clean them at a point to make the db clean and save some disk spaces. Two solution is possible:

• Add a task in the current cron job, and clean the old snapshots after publishing a new snapshot. Pros: The aptly db only contains currently published snapshots, so db is very clean. Cons: All history snapshots are removed. So unable to rollback in case of problems.
• Create a new cronjob that cleans the snapshot. Pros: Can have a different run policy. So we can run it monthly, and we keep 4 weekly snapshots for rollback when needed.

For now, I choose to use solution 2. The below cron job clean_snapshot.bash which cleans the snapshot

# script config, you can change the days to set the snapshot interval
days=30
# you need to change this value to the uid who install aptly
uid=changeMe
root_path=/path/to/.aptly
log_file_path="${root_path}/log/aptly_log.out"
export_repo_path="${root_path}/export"


day_in_sec=86400
interval=$((days * day_in_sec))

# setup log 

exec 3>&1 4>&2
trap 'exec 2>&4 1>&3' 0 1 2 3
# By default, the log will be appended to the log file. If you only want to keep the log
# of the last run, replace >> by >
exec 1>>"${log_file_path}" 2>&1

# get the list of the snapshot
SNAP_SHOTS=$(sudo -u $uid aptly snapshot list --raw | cut -d '#' -f1 | sort | uniq | xargs)

# for each snapshot check if it is older than the given day or not. If yes, then delete
for SNAP in $SNAP_SHOTS; do
   echo "Examing the snapshot: $SNAP"
   # delete the longest match of pattern "-" from the beginning
   release_date=${SNAP##*-}
   # convert the string date to the numeric timestamp, here we choose %s (seconde) as prcision. 
   # It can be %m (month), %d (day)
   # current date timestamp
   cd_timestamp=$(date --date "$date" +'%s')
   # release date timestamp
   rd_timestamp=$(date --date "$release_date" +'%s')
   # get the expected timestamp
   expected_timestamp=$((cd_timestamp - interval))
   # compare it
   if [ $rd_timestamp -lt $expected_timestamp ]
   then
      echo "INFO: snapshot $SNAP is older than $days days" >&2
      sudo -u $uid aptly snapshot drop -force $SNAP
   fi
done

# clean the zip file older than the given day
find "${export_repo_path}" -name "*.zip" -type f -mtime +"${days}" -delete

Extra repo mirror for k8s

k8s-main

# add gpg key of k8s repo as trusted key
gpg --no-default-keyring --keyring trustedkeys.gpg --keyserver keyserver.ubuntu.com --recv-keys B53DC80D13EDEF05

# create mirror
aptly mirror create -architectures=amd64 k8s-main  http://apt.kubernetes.io/ kubernetes-xenial main

# update mirror
aptly mirror update k8s-main

containerd.io

This mirror contains the packages of containerd (container runtime).

# add gpg key for the containerd.io repo
gpg --no-default-keyring --keyring trustedkeys.gpg --keyserver keyserver.ubuntu.com --recv-keys 7EA0A9C3F273FCD8

# create mirror
aptly mirror create -architectures=amd64 docker-main http://download.docker.com/linux/debian/ bullseye stable

# update mirror
aptly mirror update docker-main