July 3, 2017 David Blaskow

Package manager tools for Debian like

Introduction

The needs are simple, create local repository and mirror the official one for Debian like systems.

RedHat like systems got multiple, Debian like, well not so much.

In this case we need to manage Ubuntu servers, well Canonical has Landscape, but we want to keep it simple.

The possibilities

For Debian Like there’s 2 possibilities :

  • aptly which can seems to be integrate with ansible
  • pulp_deb which is a community project to add to an installation of Katello/Pulp

Pulp_deb

Can be found here : https://github.com/pulp/pulp_deb

This project seemed at first sight what we need, it’s had deb repository package management to pulp which already manage rpm, so it could be wonderful in a heterogenous environment.

But let’s get back to the reality quickly, this is not really interesting for a enterprise, let’s see why :

– hosted source wasn’t moved with the change of the fedorahost change, you need to get it back from git, and change the spec file

– packages version needed to the installation of the generated RPMs are only available to Fedora, if you want to use it with RHEL or CentOS, you’ll create a monster

– need to change some parameters in the spec file, in order to use the python module

– If you pass those steps, there’s a problem with a SSL parameter which not working well

Well, time is precious and since we work to get the most stable environments, we won’t go with this tools right now.

Aptly

Can be found here : https://www.aptly.info/ but can also be installed from Ubuntu fresh installed repository.

Installation

Seems pretty easy to install it :

# apt-get install aptly apache2

With the coming of Ubuntu 16.04 and Debian 9, the SHA1 will not be used anymore, you’ll need to install the latest version published on https://www.aptly.info/ to get it right for this new version.

Configuration

By default aptly will use configuration file created in your home, in order to maintain all configuration file in the same place, move it to /etc/ :

# mv .aptly.conf /etc/aptly.conf

In this file you’ll need to change the rootdir to a place you want your packages and repository definition (somewhere with space if you want to mirror the whole ubuntu repository), in my case it will be /opt/

"rootDir""/opt/aptly",

Now we can configure the web server to serve the packages to client :

I’ve created a virtualhost name aptly-test by creating the file /etc/apache2/sites-available/aptly-test.conf, and set this configuration :

<VirtualHost *:8040>
 ServerAdmin toto@poc.local
 DocumentRoot /opt/aptly/public/
 <Directory /opt/aptly/public/>
 Options +Indexes +FollowSymLinks
 Require all granted
 </Directory>
 LogLevel info
 CustomLog /var/log/apache2/aptly-access.log combined
 ErrorLog /var/log/apache2/aptly-error.log
 ServerSignature On
</VirtualHost>

Add the port 8040 to the file /etc/apache2/ports.conf as is :

# If you just change the port or add more ports here, you will likely also# have to change the VirtualHost statement in# /etc/apache2/sites-enabled/000-default.conf
 Listen 80
 Listen 8040
...

Then I enable the virtualhost :

# a2ensite aptly-test.conf

Virtualhost is added to the enable site, but we need to reload the apache configuration :

# systemctl reload apache2

Since package installation can destroy your servers we need to validate that the packages we installed are valid, to do so we need to create a GPG key to sign the packages.

# gpg --gen-key
gpg (GnuPG) 1.4.18; Copyright (C) 2014 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
gpg: directory `/root/.gnupg' created
gpg: new configuration file `/root/.gnupg/gpg.conf' created
gpg: WARNING: options in `/root/.gnupg/gpg.conf' are not yet active during this run
gpg: keyring `/root/.gnupg/secring.gpg' created
gpg: keyring `/root/.gnupg/pubring.gpg' created
Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048)
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y
You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"
Real name: Aptly Signing Key
Email address: aptly.poc.local
You selected this USER-ID:
    "aptly <aptly.poc.local>"
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key.
You don't want a passphrase - this is probably a *bad* idea!
I will do it anyway.  You can change your passphrase at any time,
using this program with the option "--edit-key".
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
..........+++++
.......+++++
Not enough random bytes available.  Please do some other work to give
the OS a chance to collect more entropy! (Need 99 more bytes)
..+++++
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key 071B4856 marked as ultimately trusted
public and secret key created and signed.
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   2048R/071B4856 2016-01-10
      Key fingerprint = E80A D275 BAE1 DEDE C191  196D 078E 8ED8 071B 4856
uid                  aplty <aptly.poc.local>
sub   2048R/FFF787F6 2016-01-10

Creation is ok, but we need to create a public key for the client :

# gpg --export --armor 071B4856 > aptly-pubkey.asc

Repository creation

First thing to do is to create a repository to put it our own repository :

# aptly repo create -distribution="xenial" -architectures="amd64,i386,all" -component="main" locarepo
Local repo [locarepo] successfully added.
You can run 'aptly repo add locarepo ...' to add packages to repository.

We can  then add the packages we want to add :

# aptly repo add locarepo vlc_2.2.4-4ubuntu0.16.10.2_amd64.deb
Loading packages...
[+] vlc_2.2.4-4ubuntu0.16.10.2_amd64 added

Check if everything is OK :

# aptly repo show locarepo
Name: locarepo
Comment:
Default Distribution: xenial
Default Component: main
Number of packages: 1

Publish the repository

Since we have  created our local repository we can publish it :

# aptly publish repo -architectures="amd64,i386,all" locarepo
Loading packages...
Generating metadata files and linking package files...
Finalizing metadata files...
Signing file 'Release' with gpg, please enter your passphrase when prompted:
Clearsigning file 'Release' with gpg, please enter your passphrase when prompted:Local repo locarepo has been successfully published.
Please setup your webserver to serve directory '/opt/aptly/public' with autoindexing.
Now you can add following line to apt sources:
 deb http://your-server/ xenial main
Don't forget to add your GPG key to apt with apt-key.You can also use `aptly serve` to publish your repositories over HTTP quickly.

Client configuration

For the client there is 2 actions to do, add the key we’ve created previously, and change the sources.list file to get our repository

First thing first, the key that we previously copy to the client:

# apt-key add aptly-pubkey.asc

Then we can change the repository :

# mv /etc/apt/sources.list /etc/apt/sources.list.old
# echo "deb http://172.20.10.5:8040/ xenial main" > /etc/apt/sources.list

Now we can test :

# apt-get update
Get:1 http://172.20.10.5:8040 xenial InRelease [6,147 B]
Get:2 http://172.20.10.5:8040 xenial/main amd64 Packages [1,455 B]
Fetched 7,602 B in 0s (112 kB/s)
Reading package lists... Done
# apt-cache policy vlc
vlc:
 Installed: (none)
 Candidate: 2.2.4-4ubuntu0.16.10.2
 Version table:
 2.2.4-4ubuntu0.16.10.2 500
 500 http://172.20.10.5:8040 xenial/main amd64 Packages

Well we have our package from our repository.

Snapshots

Since we have a working repository now, let’s play with snapshot.

Snapshot are really useful when you have mirrors, mirrors will be update every or so, but sometimes your environment can update this fast.

By creating snapshot, you’ll keep your client repository the same even if there is packages change on  the main repo.

This way you can also create updated repositories for different envirnement : test, prod, …

Before create our snapshot we can remove the publication of our previous repo :

# aptly publish list
Published repositories:
 * ./xenial [amd64] publishes {main: [locarepo]}
# aptly publish drop xenial
Removing /opt/aptly/public/dists...
Removing /opt/aptly/public/pool...Published repository has been removed successfully.

Now we can create our snapshot :

# aptly snapshot create -architectures="amd64,i386,all" locarepo20170630 from repo locarepo
Snapshot locarepo20170630 successfully created.
You can run 'aptly publish snapshot locarepo20170630' to publish snapshot as Debian repository.

Publish the snapshot :

# aptly publish snapshot -architectures="amd64,i386,all" locarepo20170630
Loading packages...
Generating metadata files and linking package files...
Finalizing metadata files...
Signing file 'Release' with gpg, please enter your passphrase when prompted:
Clearsigning file 'Release' with gpg, please enter your passphrase when prompted:Snapshot locarepo20170630 has been successfully published.
Please setup your webserver to serve directory '/opt/aptly/public' with autoindexing.
Now you can add following line to apt sources:
 deb http://your-server/ xenial main
Don't forget to add your GPG key to apt with apt-key.You can also use `aptly serve` to publish your repositories over HTTP quickly.

Check on the client :

# apt-get update
Get:1 http://172.20.10.5:8040 xenial InRelease [6,147 B]
Get:2 http://172.20.10.5:8040 xenial/main amd64 Packages [1,455 B]
Fetched 7,602 B in 0s (112 kB/s)
Reading package lists... Done
# apt-cache policy vlc
vlc:
 Installed: (none)
 Candidate: 2.2.4-4ubuntu0.16.10.2
 Version table:
 2.2.4-4ubuntu0.16.10.2 500
 500 http://172.20.10.5:8040 xenial/main amd64 Packages

now add a new package to the repo :

# aptly repo add locarepo steam_latest.deb
Loading packages...
[+] steam-launcher_1.0.0.54_all added

check the repository :

# aptly repo show locarepo
Name: locarepo
Comment:
Default Distribution: xenial
Default Component: main
Number of packages: 2

We can see 2 packages now, let’s check on the repository, let’s create a new snapshot to publish :

# aptly snapshot create -architectures="amd64,i386,all" locareponewsnap from repo locarepo
locareponewsnap successfully created.
You can run 'aptly publish snapshot locareponewsnap' to publish snapshot as Debian repository.

We can publish this new snapshot but with another distribution name, since it will be in conflict with the previous one which we wan to keep (for production platform for example) :

# aptly publish snapshot -architectures="amd64,i386,all" -distribution=xenial2 locareponewsnap
Loading packages...
Generating metadata files and linking package files...
Finalizing metadata files...
Signing file 'Release' with gpg, please enter your passphrase when prompted:
Clearsigning file 'Release' with gpg, please enter your passphrase when prompted:Snapshot locareponewsnap has been successfully published.
Please setup your webserver to serve directory '/opt/aptly/public' with autoindexing.
Now you can add following line to apt sources:
 deb http://your-server/ xenial2 main
Don't forget to add your GPG key to apt with apt-key.You can also use `aptly serve` to publish your repositories over HTTP quickly.

As you may see it the link for the apt source is not the same, so we need to change it on the client configuration :

# echo "deb http://172.20.10.5:8040/ xenial2 main" > /etc/apt/sources.list
# apt-get update
Get:1 http://172.20.10.5:8040 toto InRelease [6,927 B]
Get:2 http://172.20.10.5:8040 toto/main amd64 Packages [2,085 B]
Get:3 http://172.20.10.5:8040 toto/main i386 Packages [731 B]
Get:4 http://172.20.10.5:8040 toto/main all Packages [728 B]
Fetched 10.5 kB in 0s (175 kB/s)
Reading package lists... Done
# apt-cache policy steam-launcher
steam-launcher:
 Installed: (none)
 Candidate: 1.0.0.54
 Version table:
 1.0.0.54 500
 500 http://172.20.10.5:8040 toto/main amd64 Packages
 500 http://172.20.10.5:8040 toto/main i386 Packages
 500 http://172.20.10.5:8040 toto/main all Packages

The new package is published, we can get back to the old snapshot published and see that the steam package is not there anymore :

# echo "deb http://172.20.10.5:8040/ xenial main" > /etc/apt/sources.list
# apt-get update
Get:1 http://172.20.10.5:8040 xenial InRelease [6,147 B]
Get:2 http://172.20.10.5:8040 xenial/main amd64 Packages [1,551 B]
Fetched 7,698 B in 0s (133 kB/s)
Reading package lists... Done
# apt-cache search steam
#

No result we’re good.

Graphs

Oh there’s more than just published repositories, aptly can make you a graph of the repository tree you make and published or even the snapshots :

# aptly graph -output="/root/graph.png"
Generating graph...
Output saved to /root/graph.png

And you can see the result just here :

Conclusion

Well, Aptly is really great, easy to process, simple to configure and pretty quick to publish new repositories.

Even if the possibilities of manage repositories of heterogenous environment from one place is not fulfilled, we have here a great tool for Debian like environment.

Contact Us