SETTING UP STALWART WITH PODMAN AND QUADLET

SECTION 1 – BACKGROUND

This document comes from my many attempts to make this work with podman-compose setup. I got it working and hope to document those details at a later date. I had to give up on that because when I exited the ssh session it would kill the server. I found this reddit post which made me conclude that I had to learn the ‘ubuntu way’ which would be podman + quadlet combo. Thankfully, I found this excellent post which helped lay the foundation of this post.

There is also the official podman page for quadlets which to me is interesting since it doesn’t even have ‘quadlets’ in the URL… this caused me to search up the actual meaning of ‘quadlet’. For example, was it a pre-existing term that moved from other parts of ubuntu / linux into the podman world, or was it a new term designated for podman-specific work? This seems like a great question to throw at AI! Here is what it threw back:

“The term quadlet in the context of Linux and Podman is specific to the Podman project. It was created as part of Podman’s tooling to simplify container management using systemd unit files. The name “quadlet” is unique to Podman and doesn’t have a general pre-existing definition in Linux.”

Ok, great. Then the podman project should have had the term in their URL for better SEO, but I digress..

This document is not a Stalwart configuration tutorial. That should follow soon. This is just to get you ‘up and running’ to the login screen of Stalwart.

Well, here we go.

SECTION 2 – HIGH LEVEL VIEW OF STEPS

I like to take helicopter trips above the construction site before starting to build. I find it’s kind of nice to have a rough idea what needs to be done. Probably the best summary of where we’re heading is this quote from the post above:

“Instead of defining multi-containers stacks in a single file, like we do when using docker-compose, with Quadlet, we define containers, volumes and networks using dedicated Systemd units.”

Here is our high-level journey view:

1. Set up DNS so that it has propagation time if required
2. Get an Ubuntu server set up
3. Determine whether your container (stalwart) will run as root or as a different user
4. a) If root, you will save your “units” in /etc/containers/systemd
   b) If not root, you will save your “units” in ~/.config/containers/systemd
5. Build a systemd network “unit” (if required) so the container can communicate where it needs to
6. Build a systemd container “unit” which has all the Stalwart server stuff in it
7. Build a systemd volume “unit” which will allow the data to persist when the container goes down

SECTION 3 – PRE-INSTALLATION TIPS AND STEPS

STEP 1 – DO THE DNS

By the time you spend all this time doing this and reach the end, you’ll be pumped to be ready to use your email server. The worst is when you forget to have a domain ready, or nameservers are not pointing to cloudflare, or wherever, and then you have to wait.

Further, I just learned that letsencrypt cannot secure IP addresses (the number) but requires a legitimate domain to work. So the fact of the matter is that, even if you want to test out an email server, you really must have:

a) A dedicated domain
b) A non-residential server

STEP 2 – DO THE VPS

In the beginning of this journey I had tried to (mainly for educational purposes) run this on a self-hosted machine in a residential ISP environment. It was just too hard and not worth the battle in the end.

I got a cheap VPS for basically $1/month at the end you can take it from me that there is no point to really starting without being ready to get these two items or have access to them. You should be able to run this on another server if you want but I wanted a focused ‘playground’ – and I’m glad I did it that way.

Start sooner than later so that your DNS records are pointing to it sooner than later.

STEP 3 – DO THE PTR

Assuming you will use this as a dedicated email server, you’ll need a valid PTR record. As important as this is there isn’t a lot of simple info on this so allow me to save you some time.

If you go to your terminal and punch in dig -x google.com you’ll see something that looks like this:

;com.google.in-addr.arpa. IN PTR

That’s your PTR record and you need to have one to have good performance or, in some cases, any performance. Each VPS is different. Some you can adjust your own, some you have to open a ticket, etc etc. Figure that out now and get it started.

For full transparency, I don’t know that much about PTR recorsds and it seems that somehow all the domains I added into Stalwart later, when running the command above, do well and have their own PTR record. Not sure if that is done by Stalwart or not… but gmail seens to demand a valid PTR record if you plan to send to a gmail address without landing in spam.

SECTION 4 – SETUP YOUR SERVER

STEP 4 – SET UP THE UBUNTU SERVER

I can’t perfectly recommend what size server is best. My cheap 1GB VPS seems to be working just fine but if you plan to be serious, I’d reach out to Stalwart community to be sure.

OS – I did Ubuntu. My VPS server came with Ubuntu 22.04, but I went through process before beginning of upgrading to 24.04 (details about why to follow). If you can get it out of box with 24.04, that’s ideal.

Following this blog. For me, since mine was a fresh server, I always chose ‘package mantainers’ version of everything when prompted. Your situation may be different.

This actually matters because the podman-compose (coming in another post one day…) package, for example, is harder to install in ubuntu 22…

STEP 5 – CREATE NON-ROOT USER IN UBUNTU

Always smart to run things as non root in a server. So since this will be a Stalwart email server, go ahead and make a stalwart server username:

1. adduser stalwart
2. Enter credentials and a cool room number…I heard room 237 is quite popular…
3. Add ’em to the sudo group: usermod -aG sudo stalwart
4. switch to this user: su stalwart
5. test the superpowers with something like sudo apt update, enter pw and see if stuff works
6. Log out completely as stalwart: exit
7. Log out of server completely exit
8. Log back in with SSH as new stalwart user – doing so tests ends session completely which you need for podman stuff (coming below).

STEP 6 – DETERMINING WHETHER YOU WILL RUN AS ROOT OR NOT

For this tutorial I’m going to try to do this as non-root since I was told by people much smarter than myself: “It’s never good to run a container as root”.

And so that is that for now and I will resort to root if I fail, haha

SECTION 5 – BECOMING A REAL PODMAN

Podman is pretty awesome to say the least. It’s like Docker but more safe and ‘poddable’. Look into it more if you want but I decided to spend my time learning podman rather than Docker for these reasons.

STEP 7 – SETTING UP PRIVILEGED PORTS FOR NON-SUDO CONTAINER

General rule out there is that you should not let a container run as root. That’s why we are using podman. Docker runs as root. This has some risk. We prudent folks use podman to give the signal of wisdom…but anyways, you have to do more work to be prudent including making sure your non-root container can reach the ports you need opened on an email server…

Make it so the container has full access to proper email ports above without running container as root.

Apparently there is a way to do this by following this link but, what I did was this move, a favourite amongst Docker users too:

1. Open this for editing: sudo nano /etc/sysctl.d/99-sysctl.conf (may have different final file name on your version?)
2. Paste this in at the bottom of file:net.ipv4.ip_unprivileged_port_start=25
3. Control x, yes, enter, bye-bye to save changes and exit editor…
4. restart to apply changes: sudo sysctl -p (you should see your change echoed back in terminal)

STEP 8 – MAKING PODMAN REAL

If you are on ubuntu 24, you should be able to use this lazy adventurous Pod-man’s command:
sudo apt install podman && sudo apt install podman-compose
Add a -y at the end to make it even more adventurous and lazy, if you want…

Otherwise, the step-by-step version for the conservative Pod-man:

• install podman sudo apt install podman
• install podman-compose sudo apt install podman-compose (so you can run your startup file)

If your OS is not upgraded to 24, the above podman-compose install may not work with packages. You likely have to do the following steps:

On ubuntu 22.04 and can’t find package for podman-compose?

1. apt install python3-pip
2. apt install python3.10-venv
3. python3 -m venv stalwart-env
4. source stalwart-env/bin/activate
5. pip3 install podman-compose

(bit of a pain – 24.04 is way easier so I’d spend time on doing that first if you can)

NOTE THE STALWART DOCKER IMAGE: Here is the official Stalwart docker image which you will need: docker.io/stalwartlabs/mail-server

BONUS – ADDING DOCKER AS A REGISTRY TO MAKE OTHER DOCKER IMAGES SEARCHABLE IN THE FUTURE

While you’re at it, you might choose to add Docker as a registry (like a repository). You’d think this would be done by default, but no go. Quite annoying thing, but here is a link to backstory

1. sudo nano /etc/containers/registries.conf
2. add unqualified-search-registries = ["docker.io"]
3. sudo systemctl restart podman

STEP 9 – CREATING MY QUADLET SAVE-TO LOCATION

As per above, I will be creating “units” and saving them all to this location:

~/.config/containers/systemd

What seems to be lacking from all documentation on this subject out there is this:

You need to manually create the directory structure ~/.config/containers/systemd/ if it doesn’t already exist. Quadlet does not automatically create these directories for you.

So, let’s run this command to make sure the directories are there to save these ‘units’:
mkdir -p ~/.config/containers/systemd/

Keep in mind, we’ll need to run systemctl --user daemon-reload so that the system knows the unit files are in there and then also enable them, but we’ll cover that later.

STEP 10 – BUILDING A DOT NETWORK UNIT FILE (IF REQUIRED)

If you do not explicitly define a network in a Podman .container unit file (or in any container definition), Podman will use the default network configuration. This means that the container will typically run in the default podman network unless otherwise specified. So, this may not be needed for your setup but it’s here as a consideration.

1. create the Stalwart .network blank placeholder file: nano ~/.config/containers/systemd/stalwart.network
2. paste in some template info:

[Unit]
Description=Stalwart Network

[Network]
#FILL IN YOUR APPROPRIATE NETWORK INFORMATION HERE AND THEN UN-COMMENT
#Subnet=123.123.1.123/24
#Gateway=123.123.1.123

STEP 11 – BUILDING A DOT VOLUME UNIT FILE FOR THE CONTAINER

In this step we’ll build the file that will hold the data for our container

1. create the Stalwart .volume file: nano ~/.config/containers/systemd/stalwart.volume
2. Copy this and paste it into the opened file above

[Unit]
Description=Stalwart Volume

[Volume]
#Left blank (because we defined these details in the .container file)

3. Control x, Y, Enter to save and close out of editor
4. If you are logged in with SSH session, you’ll have to do this convoluted way of resetting the daemon:

export XDG_RUNTIME_DIR=/run/user/$(id -u)
export DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/bus
systemctl --user daemon-reload

The good news about this command agove is that it seems to persist after running it, perhaps endures the whole ssh session?

If you are logged in on the physical machine (not-ssh) this should work as is: systemctl --user daemon-reload

TIP 1: Remember this command: podman volume inspect stalwart.volume This will show you where the storage actually is on your local machine if you need to do stuff to it (like I did) later

STEP 12 – SETTING UP THE DOT CONTAINER UNIT FILE

Here is my simple attempt to write a container unit file for Stalwart… is this called a Quadlet? No one will ever know….

Anyways, a few notes before copying mine:

• I left the Network section in there as a reminder but I’m not using as I don’t need it. Network=podman seems to ‘just work’ in my case

To copy mine and use it,

1. nano ~/.config/containers/systemd/stalwart.container
2. copy this and paste it into the above opened file

[Unit]
Description=A Stalwart Container
After=network.target

[Container]
#Grabbing the docker image
Image=docker.io/stalwartlabs/mail-server:latest

# Setting up a persistent volume on host where 'stalwart.volume' is volume unit file
Volume=stalwart.volume:/opt/stalwart-mail

# Setting up a custom network - if custom network required you can point 
Network=podman

#Publishing ports
PublishPort=8080:8080
PublishPort=443:443
PublishPort=25:25
PublishPort=465:465
PublishPort=587:587
PublishPort=143:143
PublishPort=993:993
PublishPort=4190:4190
PublishPort=110:110
PublishPort=995:995

[Service]
#Extend timeout to make sure enough time to pull stalwart image, if required
TimeoutStartSec=900

[Install]
#Make sure it starts on boot by default
WantedBy=multi-user.target

3. Control x, Y, enter to save and close out of the editor
4. Reload the daemon to make sure system knows about changes: systemctl --user daemon-reload

STEP 13 – RUNNING IT – THE MOMENT OF TRUTH

1. Run this: systemctl --user start stalwart.service
2. Check it! systemctl --user list-unit-files --type=service

If it’s running, congratulations, HOWEVER… I strongly advise you continuing further in this long post and skim through my troubleshooting journey section below! Mine worked, then it didn’t and I think you’ll value following along so you are aware.

SECTION 7 – TROUBLESHOOTING JOURNEY

Along the way, I had a lot of pain trying to get this working so I thought I would include a bit of that journey, mainly so the keywords will be SEO optimized for those who might be going through the same thing.

At the end of the troubleshooting section, I also give a bunch of great resource links that I think you’ll like.

CHALLENGE WITH STALWART BANNING MY CONTAINER IP – THE SNEAKY PROBLEM WITH THE SOLUTION

I was experiencing the most crazy thing for a long time as follows:

• Stalwart container in podman successfully launched
• Stalwart Webmin would begin to work perfectly
• I was able to add Domains and Email accounts, no problem.
• I was able to send and receive emails no problem.
• I would then log out of my Stalwart server ssh session
• After a period of time (it seemed like between 1.5 and 3 hours) the following problems happened:
  a) webmin access would die
  b) emails would no longer send or receive
• However, using the famous port checking tool, all ports were open
• using the podman ps command showed the starwart server was ‘mostly working’
• Looking deep into the containers logs by running this command: enter container bash command here
• Errors of ‘banned due to scan’ appeared with an ip of, in my case 10.88.0.7, appeared.
• Thankfully I recognized this was probably the container IP (they are often (always?) in the 10 range)

Was my container being banned by Stalwart? It seems so…

• Ran this to confirm container IP: podman inspect <container-name> | grep -A 5 NetworkSettings
• got this:

"NetworkSettings": {
               "EndpointID": "",
               "Gateway": "10.88.0.1",
               "IPAddress": "10.88.0.7",
               "IPPrefixLen": 16,
               "IPv6Gateway": "",

• Rebooted Stalwart with podman down <stalwart-container>
• Brought it back up with systemctl --user start stalwart.service
• Knowing I have only a limited time before it goes down again, and knowing that I probably had a fresh IP address assigned with the container rebuild (seems to do that), I checked logs via the webmin to see, no surprise at this point, the ‘banned due to scan’ errors and a bunch of failing events.
• Discovered the Security section of Stalwart under “Settings > Security” and, in particular, the ‘blocked IPs’ section (/settings/blocked-ip). Sure enough, there were a bunch of my previous container IPs including the 10.88.0.7 IP above.
• I removed / deleted these entries from the blocked-ip
• Went to Settings>Security and ‘Allowed IPs'(/settings/allowed-ip)and added the Gateway and IPAddress IPs as per above to the ‘Allowed IPs’ list.
• I reloaded the config a few times to make sure it saved

Almost immediately after saving these changes, the webmin returned from a broken state and emails began sending.

Conclusion: Stalwart was banning itself, basically, as it banned the container IP address. Once white-listed, all was well.

SECTION 8 – THE BONUS SECTION – ALMOST AS GOOD AS A BOGO

Here are a bunch of related but ‘outside of scope’ sections that I think have some value for whoever has persevered this far.

BONUS 1 – CHALLENGE LEARNING PODMAN QUADLETS IN GENERAL

To achieve my goal of using Podman on ubuntu and being able to spin up a Stalwart email server using the ‘quadlet way’, I had to learn, from scratch, quadlet, which is, effectively, learning systemd – the thing that controls ubuntu’s processes.

This introductory tutorial to systemd helped quite a bit.

This focused tutorial on podman quadlet was very good.

BONUS 2 – CHALLENGE WITH DNS RECORDS LIVING ON AN OLD CPANEL MACHINE

See this post Here for details about how one old cpanel record messed up my whole week!

BONUS 3 – HOW TO CHECK YOUR PODMAN CONTAINER’S IP ADDRESS AND EDIT YOUR STALWART CONFIG FILE

1. get podman network range: podman network inspect podman
2. grab the subnet range. In my case it looked like this: "subnet": "10.88.0.0/16"
3. bash into your stalwart container: podman exec -it systemd-stalwart bash
4. go to your .toml file: cd /opt/stalwart-mail/etc
5. open the file for editing and apt install nano if required to get editor: nano config.toml

TIP: Troubleshooting your containers network info is quite easy with this command: podman inspect <container-name> | grep -A 5 NetworkSettings

BONUS 4 – YOU HAVE TO LET IT LINGER

In the beginning of my troubleshooting journey I was having an issue where I was ‘pretty sure’ that my logging out of my SSH session was the cause of all my pain. It seemed that when I logged back in with SSH that everything started working again. One of the suggested causes of this behaviour was something called ‘Linger’ related to the session.

Apparently – sometimes – you may have to allow ‘lingering’ for a user so that the session ‘stays alive’ after you break the SSH connection, otherwise all the services you set up die with you on your exit.

It’s literally this easy:

1. Enable it: loginctl enable-linger <user>
2. Check to make sure there is a ‘yes’ by “linger’ after running this: loginctl user-status stalwart
3. Test by logging out of ssh and seeing if your Stalwart is still alive

You may or may not need this trick. Don’t bother unless you need, of course.

BONUS 5 – FORWARDING IPS

This section I’m leaving mainly as notes for myself because I can’t perfectly remember why I did it. I ‘think’ I was trying to enable IP address forwarding from the Podman container to the Host which I – at the time – believed was not working. Disregard this section unless it’s something fun and quick to try…

1. sudo nano /etc/sysctl.d/99-sysctl.conf
2. around line 28, uncomment #net.ipv4.ip_forward=1
3. restart systemd stuff: sudo sysctl -p

SECTION 9 – AMAZING EMAIL ADMIN RESOURCES

I wanted to throw a general list of resources in this post because some of these posts would have made my life so much easier if I had them before I began. Hope they help others too:

• SPF-FOCUSED POST
• DEEP DIVE ON MX RECORDS
• NICE TELNET HOW TO GUIDE
• NICE OVERVIEW OF SETTING UP AN EMAIL SERVER
• Greatest SSH Video I’ve probably ever seen here
• Great tutorial for using Thunderbird as a kind of Telnet tool

Tagged config, container, podman, quadlet, stalwart, tutorial, Ubuntu