podman-compose and systemd

I’m using more and more podman and especially podman-compose. podman-compose is not part of RHEL, but it is available in EPEL and it is in Fedora. Of course I run it as a non-root user. It really works great, but creating systemd unit files for podman-compose is ugly. I had it running for about a year, but I wanted to have a look for something better. This blog post talks about Fedora (tested with 39), RHEL8 and RHEL9. All have some smaller problems, but sometimes different ones.

I wanted to try Quadlet-podman for over a year. This week I had a closer look and found that it is more complicate than I thought. I really like the simple one-file solution of a compose file. I found podlet to migrate compose files to quadlet. (Use podlet musl, if you have problem with the glibc of the gnu version).

But at the end I really like to continue using the compose file that are provided by most of the tool that I use and I had only very small problems with podman-compose, all of them easy fixable. At the end I decided to use podman-compose systemd. There is not a lot of documentation, but I really liked it.

I had quite a lot of problems, but I will show you here how to fix them and how my setup is working now.

Setup as root

First things first. I run it always as non-root of course. If you do too, please do a “restorecon -R” to the homedir of the user that runs the container. audit2allow and the logs will not show the problem (you have to disable noaudit to see it – I fear), but it will interferes with the startup of your container.

You want to make sure the container user can run even when he is not loged in. So you have to enable linger with:

loginctl enable-linger USERNAME.

I enabled cgroupv2 on my RHEL8 and there is a bug that you have to fix. The Problem can be fixed by different solution, but I choose to change the file /etc/containers/containers.conf. Of course this is not needed on RHEL9 or Fedora.

#/etc/containers/containers.conf
[containers]
pids_limit=0

To use podman-compose systemd you need a systemd template and I choose to set it up in /etc, because I have multiple user running multiple applications. (If the filename of the output of systemctl status on Fedora/RHEL9 looks strange to you. There is a link: /etc/xdg/systemd/user -> ../../systemd/user).

You can run the command podman-compose systemd -a create-unit as root. Or you can run this command as a normal user and paste the output to /etc/systemd/user/podman-compose@.service. But all platforms (podman-compose runs with version 1.0.6 on all) the template has an error that prohibits the successful startup of the containers. You have to add “–in-pod pod_%i” to the up command – thanks to RalfSchwiete. I also added the ExecStopPost line. Here my complete template:

# /etc/systemd/user/podman-compose@.service
[Unit]
Description=%i rootless pod (podman-compose)

[Service]
Type=simple
EnvironmentFile=%h/.config/containers/compose/projects/%i.env
ExecStartPre=-/usr/bin/podman-compose --in-pod pod_%i up --no-start
ExecStartPre=/usr/bin/podman pod start pod_%i
ExecStart=/usr/bin/podman-compose wait
ExecStop=/usr/bin/podman pod stop pod_%i
ExecStopPost=/usr/bin/podman pod rm pod_%i

[Install]
WantedBy=default.target

As User

With the preparation as root finished, the setup as non-root is quit simple – Almost 😉 . First stop the container with “podman-compose down”. Then go to the directory with the podman-compose.yml (or docker-compose.yml if you use the old name) file and run “podman-compose systemd”. Be careful as this commands starts the containers again. I always stop the containers again with podman-compose down and start it up again with “systemctl –user enable –now ‘podman-compose@COMPOSE ‘ “. Otherwise you are not sure if the systemctl command is working.

But not on Fedora and RHEL9. Here I always got the error message “Failed to connect to bus: No medium found“. The solution was not to use “su – USERNAME” but instead

machinectl shell USERNAME@ 

With su the DBUS_SESSION_BUS_ADDRESS is missing on Fedora and RHEL9. This is a know issue, but RedHat states that “Using su or su – is not a currently supported mechanism of rootless podman.” I’m not sure it machinectl is supported or not, but I can tell you it is working. If you never heard of machinectl before or didn’t know that machinectl has a shell options – you are not alone. 🙂 The official way it to ssh as USERNAME into the maschine. (I prefer my way better :-))

Running but where are the log

If it is working you get this output at the end of the podman-compose systemd command like this:

you can use systemd commands like enable, start, stop, status, cat
all withoutsudo like this:

            systemctl --user enable --now 'podman-compose@COMPOSE'
            systemctl --user status 'podman-compose@COMPOSE'
            journalctl --user -xeu 'podman-compose@COMPOSE'

and for that to work outside a session
you might need to run the following command once

            sudo loginctl enable-linger 'USERNAME'

you can use podman commands like:

            podman pod ps
            podman pod stats 'pod_COMPOSE'
            podman pod logs --tail=10 -f 'pod_COMPOSE'

systemctl –user status ‘podman-compose@COMPOSE’ did work fine on RHEL8 and shows the output of the command. But on Fedora and RHEL9 it did not show anything. On all Version the command journalctl –user -xeu ‘podman-compose@COMPOSE’ never shows any output.

To fix this your non-root user has to become member of the systemd-journald group. But even then you have to use the right command on all Platforms. Not the one from the output above, but this instead:

journalctl  -xe --user-unit 'podman-compose@COMPOSE'

As you can see the podman-compose is quite nice, but there are a lot of stumbling block. After you know how to avoid them, it works quite well.