Podman fails to start at boot because it starts before the network is up

a commented

2024-04-20 02:03:35 +00:00

Owner

I noticed that some user services aren't starting automatically at boot because although they have the lines

Wants=network-online.target
After=network-online.target

systemd is starting them before that target is reached, which causes them to throw a network error since the network isn't up yet. This is because the systemd user and system managers are completely separate and it's not possible for a user unit to depend on system units. This is also related to Podman 5.0's switch from slirp4netns to pasta.

There are several forum posts online about making a user service wait for network-online.target such as https://unix.stackexchange.com/questions/216919/how-can-i-make-my-user-services-wait-till-the-network-is-online and https://bbs.archlinux.org/viewtopic.php?id=278852. One of those posts recommends the magic command systemctl --global link /usr/lib/systemd/system/network-online.target... and it doesn't work! When making a symlink to a system unit, systemd won't sync the status of the system unit to the user unit, but will actually run another copy of this unit in the systemd user instance.

So this leaves me with a few solutions.

Switch back to slirp4netns. Upstream podman says they're permanently switching to pasta so I think we should do the same, and plus pasta is faster.
Delay the start of user sessions until the network is up. This adds some startup delay, but if the network isn't even accessible yet, that delay doesn't really matter.
Write our own network-online.target for user sessions using nm-online. This requires creating quite a few files since we need a target file, a service that runs nm-online, and enable the service.
Write a wrapper script for pasta that runs nm-online first.

I ended up choosing solution 2 because it's the least gross, but if anyone has other ideas, I'd love to hear it!

I noticed that some user services aren't starting automatically at boot because although they have the lines ``` Wants=network-online.target After=network-online.target ``` systemd is starting them before that target is reached, which causes them to throw a network error since the network isn't up yet. This is because the systemd user and system managers are completely separate and [it's not possible for a user unit to depend on system units](https://github.com/systemd/systemd/issues/3312). This is also related to [Podman 5.0's switch from slirp4netns to pasta](https://github.com/containers/podman/issues/22197). There are several forum posts online about making a user service wait for `network-online.target` such as https://unix.stackexchange.com/questions/216919/how-can-i-make-my-user-services-wait-till-the-network-is-online and https://bbs.archlinux.org/viewtopic.php?id=278852. One of those posts recommends the magic command `systemctl --global link /usr/lib/systemd/system/network-online.target`... and it doesn't work! When making a symlink to a system unit, systemd won't sync the status of the system unit to the user unit, but will actually run another copy of this unit in the systemd user instance. So this leaves me with a few solutions. 1. Switch back to slirp4netns. [Upstream podman says they're permanently switching to pasta](https://github.com/containers/podman/issues/22363) so I think we should do the same, and plus pasta is faster. 2. [Delay the start of user sessions until the network is up](https://github.com/systemd/systemd/issues/3312#issuecomment-2039973852). This adds some startup delay, but if the network isn't even accessible yet, that delay doesn't really matter. 3. Write our own `network-online.target` for user sessions using `nm-online`. This requires creating quite a few files since we need a target file, a service that runs `nm-online`, and enable the service. 4. Write a wrapper script for pasta that runs `nm-online` first. I ended up choosing solution 2 because it's the least gross, but if anyone has other ideas, I'd love to hear it!

a added the

bug

label 2024-04-20 02:03:35 +00:00

a closed this issue

2024-04-20 02:05:58 +00:00

a commented

2024-04-20 05:58:27 +00:00

Author

Owner

The server just mysteriously rebooted (maybe a very short power outage), and this caused pasta errors at boot time again despite the fix above, so I'm guessing the router also rebooted and the router takes a while to start up. I don't think there's much we can do in this case.

Podman fails to start at boot because it starts before the network is up #218