Starting services only when the network is ready on Debian/systemd

TL;DR: ^

  • Make sure that whatever configures your network supports
  • Override the service unit to have and the same for After=

Overview ^

Sometimes you only want services to start up once there is a network configured. Most network services can handle the situation where there is initially no network, waiting until the network appears, because this is a very common situation.

Other services though may not in themselves be expecting to use the network, and so have never thought about it. Also a great thing about open source software is that it tends to be very composable, so it’s not possible to predict the ways that people will use combinations of software.

The problem ^

systemd will tend to start things as soon as it can. If your service is not configured to wait for the network that means it will most likely be started up before the network exists. If your service then tries to do something that requires a network it will receive an error, which it may not be prepared to handle.

A concrete example: ulogd2 ^

A real life example for me is ulogd2. ulogd2 allows your firewall rules to log things in a variety of ways, in incredible detail.

Most of the ways people configure it involve just logging to the local filesystem, so it doesn’t actually require the network to be configured first.

The default systemd configuration in Debian buster for the ulogd2 service looks like this:

$ sudo systemctl cat ulogd2.service
Description=Netfilter Userspace Logging Daemon
ExecStart=/usr/sbin/ulogd --daemon --uid ulog --pidfile /run/ulog/

As you can see there’s nothing in there that says to wait for a network.

I use a database plugin for ulogd2 that makes it log to a (remote) database. As a consequence as soon as it starts up it tries to establish a database connection, immediately fails as there is no route to any remote host, retries a few times and then bails out.

Most of the time it exhausts its retries before the network is up, so the result is that the service is in a failed state. Simply manually starting the service (or having config management do it) resolves that, but that’s a mess.

Ideally I don’t want systemd to start ulogd2 until there is a network.” mate. Job done. No! ^

If like me you know just enough about systemd to be dangerous, you figure that what you want to do is add something like this to the [Unit] section of the service unit file:


This is only part of the correct solution. If you do only this then you’ll probably find that nothing actually changes.

About ^

The thing about the network-online target is that it doesn’t exist unless you’re using a “modern” method of bringing up your networking, like NetworkManager or systemd-networkd.

If you’re not doing that then systemd works out that the network-online target can never be reached and ignores it as a Want.

I’m using ifupdown on servers as it still does everything I need it to. To make ifupdown support the network-online target on Debian, you should enable the ifupdown-wait-online service:

$ sudo systemctl enable ifupdown-wait-online.service

This will inject the network-online “target reached” state when every interface that is marked as “auto” in /etc/network/interfaces is up.

Editing a service file ^

The temptation now may be to edit the ulogd2 service file that’s under /lib/systemd/system/ to contain the Want/After bits.

That will work but it isn’t the correct way because if there is a package update then your changes will be overwritten.

A better way is to place a new service file into /etc/systemd/system/. That will entirely override the distributed copy. The obvious downside there is that if there’s an improvement to the packaged service file then you’ll never use it, as you’ve entirely overridden it with your own file.

Overrides to the rescue ^

The best way is to use an override file, and the easiest way to do that is with systemctl edit:

$ sudo systemctl edit ulogd2
[your favourite editor starts]

Check your changes took effect:

$ sudo systemctl cat ulogd2.service
# /lib/systemd/system/ulogd2.service
Description=Netfilter Userspace Logging Daemon
ExecStart=/usr/sbin/ulogd --daemon --uid ulog --pidfile /run/ulog/
# /etc/systemd/system/ulogd2.service.d/override.conf

Note that this shows you where the files actually are. That makes it easy to distribute this through config management.

systemd on Debian, reading the persistent system logs as a user

All the documentation and guides I found say that to enable a persistent journal on Debian you just need to create /var/log/journal. It is true that once you create that directory you will get a persistent journal.

All the documentation and guides I found say that as long as you are in group adm (or sometimes they say group systemd-journal) it is possible to see all system logs by just typing journalctl, without having to run it as root. Having simply done mkdir /var/log/journal I can tell you that is not the case. All you will see is logs relating to your user.

The missing piece of info is contained in /usr/share/doc/systemd/README.Debian:

Enabling persistent logging in journald

To enable persistent logging, create /var/log/journal and set up proper permissions:

install -d -g systemd-journal /var/log/journal
setfacl -R -nm g:adm:rx,d:g:adm:rx /var/log/journal

-- Tollef Fog Heen <>, Wed, 12 Oct 2011 08:43:50 +0200

Without the above you will not have permission to read the /var/log/journal//system.journal file, and the ACL is necessary for journal files created in the future to also be readable.

Paranoid, Init

Having marvelled at the er… unique nature of MikeeUSA’s Systemd Blues: Took our thing (Wooo) blues homage to the perils of using systemd, I decided what the world actually needs is something from the metal genre.

So, here’s the lyrics to Paranoid, Init.

Default soon on Debian
This doesn’t help me with my mind
People think I’m insane
Because I am trolling all the time

All day long I fight Red Hat
And uphold UNIX philosophy
Think I’ll lose my mind
If I can’t use sysvinit on jessie

Can you help me
Terrorise pid 1?
Oh yeah!

Tried to show the committee
That things were wrong with this design
They can’t see Poettering’s plan in this
They must be blind

Some sick joke I could just cry
GNOME needs logind API
QR codes gave me a feel
Then binary logs just broke the deal

And so as you hear these words
Telling you now of my state
Can’t log off and enjoy life
I’ve another sock puppet to create