How to make a systemd service recover from crashes without looping forever: Restart policies and Dependencies

The minimal service from the previous document starts your application at boot, but it has a significant weakness: if the application crashes, systemd simply leaves it in a failed state. I don’t have to remind that for a device running unattended in the field, something you might not physically access for weeks or months, a silent crash that’s never recovered from is unacceptable. This document covers the two additions that transform a basic service into a resilient one: restart policies and dependency declarations.

If you want a concrete working example to follow along with, use the script from examples/python-service/hello_service.py in the GitHub repository. It’s a simple loop that logs a heartbeat message every ten seconds. It is straightforward enough to understand at a glance, but long-running enough to behave like a real service.

Restart Policies: For Surviving Crashes

The Restart directive in the [Service] section tells systemd what to do when your application’s process exits. The default is Restart=no. In this case systemd does nothing and your service just stays dead.

There are several possible values, but in practice you’ll almost always choose between two of them based on this simple question: ” Should this service restart if you deliberately stop it with systemctl stop? ”.

Restart=on-failure means: restart the service if the process exits with a non-zero exit code. It restarts if it is killed by a signal, or hits a timeout, but not if it exits cleanly with exit code 0. The practical consequence is that systemctl stop is a clean exist. So it doesn’t trigger a restart, because systemd sends a clean termination signal which your application handles gracefully. For most production services this is the right choice because it means you can stop a service for maintenance without a restart loop annoying you.

Restart=always means: restart no matter what, even if the process exits with code 0. Obviously, this is appropriate for services that should never be down or where even a clean exit is considered abnormal. It is a less common case but useful for things like a watchdog process or a service whose restart you want explicit control over from systemd alone.

For the vast majority of embedded applications, on-failure is the correct choice.

The RestartSec Pairing

Always pair Restart with RestartSec. RestartSec sets the delay in seconds before systemd attempts to restart the process. Without it, systemd restarts immediately. This may sound desirable, but think about why this creates a problem. Your service is crashing for a reason. That reason could be a dependency it needs isn’t yet ready. For example, the network stack is still initialising, or a database is still starting up. So an immediate restart will just crash again, and again, and again. Even a small delay of five seconds gives the rest of the system time to settle into a stable state.

Restart=on-failure
RestartSec=5

How to Prevent Infinite Crash Loops

Now consider a more serious scenario: your service has a real bug that makes it crash immediately on every startup. Perhaps it could be a malformed configuration file or a missing Python dependency. If don’t set a limit, systemd will restart it, watch it crash, restart it, watch it crash, forever. This consumes CPU, floods your logs with crash reports, and masks the real problem.

The StartLimitIntervalSec and StartLimitBurst directives work together to prevent this. StartLimitBurst defines the maximum number of restart attempts allowed, and StartLimitIntervalSec defines the time window those attempts are counted in. To put simply, if the service restarts more than StartLimitBurst times within StartLimitIntervalSec seconds, systemd declares the service as failed and stops trying.

StartLimitIntervalSec=60
StartLimitBurst=5

With these settings, systemd makes up to five restart attempts within any 60-second window. If the sixth crash happens within that window, it marks the service as failed and stops. At that point systemctl status clearly shows that the restart limit was reached. In my personal experience, I have found this to be a very valuble signal. It means something is fundamentally broken and needs human attention, not more restart attempts.

I used to imagine StartLimitBurst as a circuit breaker. In electrical engineering, a circuit breaker trips when it detects a fault and it doesn’t keep trying to send power through a short circuit. systemd’s start limit is the same idea: try a reasonable number of times, then stop and let a human investigate rather than hammering a broken process indefinitely. How beautiful is that!

Dependency Declarations: Starting things in the Right Order

Most real applications don’t run in isolation. An MQTT client is useless if it tries to connect before the network interface exists. A database-backed service will crash if the database process hasn’t finished starting. A camera streaming service first needs the camera hardware to be initialised. There is a way to express these dependencies explicitly in systemd. This way your service starts at the right moment in the boot sequence rather than just racing against everything else.

The two most important dependency directives live in the [Unit] section.

After: Ordering Without Hard Requirements

After=some.target tells systemd not to start your service until some.target has been reached. This is purely about ordering and it doesn’t create a hard requirement. If some.target fails entirely, your service still starts after it in the sequence. It just may find that what it was waiting for isn’t actually available.

The target you’ll use most often is network.target, which represents the point where the network subsystem has been initialised. On Raspberry Pi OS, this typically means the kernel network stack is ready and configured interfaces have been brought up.

After=network.target

A more reliable option for network-dependent services is network-online.target. This is particularly valuble for those that connect to remote hosts. It doesn’t just wait for the network subsystem to initialise but waits until at least one network interface has an active connection and is reachable. This is more conservative and also means a slightly longer boot time. But think about why it prevents some of the common failure modes. Imagine i senario where your service starts, tries to open a TCP connection to a remote server but fails because the network is still negotiating a DHCP address, and crashes before the restart delay kicks in.

After=network-online.target

Wants and Requires: What are Soft and Hard Dependencies

Wants=some.target and Requires=some.target declare what your service needs, not just what it should start after. The difference is what happens when the dependency isn’t available.

Wants is what is called a soft dependency. If some.target isn’t available or fails to start, systemd will still attempt to start your service. Wants is like saying “I’d prefer this to be available, but I can try to run without it.” This is appropriate when your application has internal handling for the dependency not being there. Which means you have thought about these cases and written you application code to handle them. For example, you write an MQTT client that queues messages if the broker is ever unreachable. Now you can use a soft dependency because you know that your application handles it.

Requires is a hard dependency. If some.target fails to start, systemd won’t start your service either. Furthermore, if some.target is stopped while your service is running, systemd will also stop your service. This is appropriate when your service cannot function at all without the dependency. For example, a service that reads from a database where the database is also managed by systemd.

In practice, the combination you’ll use most often for network-dependent services is:

After=network.target
Wants=network.target

This says “start after the network is ready, and express a preference for it to be available but if the network fails, keep the service running, because I handle network loss internally.”

The Updated Service File

Putting the restart policy and dependency declarations together, here’s the service file updated from the minimal version:

[Unit]
Description=My Application
After=network.target
Wants=network.target

[Service]
ExecStart=/usr/bin/python3 /home/pi/my-app/main.py
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=60
StartLimitBurst=5

[Install]
WantedBy=multi-user.target

After editing the file, reload systemd and restart the service to apply the changes:

sudo systemctl daemon-reload
sudo systemctl restart my-app.service

You can test the restart policy right now without rebooting. Find the process ID from systemctl status, then kill it:

sudo kill -9 <PID>

Wait five seconds and run systemctl status again. You should see the service back in active (running) state with a new PID and a note that it was restarted. That automatic recovery is exactly what you want from a service running unattended in the field.

Common Dependency Targets Reference

Beyond network.target, a handful of other targets come up regularly in embedded service files.

local-fs.target ensures all local filesystems are mounted. This is useful if your service reads or writes files on a specific mount point.
time-sync.target ensures the system clock has been synchronised via NTP, which matters for any service that timestamps data or signs tokens.
bluetooth.target ensures the Bluetooth subsystem is available, relevant for Raspberry Pi projects that communicate with BLE devices.

You can also use your services written by yourself. If a service depends on another service you’ve written yourself, you can declare After=my-other-app.service and Requires=my-other-app.service directly by service name. Clever isn’t it?

In this next article explains how to fix the relative-path failures that appear under systemd, run as a non-root user, and how to keep secrets out of the service file.