Implementing Exponential Backoff: Preventing Thundering Herd Problems

When a broker goes down and a thousand edge devices all try to reconnect every second, the moment the broker comes back online it gets hit with a thousand simultaneous connection requests and dies again. This is the thundering herd problem, and it is entirely self-inflicted. The fix is exponential backoff with jitter — a reconnection strategy every networked embedded application should implement from day one.

See project in Github: Resilient Edge MQTT Client

The Wrong Way

# Every device retries every second. 1000 devices = 1000 req/s on reconnect.
while True:
    try:
        client.connect(broker_host, broker_port)
        break
    except Exception:
        time.sleep(1)  # Fixed delay — all devices retry in lockstep

Fixed-delay retry is the problem. Every device that disconnected at the same time will retry at exactly the same time, forever, until one of them happens to succeed. At scale, this makes outages self-perpetuating.

The Right Way

import time
import random

def connect_with_backoff(client, broker_host, broker_port,
                         min_delay=1, max_delay=300):
    """
    Reconnect with exponential backoff and jitter.

    Delay sequence (approximate): 1s, 2s, 4s, 8s, 16s, 32s, 60s, 60s...
    Jitter adds ±10% randomness so devices that disconnected together
    do not retry together.
    """
    delay = min_delay

    while True:
        try:
            # Jitter is proportional to the current delay, not fixed.
            # At delay=1 it adds up to 0.1s. At delay=60 it adds up to 6s.
            # This keeps devices spread out even after long outages.
            jitter = random.uniform(0, delay * 0.1)
            time.sleep(delay + jitter)

            client.connect(broker_host, broker_port)
            return  # Success — caller resets delay to min_delay

        except Exception as e:
            print(f"Connection failed: {e}. Retrying in {delay:.1f}s")
            delay = min(delay * 2, max_delay)

The three decisions baked into this function are worth understanding individually.

Doubling the delay (the exponential part) means the load on the broker drops by half with each retry cycle. After a few rounds, devices are spread so far apart in time that the broker recovers comfortably before the next wave arrives.

Capping at max_delay prevents devices from backing off indefinitely. A cap of 60–300 seconds is usually right for embedded systems — long enough to give the broker real recovery time, short enough that you do not lose hours of data during a brief outage.

Proportional jitter is the part most implementations get wrong. Adding a small fixed jitter like random.uniform(0, 0.5) works when delay is 1 second but is meaningless when delay is 60 seconds — all devices are still retrying within the same half-second window. Making jitter proportional to the current delay (10% here) keeps devices spread out at every stage of backoff.

Reset the Delay on Success

This is the step that gets forgotten. After a successful connection, the delay must reset to min_delay. If it stays at whatever value it reached during the outage, the next disconnection — even a brief one — will start with a 60-second wait instead of a 1-second wait.

class ResilientMQTTClient:
    def __init__(self, broker_host, broker_port):
        self.broker_host = broker_host
        self.broker_port = broker_port
        self.min_delay = 1
        self.max_delay = 300
        self.current_delay = self.min_delay  # Reset this on every successful connect

    def on_connect(self, client, userdata, flags, rc):
        if rc == 0:
            self.current_delay = self.min_delay  # Always reset here
            print("Connected — backoff delay reset to minimum")

    def reconnect(self):
        jitter = random.uniform(0, self.current_delay * 0.1)
        time.sleep(self.current_delay + jitter)

        try:
            self.client.reconnect()
        except Exception:
            self.current_delay = min(self.current_delay * 2, self.max_delay)

With Paho MQTT, the right place to reset is inside the on_connect callback, which fires on the background network thread whenever a connection is established. Resetting in the reconnection loop itself is too early — the connection might still fail after the reset.

Add a Restart Rate Limit at the System Level

Backoff handles network-level reconnection. But if your application itself is crashing and restarting — not just reconnecting — you need a second line of defence. systemd’s StartLimitBurst and StartLimitIntervalSec directives cap how many times a service can restart within a given window before systemd stops trying and marks it failed.

[Service]
Restart=on-failure
RestartSec=5                  # Base delay before first restart
StartLimitIntervalSec=120     # Window for counting restarts
StartLimitBurst=5             # Give up after 5 restarts in 120 seconds

This prevents a crashing service from hammering a broker with rapid reconnections that bypass your application-level backoff entirely.

Quick Reference

Enable exponential backoff on every service that connects to an external broker, API, or database — not just MQTT. Start with min_delay=1, max_delay=60 for most edge applications. Use proportional jitter (10% of current delay), not fixed jitter. Always reset the delay counter inside the successful-connection callback. Pair application-level backoff with systemd restart rate limiting for complete coverage.