13 October 2026

📌 Monday night in Gourock, Invisible Circles by After Forever in the headphones. Yesterday's note ended with ten services and three machines that tell me when somebody logs in. It also ended with a question I had left for today: what tells me when the thing that tells me has stopped?

failed, every time

The answer started with a line I was not looking for. While checking one of the new watchers on the lab I read its journal instead of its status, and the journal said this, once for every time the service had been restarted since the day it was born:

Failed with result 'exit-code'

The status said active (running). Both were true. The watcher worked, sent its alerts, and then failed - but only when asked to stop.

The script follows several logs at once, each in a background process, and waits. If one of those processes dies on its own, the script is supposed to give up loudly so that systemd restarts everything:

echo "login-ntfy-watch: a watcher exited, stopping" >&2
exit 1

That part is right. The problem was one line above it:

trap 'trap - INT TERM EXIT; kill_tree $$' INT TERM EXIT

When systemd stops a service it sends TERM. The trap caught it, killed the children, and returned. The script then carried on from where it was, found its watchers dead (it had just killed them), concluded that a watcher had exited, and left with status 1. A clean stop, reported as a crash.

Nothing was broken, which is exactly why it matters. A journal that says "failed" at every restart is a journal where the day it fails for real looks like any other day.

The fix is two lines where there was one:

trap 'trap - INT TERM EXIT; kill_tree $$; exit 0' INT TERM
trap 'trap - INT TERM EXIT; kill_tree $$' EXIT

Being asked to stop is now a success. Stopping for any other reason still is not.

who watches the watchers

Each machine already runs a small script from cron, bottega-watch.sh, with one function per check: is the site up, are the containers running, is the disk filling. Each check alerts once when something breaks and once when it recovers, and stays quiet in between. The watchers got a function of their own:

check_avvisi() {
  local s trovati fermi="" attivi=""
  trovati=$(systemctl list-unit-files '*-ntfy-watch.service' --state=enabled --no-legend 2>/dev/null | awk '{print $1}')

  if [[ -z "$trovati" ]]; then
    problema avvisi "Servizi di avviso spariti su $MACCHINA" "Nessun servizio *-ntfy-watch risulta abilitato: gli avvisi sui login non partono."
    return
  fi

  for s in $trovati; do
    if systemctl is-active --quiet "$s"; then
      attivi+="${s%.service} "
    else
      fermi+="${s%.service} "
    fi
  done

  if [[ -n "$fermi" ]]; then
    problema avvisi "Servizi di avviso fermi su $MACCHINA" "Non attivi: ${fermi% }. Finche' restano fermi, i login che sorvegliano non vengono segnalati."
  else
    risolto avvisi "Servizi di avviso di nuovo attivi" "Su $MACCHINA sono attivi: ${attivi% }."
  fi
}

The texts are in Italian because the scripts on these machines are. The part I care about is the first line of code: the function does not carry a list of services. It asks systemd which units matching *-ntfy-watch.service are enabled, and checks those. On the lab it finds three, on the web two, on the mail server three, and the function is the same file on all of them - compared by checksum, like the SSH watcher the night before.

A list would have been shorter to write. It would also have been wrong the first time I added a watcher and forgot the list, and that is the kind of mistake that makes no noise.

The first branch is there for the opposite case. If the function finds nothing at all, that is not "nothing to check", it is the worst answer it can get, and it says so.

It runs every ten minutes. To test it I stopped one watcher on each machine, waited for the phone, and started it again.

copies

Last piece: getting all of this into git. The obvious way is to move the scripts into the repository and leave links where they used to be. I did not do it. These files are called by cron, by systemd units, by each other, and a path that changes under a running service is a way to find out at three in the morning which of them I forgot.

So the live files stay where they are and the repository gets copies, made by hand. It is the less elegant choice and I know what it costs, because the repositories already had a few copies made that way, weeks ago, and when I compared them with the live files they no longer matched. One was a file that had been committed empty.

A copy made by hand is only as good as the habit of making it. For now the habit is a commit at the end of every evening like this one.

and then...

The rule I have tried to keep since the first alert is that a notification must mean a fault doing damage while nobody is looking. Everything else is noise, and a system that alerts too much is one you have already stopped reading. The watcher that failed every time it stopped never sent a single notification. It only wrote a wrong word in a log, and I found it by reading the log.

Invisible Circles is still playing.