12 October 2026

📌 Sunday night in Gourock that turned into Monday while I wasn't looking, The Moth by Devin Townsend in the headphones. On my phone, ntfy tells me every time the shared admin account logs into the Bottega's WordPress. It says nothing about the rest. A pedantic thing to fix, so I fixed it.

one line per service

Everything the Bottega runs sits behind a reverse proxy, and the proxy keeps an access log. A login is a request like any other, so the first idea was the lazy one: follow the log, recognise the login, send the alert.

For half of the services that is the whole story. The login form posts to one path, and the answer tells you how it went.

POST /admin/login   200   the form again: wrong password
POST /admin/login   302   in

One line in a table per service - host, path, the status code that means "in" - and five of them were done.

The log does not say who logged in (the user name travels in the body, which the proxy does not write down). On a service with a single admin account that does not matter. On the others it does, and that is where the evening went.

the same line twice

The webmail was the first to refuse. It posts everything to whatever URL is open, so the path says nothing. Worse, a logout and a successful login leave the same line.

22:04:01   POST   303   1 byte      then a page of 166966 bytes   logout
22:04:37   POST   303   961 bytes   then a page of 166961 bytes   wrong password
22:04:54   POST   303   1 byte      then a page of 169475 bytes   logged in

The only difference between going out and coming in is the size of the page served right after, and a rule built on "the next page is bigger than this" breaks the day the inbox changes or the webmail is upgraded. The container itself logs nothing at login - I looked.

What does differ is behaviour. A browser that has just logged in starts talking in the background within a second. A browser that has just logged out stays quiet: between 22:04:01 and 22:04:54 there is not a single ordinary request from me in the log. So the rule became a sequence - that redirect, followed within a few seconds by a normal answer from the same address - and the script had to learn to remember things for a few seconds. I have one observation of this, not a hundred. It was a clean one.

who, but not from where

Kanboard is used by the whole editorial group, and only one account there is worth an alert. The proxy log knows where a login came from and not who it was. Kanboard knows who: it keeps a table of last logins, the same one it shows in each user's profile. I opened it, read-only, expecting the answer.

proxy log   22:35:24   POST /login/check   302   <the visitor's address>
database    22:35:24   <the shared admin>        <the proxy's own address>

Kanboard sees every request arrive from the proxy, so that is the address it writes down. Each source had half of the fact. The two halves agree on one thing, the second it happened, and that was enough: when the log shows a login from somewhere unexpected, the script asks the database who came in during those few seconds. A wrong password leaves no row, so there is nothing to recognise and nothing to get wrong.

(There was no sqlite3 on the host. Python was already there for the newsletter and reads SQLite on its own, so nothing got installed.)

Mattermost turned out to be the same shape with a different database: three rows per login in its audit table, the user name in each, and again the proxy's address where the visitor's should be. Same join, same few seconds.

the other machine

Sympa, the mailing list manager, lives on the mail server and is published by the proxy on another machine. Its journal is generous:

do_login(<user>) ... [client <address>]
Incorrect password for user <user>
Authentication failed

or, when it goes well, the same first line followed by a redirect. Who, and how it went. The address, once more, was the proxy's.

This time the two halves were on different machines, and joining logs across servers by timestamp is the kind of thing I'd rather not own. The proxy was already sending the visitor's address in a header; the nginx in front of Sympa just wasn't using it. Two lines.

Except that right under where those two lines would go there were these:

allow <redacted>;   # the proxy
deny  all;

The moment nginx starts believing the header, allow judges the visitor's address instead of the proxy's, and everybody gets a 403. The guard had to be rewritten to look at the connection, not at the visitor:

set_real_ip_from <redacted>;
real_ip_header   X-Real-IP;

if ($realip_remote_addr != "<redacted>") {
    return 403;
}

After the reload Sympa's journal started showing real addresses, a request that did not come through the proxy still got its 403, and a small watcher on the mail server does the rest: it reads the login line, waits for the next line from the same process, and decides.

three machines, one file

The proxy has an admin panel of its own, and that one never shows up in an access log: it is not published at all, I reach it through an SSH tunnel. So the alert that covers it is an alert on SSH, and an alert on SSH is worth having anyway.

sshd writes one line for every login that succeeds:

sshd[<pid>]: Accepted publickey for <user> from <address> port <port> ssh2

Who and from where, both in the same line, on the same machine. After the rest of the evening it felt like cheating.

The rule is the reverse of the others. Everywhere else I name the account worth an alert and stay quiet about the rest. Here I name the one login that is expected - the usual user from the usual address - and everything else alerts: that user from anywhere else, anybody else from anywhere, the usual address included.

I wrote it on the first machine and was about to call it done. Then I counted the machines. There are three - the lab, the web server, the mail server - and an alert that exists on one of them is a habit, not a rule. So it became one script and one systemd unit, with nothing in them that belongs to a single host (the machine's name in the title of the alert is asked for at runtime), copied as they are to all three and compared by checksum once installed. Six checksums, two values.

Then the test, three times: wifi off, VPN off, phone as a hotspot, log in. Three alerts, one per machine, each with a mobile operator's address in it.

and then...

Ten services and three machines now tell me when somebody walks in. What they do not tell me is when the thing that does the telling has stopped - and one of the watchers, as it turned out, had been announcing its own failure every time it was asked politely to stop. That is tomorrow's note.