First commit Go counter service

2026-10-15 02:12:18 +02:00
commit c43ff17627
59 changed files with 11205 additions and 0 deletions
@@ -0,0 +1,43 @@
+# contrib: getting the domains CDB onto the service host
+
+migadu_counters reads limits from the rspamd domains CDB that the agent builds
+(`extract.sh` → `/usr/local/etc/agent/build/rspamd/rspamd_domains_db`). The
+files here fetch it the way rcptd fetches its maps (`ansible-freebsd3`
+`roles/rcptd/templates/{build.sh.j2,rcptd-maps.cron.j2}`); the future Ansible
+role templates them.
+
+| File | Purpose |
+|---|---|
+| `pull-domains.sh` | rsync the CDB from the agent, `cmp` against the installed `.orig`, and only on change verify (`cdb -s`) and install it with an atomic `mv`; `.pull-lock` with a 120 s max age; log to `/var/log/migadu_counters/pull.log` via `tee -a`. |
+| `migadu_counters-pull.cron` | FreeBSD `@15` crontab fragment (15 s after the previous run completes) wrapped in `timeout -k 30 300`. |
+| `pull-domains_test.sh` | Test against a local source directory (`RSYNC_SRC` override): unchanged → no `mv`, no reload; changed → atomic install, `.orig` updated, reload; corrupt or empty CDB → not installed, non-zero exit, old file untouched; lock handling. Run with `make test-contrib` (needs tinycdb's `cdb` and `rsync`). |
+
+## What the Ansible role must template
+
+Environment of `pull-domains.sh` (every variable has the production default
+shown in the script):
+
+- `AGENT_HOST` (`agent@<agent host>`) and `AGENT_PATH`; ssh is forced to IPv4
+  (`RSYNC_RSH="ssh -4"`), the agent's host key must be accepted beforehand
+  (rcptd has `accept-agent-hostkey.sh.j2` for that).
+- `MAPS_DIR` (pulled copy), `DOMAINS_DIR` / `DOMAINS_CDB` (what `domains_cdb`
+  in `config.toml` points at), `LOG_FILE`, `LOCK_FILE`.
+- `RELOAD_CMD` (default `service migadu_counters reload`, i.e. SIGHUP). The
+  daemon's fsnotify watcher already reloads on the `mv`; the reload is belt
+  and braces.
+- The cron user (`migadu_counters`) must own `MAPS_DIR`, `DOMAINS_DIR` and the
+  log directory, and have the ssh key for the agent host. `timeout`, `rsync`
+  and `cdb` (tinycdb) must be installed.
+
+## Rules the daemon relies on
+
+- **Never rewrite the CDB in place.** The daemon keeps the file open; an
+  in-place rewrite truncates the inode under it and breaks every lookup until
+  the next reload. The script only ever `mv`s a verified dotfile over the
+  target, which is atomic on the same filesystem (`DOMAINS_DIR` must hold both).
+- A file that does not open as a CDB or holds no records is refused and the
+  installed file kept; the daemon applies the same rule on reload.
+- Memory stays flat across the 15-second cycles: a reload replaces the whole
+  in-memory DB object (2 KB index plus a bounded parse cache,
+  `max_cached_domains`, default 20 000 records) and the old one is closed after
+  a 30 s grace period. Unchanged pulls are a `cmp` and nothing else.