dimitrije/counters
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

First commit Go counter service

Browse Source
This commit is contained in:
Dimitrije Dimitrijevic
2026-10-15 02:12:18 +02:00
commit c43ff17627
59 changed files with 11205 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
contrib/README.md
+43
View File
@@ -0,0 +1,43 @@
# contrib: getting the domains CDB onto the service host
migadu_counters reads limits from the rspamd domains CDB that the agent builds
(`extract.sh` → `/usr/local/etc/agent/build/rspamd/rspamd_domains_db`). The
files here fetch it the way rcptd fetches its maps (`ansible-freebsd3`
`roles/rcptd/templates/{build.sh.j2,rcptd-maps.cron.j2}`); the future Ansible
role templates them.
| File | Purpose |
|---|---|
| `pull-domains.sh` | rsync the CDB from the agent, `cmp` against the installed `.orig`, and only on change verify (`cdb -s`) and install it with an atomic `mv`; `.pull-lock` with a 120 s max age; log to `/var/log/migadu_counters/pull.log` via `tee -a`. |
| `migadu_counters-pull.cron` | FreeBSD `@15` crontab fragment (15 s after the previous run completes) wrapped in `timeout -k 30 300`. |
| `pull-domains_test.sh` | Test against a local source directory (`RSYNC_SRC` override): unchanged → no `mv`, no reload; changed → atomic install, `.orig` updated, reload; corrupt or empty CDB → not installed, non-zero exit, old file untouched; lock handling. Run with `make test-contrib` (needs tinycdb's `cdb` and `rsync`). |
## What the Ansible role must template
Environment of `pull-domains.sh` (every variable has the production default
shown in the script):
- `AGENT_HOST` (`agent@<agent host>`) and `AGENT_PATH`; ssh is forced to IPv4
(`RSYNC_RSH="ssh -4"`), the agent's host key must be accepted beforehand
(rcptd has `accept-agent-hostkey.sh.j2` for that).
- `MAPS_DIR` (pulled copy), `DOMAINS_DIR` / `DOMAINS_CDB` (what `domains_cdb`
in `config.toml` points at), `LOG_FILE`, `LOCK_FILE`.
- `RELOAD_CMD` (default `service migadu_counters reload`, i.e. SIGHUP). The
daemon's fsnotify watcher already reloads on the `mv`; the reload is belt
and braces.
- The cron user (`migadu_counters`) must own `MAPS_DIR`, `DOMAINS_DIR` and the
log directory, and have the ssh key for the agent host. `timeout`, `rsync`
and `cdb` (tinycdb) must be installed.
## Rules the daemon relies on
- **Never rewrite the CDB in place.** The daemon keeps the file open; an
in-place rewrite truncates the inode under it and breaks every lookup until
the next reload. The script only ever `mv`s a verified dotfile over the
target, which is atomic on the same filesystem (`DOMAINS_DIR` must hold both).
- A file that does not open as a CDB or holds no records is refused and the
installed file kept; the daemon applies the same rule on reload.
- Memory stays flat across the 15-second cycles: a reload replaces the whole
in-memory DB object (2 KB index plus a bounded parse cache,
`max_cached_domains`, default 20 000 records) and the old one is closed after
a 30 s grace period. Unchanged pulls are a `cmp` and nothing else.
contrib/migadu_counters-pull.cron
+23
View File
@@ -0,0 +1,23 @@
# migadu_counters: pull the rspamd domains CDB from the agent host.
# Install as /etc/cron.d/migadu_counters-pull (system crontab fragment:
# five time/date fields OR an @ spec, then the user, then the command).
# The Ansible role templates the user and the script path.
#
# @15 reruns pull-domains.sh 15 seconds after the PREVIOUS run COMPLETES (a
# FreeBSD cron extension), not every 15s of wall clock — the same arrangement
# as rcptd's maps job:
# - cron cannot overlap this job with itself, so no flock wrapper is needed;
# the script's own .pull-lock still guards a manual or ansible-triggered
# run racing the scheduled one.
# - start times drift with run duration instead of landing on
# :00/:15/:30/:45 simultaneously across the fleet, which is what bursts
# ssh connections at the agent host (pf max-src-conn-rate, MaxStartups).
#
# timeout is what guarantees the schedule resumes: cron waits for completion
# before counting the 15s, so a hung run (ssh stall) would otherwise stall
# this job forever with nothing retrying. The budget only has to exceed the
# worst legitimate run, since overlap is no longer possible.
#
# pull-domains.sh appends to pull.log itself via tee -a, so only stderr is
# captured here; redirecting stdout to that same file would truncate it.
@15 migadu_counters /usr/bin/timeout -k 30 300 /usr/local/etc/migadu_counters/pull-domains.sh >/dev/null 2>>/var/log/migadu_counters/pull.log
contrib/pull-domains.sh
Executable
+93
View File
@@ -0,0 +1,93 @@
#!/usr/bin/env bash
#
# pull-domains.sh — fetch the rspamd domains CDB from the agent host and
# install it for migadu_counters, the way rcptd's build.sh pulls its maps.
#
# The agent already produces the finished CDB (extract.sh: cdb -m ->
# /usr/local/etc/agent/build/rspamd/rspamd_domains_db), so this is
# pull → compare → install:
#
# 1. rsync the file from the agent into $MAPS_DIR (ssh forced to IPv4, see
# rcptd: agenthost also has an IPv6 address not reachable from every host).
# 2. cmp against the installed baseline $DOMAINS_CDB.orig; identical → done.
# Most runs end here: the agent only rewrites the file when its content
# changed, so the 15-second cron cycle is a no-op cmp.
# 3. Changed: copy to a dotfile next to the target, verify it opens and has
# records (cdb -s), then mv it over $DOMAINS_CDB — atomic, so the daemon
# never sees a half-written file — and keep the pulled copy as .orig.
# The daemon's fsnotify watcher reloads on the mv; the explicit reload is
# belt and braces.
#
# Never rewrite $DOMAINS_CDB in place: the daemon keeps the file open and an
# in-place rewrite truncates the inode under it. This script only ever mv's.
#
# Environment (templated by the Ansible role; defaults below are the
# production paths). Tests override RSYNC_SRC with a local path.
#
set -u
AGENT_HOST=${AGENT_HOST:-agent@agenthost}
AGENT_PATH=${AGENT_PATH:-/usr/local/etc/agent/build/rspamd/rspamd_domains_db}
RSYNC_SRC=${RSYNC_SRC:-$AGENT_HOST:$AGENT_PATH} # a local path skips ssh (tests, co-located agent)
RSYNC_RSH=${RSYNC_RSH:-ssh -4}
MAPS_DIR=${MAPS_DIR:-/usr/local/etc/migadu_counters/maps}
DOMAINS_DIR=${DOMAINS_DIR:-/usr/local/etc/migadu_counters/db}
DOMAINS_CDB=${DOMAINS_CDB:-$DOMAINS_DIR/rspamd_domains_db}
LOG_FILE=${LOG_FILE:-/var/log/migadu_counters/pull.log}
LOCK_FILE=${LOCK_FILE:-$DOMAINS_DIR/.pull-lock}
LOCK_MAX_AGE=${LOCK_MAX_AGE:-120} # seconds; older locks are stale (crashed run)
RELOAD_CMD=${RELOAD_CMD:-service migadu_counters reload}
NAME=$(basename "$DOMAINS_CDB")
PULLED=$MAPS_DIR/$NAME
TMP=$DOMAINS_DIR/.$NAME.tmp
log() { echo "$(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE"; }
mtime() { stat -f %m "$1" 2>/dev/null || stat -c %Y "$1"; }
cleanup() { rm -f "$TMP" "$LOCK_FILE"; }
fail() { log "FAIL: $*"; cleanup; exit 1; }
mkdir -p "$MAPS_DIR" "$DOMAINS_DIR" "$(dirname "$LOG_FILE")" || { echo "cannot create directories" >&2; exit 1; }
# Lock: cron (@15) cannot overlap this job with itself; the lock guards a
# manual or ansible-triggered run racing the scheduled one.
if [ -f "$LOCK_FILE" ] && [ "$(mtime "$LOCK_FILE")" -ge $(( $(date +%s) - LOCK_MAX_AGE )) ]; then
log "previous instance is still running, skipping"
exit 0
fi
touch "$LOCK_FILE" || { echo "cannot create $LOCK_FILE" >&2; exit 1; }
trap 'rm -f "$TMP" "$LOCK_FILE"' EXIT
# 1. pull. --checksum: the agent can rebuild the file with the same size
# within the same second, which rsync's size+mtime quick check would miss;
# checksumming one ~9 MB file every cycle is cheap.
if ! rsync -a --checksum -e "$RSYNC_RSH" "$RSYNC_SRC" "$PULLED" >>"$LOG_FILE" 2>&1; then
fail "rsync from $RSYNC_SRC failed"
fi
[ -s "$PULLED" ] || fail "pulled $PULLED is missing or empty"
# 2. compare with the installed baseline
if [ -f "$DOMAINS_CDB" ] && cmp -s "$PULLED" "$DOMAINS_CDB.orig"; then
log "$NAME unchanged"
exit 0
fi
# 3. verify, then install atomically
rm -f "$TMP"
cp "$PULLED" "$TMP" || fail "copy to $TMP failed"
records=$(cdb -s "$TMP" 2>/dev/null | awk '/number of records/ { print $4 }')
if [ -z "$records" ]; then
fail "$PULLED is not a valid CDB (cdb -s failed), keeping the installed file"
fi
if [ "$records" -eq 0 ]; then
fail "$PULLED holds no records, keeping the installed file"
fi
chmod 0644 "$TMP"
mv "$TMP" "$DOMAINS_CDB" || fail "mv into place failed"
cp "$PULLED" "$DOMAINS_CDB.orig" || log "warning: could not update $DOMAINS_CDB.orig (next run will reinstall)"
log "$NAME installed ($records records)"
# The fsnotify watcher already picked up the mv; make sure anyway.
$RELOAD_CMD >>"$LOG_FILE" 2>&1 || true
exit 0
contrib/pull-domains_test.sh
Executable
+102
View File
@@ -0,0 +1,102 @@
#!/usr/bin/env bash
# Tests for pull-domains.sh against a local "agent" directory (RSYNC_SRC
# override) instead of ssh. Needs tinycdb's `cdb` and rsync in PATH.
# Run with `make test-contrib`.
set -u
HERE=$(cd "$(dirname "$0")" && pwd)
SCRIPT=$HERE/pull-domains.sh
TMP=$(mktemp -d)
trap 'rm -rf "$TMP"' EXIT
export RSYNC_SRC=$TMP/agent/rspamd_domains_db
export MAPS_DIR=$TMP/maps
export DOMAINS_DIR=$TMP/db
export DOMAINS_CDB=$DOMAINS_DIR/rspamd_domains_db
export LOG_FILE=$TMP/pull.log
export LOCK_FILE=$TMP/.pull-lock
export RELOAD_CMD="touch $TMP/reloaded"
mkdir -p "$TMP/agent" "$DOMAINS_DIR"
fails=0
pass() { echo "ok - $1"; }
fail() { echo "FAIL - $1"; fails=$((fails + 1)); }
check() { if eval "$2"; then pass "$1"; else fail "$1"; fi; }
# make_cdb FILE KEY VALUE: build a one-record CDB the way the agent does (cdb -c).
make_cdb() {
printf '+%d,%d:%s->%s\n\n' "${#2}" "${#3}" "$2" "$3" | cdb -c "$1"
}
inode() { stat -f %i "$1" 2>/dev/null || stat -c %i "$1"; }
# --- 1. first install -------------------------------------------------------
make_cdb "$RSYNC_SRC" example.com "1049|A/i=200,A/o=20"
rm -f "$TMP/reloaded"
"$SCRIPT" >/dev/null 2>&1; rc=$?
check "first run exits 0" "[ $rc -eq 0 ]"
check "first run installs the CDB" "[ -f '$DOMAINS_CDB' ] && cmp -s '$RSYNC_SRC' '$DOMAINS_CDB'"
check "first run keeps the .orig baseline" "cmp -s '$RSYNC_SRC' '$DOMAINS_CDB.orig'"
check "first run reloads the daemon" "[ -f '$TMP/reloaded' ]"
check "no temp file left behind" "! ls '$DOMAINS_DIR'/.rspamd_domains_db.* >/dev/null 2>&1"
check "lock released" "[ ! -f '$LOCK_FILE' ]"
check "log written" "grep -q 'rspamd_domains_db' '$LOG_FILE'"
# --- 2. unchanged source ----------------------------------------------------
before=$(inode "$DOMAINS_CDB")
rm -f "$TMP/reloaded"
"$SCRIPT" >/dev/null 2>&1; rc=$?
check "unchanged run exits 0" "[ $rc -eq 0 ]"
check "unchanged run does not replace the file (same inode)" "[ '$(inode "$DOMAINS_CDB")' = '$before' ]"
check "unchanged run does not reload" "[ ! -f '$TMP/reloaded' ]"
check "unchanged run logs the skip" "grep -q 'unchanged' '$LOG_FILE'"
# --- 3. changed source -------------------------------------------------------
make_cdb "$RSYNC_SRC" example.com "1049|A/i=500,A/o=50"
rm -f "$TMP/reloaded"
"$SCRIPT" >/dev/null 2>&1; rc=$?
check "changed run exits 0" "[ $rc -eq 0 ]"
check "changed run installs atomically (new inode)" "[ '$(inode "$DOMAINS_CDB")' != '$before' ]"
check "changed run installs the new content" "cmp -s '$RSYNC_SRC' '$DOMAINS_CDB'"
check "changed run updates .orig" "cmp -s '$RSYNC_SRC' '$DOMAINS_CDB.orig'"
check "changed run reloads the daemon" "[ -f '$TMP/reloaded' ]"
check "changed run logs the install" "grep -q 'installed' '$LOG_FILE'"
# --- 4. corrupt source -------------------------------------------------------
good_inode=$(inode "$DOMAINS_CDB")
cp "$DOMAINS_CDB" "$TMP/good.copy"
printf 'garbage' > "$RSYNC_SRC"
rm -f "$TMP/reloaded"
"$SCRIPT" >/dev/null 2>&1; rc=$?
check "corrupt source exits non-zero" "[ $rc -ne 0 ]"
check "corrupt source leaves the installed file untouched" "[ '$(inode "$DOMAINS_CDB")' = '$good_inode' ] && cmp -s '$DOMAINS_CDB' '$TMP/good.copy'"
check "corrupt source does not touch .orig" "cmp -s '$DOMAINS_CDB.orig' '$TMP/good.copy'"
check "corrupt source does not reload" "[ ! -f '$TMP/reloaded' ]"
check "corrupt source leaves no temp file" "! ls '$DOMAINS_DIR'/.rspamd_domains_db.* >/dev/null 2>&1"
check "corrupt source releases the lock" "[ ! -f '$LOCK_FILE' ]"
check "corrupt source is logged" "grep -q 'not a valid CDB' '$LOG_FILE'"
# An empty CDB (zero records) is just as useless: refused too.
printf '\n' | cdb -c "$RSYNC_SRC"
"$SCRIPT" >/dev/null 2>&1; rc=$?
check "empty CDB is refused" "[ $rc -ne 0 ] && cmp -s '$DOMAINS_CDB' '$TMP/good.copy'"
# --- 5. lock file -------------------------------------------------------------
make_cdb "$RSYNC_SRC" example.com "1049|A/i=1,A/o=1"
touch "$LOCK_FILE"
"$SCRIPT" >/dev/null 2>&1; rc=$?
check "fresh lock skips the run (exit 0)" "[ $rc -eq 0 ]"
check "fresh lock leaves the file alone" "cmp -s '$DOMAINS_CDB' '$TMP/good.copy'"
check "fresh lock is logged" "grep -q 'still running' '$LOG_FILE'"
touch -t 200001010000 "$LOCK_FILE" # stale
"$SCRIPT" >/dev/null 2>&1; rc=$?
check "stale lock is ignored" "[ $rc -eq 0 ] && cmp -s '$RSYNC_SRC' '$DOMAINS_CDB'"
# --- 6. the pull itself fails ------------------------------------------------
rm -f "$RSYNC_SRC"
"$SCRIPT" >/dev/null 2>&1; rc=$?
check "missing source exits non-zero and keeps the installed file" "[ $rc -ne 0 ] && [ -f '$DOMAINS_CDB' ]"
check "failed pull releases the lock" "[ ! -f '$LOCK_FILE' ]"
echo
if [ $fails -eq 0 ]; then echo "all contrib tests passed"; else echo "$fails contrib test(s) FAILED"; exit 1; fi