12 KiB
migadu_counters
Real-time mail counting for Migadu. mailqueuer tells it about every ingested
recipient; it increments per-address / per-domain / per-organisation daily
counters in Redis (the keys the web app already charts) and, when an increment
reaches a configured limit, notifies the web app, which decides what happens to
the customer. It replaces the rspamd Lua plugin migadu_counters.lua and the
ledger-derived quota in stats-merger. It returns no verdicts and enforces
nothing itself.
Design record: plans/2026-10-13-migadu-counters.md.
sender ──SMTP──▶ mizu ──POST /ingest──▶ mailqueuer
│ per recipient, after a successful publish:
│ fire-and-forget HTTP call, reply ignored
▼
POST /out {"trace_id","auth_user","mail_from","recipient"} ← X-Auth-User set
POST /in {"trace_id","mail_from","recipient","junk"} ← X-Auth-User empty
Authorization: Bearer <token>
▼
┌─────────────────────────────────────┐
rspamd_domains_db ───▶│ migadu_counters (:5555, admin :9095)│
(domain → org|limits) │ validate → 202 → queue → workers: │
│ lookup domain → MULTI/INCR → notify │──▶ GET admin.migadu.com/api/limits
└──────────────────┬──────────────────┘ (on the increment that crosses a limit)
▼
Redis Sentinel: writes → master, reads → replicas
DB 0: cnt/<A|D|O>/<id>/d/<YYYY-MM-DD>/<i|o> (TTL 31 d)
counterctl --address x@y [--date YYYY-MM-DD] ──GET /counters──▶ migadu_counters ──▶ replicas
API (for mailqueuer)
Every request: Authorization: Bearer <auth_token> (same value in mailqueuer's
config; the daemon refuses to start without one), Content-Type: application/json, body ≤ 64 KB, one JSON object. Unknown fields are ignored.
mailqueuer never waits for or acts on the reply.
Counting is decoupled from the request: /in and /out validate the body,
enqueue a job and answer 202 at once; a worker pool does the CDB lookup, the
Redis transaction and the notification under its own time budget, so a client
that hangs up or times out can no longer cancel a count. The counted values
appear in the counting log and in GET /counters, not in the reply.
POST /out |
{"trace_id":"…","auth_user":"x@example.com","mail_from":"x@example.com","recipient":"y@example.com"} — one call per recipient of an authenticated submission. Sender A/D/O outgoing +1 (if the sender's domain is hosted), recipient A/D/O incoming +1 (if the recipient's domain is hosted). auth_user and recipient are validated independently: whichever side is a usable address is counted, the other is logged as not usable; 400 only when neither is. A support/token login user@dom@SUFFIX counts under user@dom (the suffix is logged). |
POST /in |
{"trace_id":"…","mail_from":"s@gmail.com","recipient":"y@example.com","junk":false} — one call per recipient of inbound mail. Recipient A/D/O incoming +1 if hosted. junk:true is skipped unless count_junk = true. mail_from may be "" (bounce) or anything mizu accepted — it only feeds the exempt check and never rejects the request. |
GET /counters?address=… or ?domain=… [&date=YYYY-MM-DD] |
Today's (UTC) or a given day's values and limits, read from the replicas. |
Responses: 202 {"accepted":true} (validated and queued; skips such as
exempt/junk/unknown domain are decided by the worker and appear only in the
counting log), 4xx {"error":"…"} from the validation layer (405 method,
413 size, 401 token, 403 source IP, 415 content type, 400 JSON /
required fields / e-mail syntax), 503 {"error":"queue full"} when the
counting queue (queue_size) is full — the drop is counted in
dropped_total{route} and logged — or 503 on a handler timeout of the
synchronous part.
Rules: addresses are trimmed and lowercased, Unicode domains are converted to
punycode (the CDB key), and a plus-address tag is dropped so b+tag@… counts
against the mailbox b@… (the counter the web app reads); mail to or from an exempt address (support@migadu.com,
contact@migadu.com) is never counted; a domain that is not in
rspamd_domains_db is not hosted here and not counted; the sender of /in
mail is never counted; there is no reject/soft-reject — counting continues past
the limit.
Limits and keys
Limits come from the rspamd domains CDB (tinycdb, built by the agent):
example.com → 1049|O/i=200,O/o=20,D/i=200,D/o=20,A/i=200,A/o=20,info@/o=5.
<local>@/<flow> overrides the A/<flow> default for one mailbox; a missing
key means unlimited. The file is reloaded on SIGHUP and automatically when it
changes on disk (atomic mv is detected); a file that cannot be read or is
empty is rejected and the current table kept. A row that exists but cannot be
parsed (non-numeric org id) is treated as "not hosted" for that domain and
warned about in the app log; the other side of the request is still counted.
Always replace the file with
mv (as the agent's cdb -c + mv do) — rewriting it in place truncates the
inode the daemon holds open and breaks lookups until the next reload.
Redis keys are exactly the web app's (migadu_web core/redis_sentinel.ex):
cnt/<A|D|O>/<id>/d/<YYYY-MM-DD>/<i|o> in DB 0, UTC day, TTL 31 days,
INCR+EXPIRE in one MULTI/EXEC. The web app may DEL a day key to reset a
customer; the next crossing notifies again.
Redis failures are handled per outcome: a key whose own INCR fails (it holds
a non-number) is logged redis error on key … while the other keys are counted
and checked for crossings; when the EXEC reply is lost after the write (timeout,
reset) the keys are re-read once from the master and the block says redis outcome unknown … re-read from master, with a crossing reported when
count >= limit; a definite failure (connection refused) logs redis error: … (nothing counted).
The CDB is pulled from the agent host by contrib/pull-domains.sh (see
contrib/README.md): rsync, compare, verify, atomic mv, every 15 s.
A limit is "reached" by the increment that moves the counter from below the
limit to at or above it (old < limit ≤ new). Because INCRBY is atomic exactly
one increment crosses, so the web app is notified once per level/id/flow/day —
GET <org_url|address_url|domain_url> with $LEVEL $ID $FLOW $PERIOD(=d) $ADDRESS $COUNT $LIMIT substituted (URL-escaped), basic auth, in the
background, 5 attempts with exponential backoff; 4xx is not retried. An empty
URL disables notifications for that level.
Counting transaction log
Every request writes one block in the shape of the old Lua plugin's log lines
(rspamd's task id replaced by the trace id) to logging.counting_output:
2026-10-06 01:12:06.412 <ad8525> out; request auth_user=noah@example.com recipient=y@example.org mail_from=noah@example.com
2026-10-06 01:12:06.412 <ad8525> out; local domain: example.com account_id=116681
2026-10-06 01:12:06.412 <ad8525> out; domain: example.com limits_map={A/i=500, A/o=100, D/i=10000, D/o=2200, O/i=10000, O/o=2200}
2026-10-06 01:12:06.412 <ad8525> out; determined limit (level=[A] flow=[o] key=[cnt/A/noah@example.com/d/2026-10-06/o]) => 100
2026-10-06 01:12:06.412 <ad8525> out; recipient domain not in domains db: example.org (not counted)
2026-10-06 01:12:06.414 <ad8525> out; tracked address counters for noah@example.com => count=12 limit=100 reached=false
2026-10-06 01:12:06.414 <ad8525> out; tracked account counters for 116681 => count=2200 limit=2200 reached=true
2026-10-06 01:12:06.414 <ad8525> out; limit reached (level=[O] id=[116681] flow=[o] by=[noah@example.com]) => notifying org_url
2026-10-06 01:12:06.903 <ad8525> out; sent limit exceeded notification to url "https://admin.migadu.com/api/limits?…"
grep '<trace_id>' returns one transaction; grep 'skipped:' every request
that counted nothing; grep 'not usable' every side that failed validation
(sender address not usable: nobody (invalid e-mail address, not counted));
grep 'dropped:' every request refused by a full queue. A token login shows as
… login_suffix=SUPPORTTOKEN on the request line. counting_format = "json" emits the same events as one
JSON object per line. Domains with thousands of per-mailbox overrides print the
base keys plus the address's own overrides and a count of the rest.
Running
migadu_counters -config /usr/local/etc/migadu_counters/config.toml # default paths: /etc/…, /usr/local/etc/…
migadu_counters -version
Foreground process; two listeners: addr (API, default :5555) and
admin_addr (/health, /metrics, no auth). Signals: SIGHUP reopens both
log files and reloads the CDB; SIGTERM/SIGINT stop accepting, drain the
counting queue (every accepted request is counted), wait for in-flight
notifications (≤ 30 s) and exit 0. On FreeBSD run it like rcptd: an rc.d script
with /usr/sbin/daemon -f -H -p <pidfile> -o <logfile>; service migadu_counters reload sends SIGHUP. There is deliberately no self-daemonising
or pidfile option (daemon(8) owns both). Startup fails without a readable CDB;
an unreachable Redis is only warned about (/health answers 503 until it is
back).
See config.toml.example for every option. auth_token is mandatory and
unknown keys (typos such as auth-token) are fatal at startup. Secrets may
come from the environment: MIGADU_COUNTERS_AUTH_TOKEN,
MIGADU_COUNTERS_REDIS_PASSWORD, MIGADU_COUNTERS_NOTIFY_PASSWORD. redis.db must stay 0. Production uses
Sentinel (sentinels, master_name = "mymaster"): writes go to the master,
reads to a replica (falling back to the master when none is known); without
sentinels the static write_addr/read_addrs are used.
Metrics (/metrics on the admin port): migadu_counters_requests_total{route,status},
_request_duration_seconds{route}, _increments_total{level,flow},
_limit_reached_total{level,flow}, _notifications_total{level,result},
_redis_errors_total{op} (incr, incr_unknown, mget, ping),
_dropped_total{route}, _queue_depth, _queue_wait_seconds, _cdb_domains,
_cdb_reloads_total{result}.
counterctl
counterctl --address me@example.com # today, UTC
counterctl --address me@example.com --date 2026-10-12
counterctl --address me@example.com --out # outgoing columns only (--in: incoming)
counterctl --domain example.com --json
Reads addr/auth_token from the daemon's config (--config, default paths,
MIGADU_COUNTERS_AUTH_TOKEN overrides the token; --url overrides the
address) and calls GET /counters. Output:
me@example.com date 2026-10-13 (UTC) domain example.com org 1234
LEVEL ID IN LIMIT OUT LIMIT
A me@example.com 12 500 200! 200
D example.com 40 1000 9 -
O 1234 120 5000 30 1000
! marks a value at or above its limit, - an unlimited level. --in /
--out restrict the table to one flow (LEVEL ID IN LIMIT or LEVEL ID OUT LIMIT; both or neither show the full table; --json ignores them). --date takes
an ISO 8601 calendar date (YYYY-MM-DD); future dates are refused and dates
older than 31 days are warned about (keys have expired). Exit 0 on success, 1
on any error.
Development
make build # bin/migadu_counters, bin/counterctl
make test # go test -race ./... (tests use test_files/rspamd_domains_db.cdb and miniredis)
make vet # go vet + gofmt
make test-contrib # shell test for contrib/pull-domains.sh (needs tinycdb cdb + rsync)
make freebsd # static FreeBSD amd64 binaries for the Ansible role
Manual check with a local Redis: redis-server --port 6379, a config in static
mode (write_addr = "127.0.0.1:6379") pointing domains_cdb at
test_files/rspamd_domains_db.cdb, then
curl -s -XPOST localhost:5555/out -H 'Authorization: Bearer <token>' -H 'Content-Type: application/json' \
-d '{"trace_id":"t1","auth_user":"a@yagolivas.com","recipient":"x@gmail.com"}' # -> {"accepted":true}
redis-cli -n 0 KEYS 'cnt/*'
go run ./cmd/counterctl -config config.toml --address a@yagolivas.com --out