dimitrije/counters
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

First commit Go counter service

Browse Source
This commit is contained in:
Dimitrije Dimitrijevic
2026-10-15 02:12:18 +02:00
commit c43ff17627
59 changed files with 11205 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plans/2026-10-13-migadu-counters.md
+480
View File
@@ -0,0 +1,480 @@
# Plan: `migadu_counters` — real-time mail counting service (Go)
## Context
Daily send/receive limits used to be counted and enforced by the rspamd Lua plugin
`ansible-freebsd2/roles/rspamd-postfix/templates/rspamd/plugins.d/migadu_counters.lua`
(Redis counters per address + org, soft-reject/reject on breach, web app notified once).
Mail now flows mizu → mailqueuer and bypasses rspamd, so nothing counts in real time.
`stats-merger/internal/quota` recomputes usage from the delivery ledger after each merge, but it is
reporting-only today: only fernrohr reads `/api/quota/{status,stats,overquota}`; nothing calls `/check`.
`migadu_counters` is a new standalone Go service that becomes the single counting source and the
enforcement trigger:
- mailqueuer calls it **fire-and-forget**, once per ingested recipient, from its publish loop;
- one route per flow: `POST /in`, `POST /out` (`/rel` later, not planned here);
- it increments Redis counters at **Address / Domain / Organization** level, in the key format the
web app already reads. The domain level is turned back on (Lua defined it but never tracked it);
- when an increment crosses a limit it **notifies the web app** (`migadu_web` `GET /api/limits`), which
decides what to do (notify the user, remove sending/receiving);
- a small CLI, `counterctl`, shows an address's incoming/outgoing counters for today or a given date
(see "CLI: counterctl" below).
The service returns no verdicts and enforces nothing itself.
Out of scope: `/rel`, the mailqueuer client change, retiring stats-merger quota, the web app's `D` branch,
deployment (see prerequisites at the end).
## How it fits together
```
sender ──SMTP──▶ mizu ──POST /ingest──▶ mailqueuer
X-Trace-Id, X-Mail-From, │ one queue message per recipient;
X-Mail-To, X-Auth-User, │ after each successful publish:
X-Junk │ fire-and-forget HTTP call, reply ignored
▼
POST /out {"trace_id","auth_user","mail_from","recipient"} ← X-Auth-User set
POST /in {"trace_id","mail_from","recipient","junk"} ← X-Auth-User empty
Authorization: Bearer <token> (mailqueuer sends it, migadu_counters checks it; 401 if wrong)
│
▼
┌─────────────────────────────────────┐
│ migadu_counters (daemon :8095) │
│ ┌─────────────────────────────────┐ │ rejected here, nothing counted:
│ │ HTTP validation layer │ │ 405 method · 413 body > 64 KB
│ │ method → size → bearer token → │ │ 401 bad/missing token · 403 source IP
│ │ source IP → content-type → JSON │ │ 415 content-type · 400 json / fields /
│ │ → fields → e-mail syntax │ │ e-mail syntax · 503 handler timeout
│ └───────────────┬─────────────────┘ │
│ ▼ validated request │
rspamd_domains_db ───▶│ counting: 1. domain in CDB? → org_id, limits │
(CDB: domain → │ 2. MULTI: INCR A/D/O keys, EXPIRE 31 d │
org_id | limits) │ 3. did this INCR cross a limit? → notify │
└───┬─────────────────────────┬───────┘ admin :9095 /health /metrics
│ │
Redis Sentinel ◀──┘ └──────▶ GET https://admin.migadu.com/api/limits
writes → master ?level=&id=&flow=&period=d&by=&count=
reads → replicas (basic auth) → migadu_web notifies the user
DB 0: cnt/<A|D|O>/<id>/d/<YYYY-MM-DD>/<i|o> or disables sending/receiving
counterctl --address x@y [--date YYYY-MM-DD] ──GET /counters (Bearer)──▶ migadu_counters ──▶ Redis replicas
```
## Existing pieces reused or matched
- **Limits**: the CDB the Lua plugin reads, `/usr/local/etc/rspamd/db/rspamd_domains_db`, built by
`ansible-freebsd2/roles/agent/templates/extract.sh.j2:478` with tinycdb `cdb -m` (no NUL terminators,
exact-byte lookups like rcptd). Key = `domain`, value = `<org_id>|O/i=..,O/o=..,D/i=..,A/o=..,alice@/o=..`.
Parsing and limit resolution follow `stats-merger/internal/quota/quota.go` (`ParseLimits` L66,
`DomainLimits.limit` L116): keys lowercased, malformed pairs skipped, `<local>@/<flow>` overrides
`A/<flow>`, missing key = unlimited. A domain present in the CDB is "Migadu-hosted".
- **Redis keys** — must match the web app's key builder (`migadu_web/apps/core/lib/core/redis_sentinel.ex:73-103`):
`cnt/<A|D|O>/<id>/d/<YYYY-MM-DD>/<i|o>` with id = address / domain / org_id, **date in UTC**
(web app uses `Date.utc_today()`), **Redis DB 0** (web app reads `:redis_sentinel_db0`; rspamd uses
`redis_rspamd_db: 0`). TTL 31 days. DB 1 and the `/s` storage keys belong to the web app; never touched.
The web app may DEL today's keys to reset a customer (`redis_sentinel.ex:145-153`); counting must keep
working after that, and a second crossing must notify again.
- **Web app notification**: `GET https://admin.migadu.com/api/limits?level=&id=&flow=&period=&by=&count=`
with basic auth (`apps/admin_web/lib/admin_web/controllers/internal_api/notification_controller.ex:33`,
`router.ex:99` `Plug.BasicAuth`). All six params are pattern-matched, so all must be present.
`id` = address for A, domain name for D, org id for O. Credentials: `vault_migadu_web.internal_api_basic_auth_*`
(see `rspamd-postfix/templates/rspamd/local.d/migadu_counters.conf.j2`).
- **Redis topology**: `ansible-freebsd2/roles/redis-sentinel`, `inventory/hosts.ini:160`: hosts `redisSentinel0-2`,
Redis :6379 + Sentinel :26379, master name `mymaster`, 1 master + 2 replicas, one password
(`vault_redis_sentinel_password`) for Redis; Sentinel itself has no `requirepass`.
- **Service conventions** from `~/Work/rcptd` (`cmd/rcptd/main.go`): TOML config via `-config` + default paths,
bearer `auth_token` with env override, slog, CDB behind `atomic.Pointer` with a 30 s grace close,
SIGHUP + fsnotify reload (`internal/watcher/watcher.go` table mode), admin port with `/health` + `/metrics`,
graceful shutdown. No dependency on rcptd's `shared` module (not present locally).
- **Libraries** (in the module cache): `github.com/colinmarc/cdb`, `github.com/BurntSushi/toml` v1.6.0,
`github.com/fsnotify/fsnotify` v1.9.0, `github.com/redis/go-redis/v9` v9.7.3, `github.com/prometheus/client_golang`;
tests: `github.com/alicebob/miniredis/v2` v2.34.0.
## API (called by mailqueuer, fire-and-forget, one request per recipient)
> **Revised 2026-10-14 (review round 1, `review/README.md` #1–#6):** `/in` and `/out` now validate, enqueue
> and answer `202 {"accepted":true}`; counting runs in a worker pool (`queue_size`, `workers`) under its own
> time budget, and the counted values appear only in the counting log and `GET /counters`. A full queue answers
> `503 {"error":"queue full"}`. Only the counted sides are validated (`mail_from` never rejects; `auth_user` and
> `recipient` independently), IDN domains are converted to punycode, `user@dom@SUFFIX` logins count under the
> base address. `INCR` replaces `INCRBY`; per-key EXEC failures and lost replies are handled individually
> (re-read from the master). Parsed CDB records are cached (`max_cached_domains`). `auth_token` is mandatory,
> unknown config keys are fatal, the default API port is `:5555`. The `200 {"counted":…}` contract below is
> the original design and is superseded by the README.
mailqueuer's ingest handler is already per recipient: it loops over `toPublish`
(`mailqueuer/internal/handler/ingest.go:235-306`), publishes one queue message per recipient (own uuid `id`,
L237) and marks dedup only after a successful publish (L263-272). Both routes are called from inside that loop,
right after a successful `Publish`, with exactly one recipient. The route is chosen by `X-Auth-User`:
non-empty → `/out`, empty → `/in`. mailqueuer never waits for or acts on the response.
**Authentication**: every request carries `Authorization: Bearer <token>` — mailqueuer sends it (the same
header it already sends to rcptd, `internal/routing/routing.go:256-258`), migadu_counters validates it
(constant-time compare against `auth_token`; missing/wrong → `401`, nothing counted). One shared token,
set in both configs; no other auth. All routes take a JSON body ≤ 64 KB. Processing is **synchronous**: one Redis
round trip, then the response. Only web-app notifications run in the background.
- `200` `{"counted":[{"level":"A","id":"x@example.com","flow":"o","count":12,"limit":100,"reached":false},…]}`
(`reached: true` on the increment that crossed the limit → notification triggered). Skipped requests return
`{"counted":[],"skipped":"exempt|unknown_domain|junk"}`.
- `400` invalid body/address, `401` bad token, `503` Redis error (nothing counted).
### `POST /out` — authenticated submission
```json
{ "trace_id": "…", "auth_user": "x@example.com", "mail_from": "x@example.com", "recipient": "y@example.com" }
```
1. **Sender outgoing +1** on A(`auth_user`) / D(its domain) / O(its org) — only if `auth_user`'s domain is in the CDB.
2. **Recipient incoming +1** on A / D / O of the recipient's domain row — only if that domain is in the CDB
(any Migadu-hosted domain, any org). That mail never passes `/in`.
Example: `x@example.com` → `y@example.com`, `s@gmail.com` = two `/out` calls.
Call 1: out A x +1, D example.com +1, O +1; in A y +1, D example.com +1, O +1. Call 2: out +1 on each; gmail not
in CDB → no incoming. Totals: outgoing +2 per level, incoming for y +1 per level.
### `POST /in` — inbound mail from outside
```json
{ "trace_id": "…", "mail_from": "s@gmail.com", "recipient": "y@example.com", "junk": false }
```
Recipient incoming +1 on A / D / O if its domain is in the CDB. The sender is never counted.
`junk = true` is skipped unless `count_junk = true` (Lua only counted `no action` mail).
### `GET /counters?address=…` or `?domain=…` `[&date=YYYY-MM-DD]`
Read-only view (used by `counterctl`), served from the **read client**. `date` is an ISO 8601 calendar date
(`YYYY-MM-DD`, interpreted as a UTC day); default = today UTC; a future date is `400`. Keys live 31 days, so
older dates return zeros. Exactly one of `address`/`domain` is required.
```json
{ "query": "me@example.com", "mode": "address", "date": "2026-10-13",
"managed": true, "exempt": false, "domain": "example.com", "org_id": 1234,
"levels": [
{"level": "A", "id": "me@example.com", "in": 12, "in_limit": 500, "out": 3, "out_limit": 200},
{"level": "D", "id": "example.com", "in": 40, "in_limit": 1000, "out": 9, "out_limit": 400},
{"level": "O", "id": "1234", "in": 120, "in_limit": 5000, "out": 30, "out_limit": 1000} ] }
```
`in_limit`/`out_limit` are `null` when unlimited. `mode: "domain"` returns the D and O rows only.
`managed: false` (domain not in the CDB) still returns the raw counters with `null` limits — Lua-era keys may exist.
Values are read with one `MGET` of the ≤ 6 keys.
### Shared rules
- Addresses are trimmed and lowercased; validation = one `@`, non-empty parts, ≤ 254 chars.
- **Exempt list** (default `support@migadu.com`, `contact@migadu.com`): if `auth_user`, `mail_from` or
`recipient` is exempt, nothing is counted for that request (Lua: mail to/from support never costs quota).
- `trace_id` (mizu's `X-Trace-Id`, shared by all recipients of one message) is logged for correlation only;
mailqueuer's internal per-recipient queue id is **not** sent. No Redis idempotency keys: mailqueuer already
guarantees at-most-once per (trace, recipient) publish; the residual case (mailqueuer restart + mizu retry)
also double-delivers the mail, so the count stays truthful.
- `O` and `D` limits come from the domain row that triggered the increment (sender's row for outgoing,
recipient's row for incoming) — same caveat as Lua if domains of one org carry different `O/*` values.
## HTTP validation layer (mailqueuer → migadu_counters)
Everything mailqueuer sends passes this layer **before** any CDB or Redis access; a request that fails any step is
answered with a JSON error `{"error":"…"}` and nothing is counted. Two parts:
**1. Middleware chain** (`internal/server/middleware.go`, applied in this order to `/in`, `/out`; `/counters`
gets the same chain with method GET and no body steps):
| # | Step | Rule | Failure |
|---|------|------|---------|
| 1 | recover | panic in a handler → log with stack, keep the daemon alive | `500` |
| 2 | method | `POST` only (`GET` for `/counters`) | `405` + `Allow` header |
| 3 | body size | `http.MaxBytesReader` 64 KB | `413` |
| 4 | bearer token | `Authorization: Bearer <token>`, constant-time compare with `auth_token` | `401` + `WWW-Authenticate: Bearer` |
| 5 | source IP | optional `allowed_cidrs` (config); matched on `RemoteAddr` only — `X-Forwarded-For` is never trusted (no proxy in front) | `403` |
| 6 | content type | `Content-Type: application/json` (parameters such as `charset` ignored) | `415` |
| 7 | JSON decode | exactly one JSON object; syntax errors rejected; **unknown fields ignored** so mailqueuer can add fields without breaking counting | `400` |
| 8 | request validation | per-route field rules below | `400` |
| 9 | timeout | `http.TimeoutHandler` (`handler_timeout`, default 4 s) around the counting work | `503` |
Every rejection is logged at `warn` with route, remote address, status, reason and `trace_id` (if parsable) and
counted in `requests_total{route,status}` — a burst of `400`/`401` from mailqueuer means contract drift or a
token mismatch and should alert.
**2. Request validators** (`internal/api/validate.go`, pure functions, table-tested):
| Field | `/out` | `/in` | Rule |
|-------|--------|-------|------|
| `trace_id` | optional | optional | ≤ 128 chars, printable ASCII; control characters/newlines stripped before it reaches any log line (log-injection guard) |
| `auth_user` | **required** | — (ignored) | e-mail syntax |
| `recipient` | **required** | **required** | e-mail syntax |
| `mail_from` | optional | optional (`""` = bounce sender `<>`) | e-mail syntax when non-empty |
| `junk` | — (ignored) | optional, default `false` | boolean |
E-mail syntax = trimmed, lowercased, ≤ 254 chars, exactly one `@`, non-empty local part and domain, no
whitespace/control characters; the domain is ASCII (punycode accepted). Deliberately permissive: the CDB lookup
decides whether an address is counted, the validator only guards against garbage and log injection.
Errors name the field: `{"error":"recipient: missing"}`, `{"error":"auth_user: invalid e-mail address"}`.
Config: `allowed_cidrs = ["10.0.0.0/8"]` (optional; empty = any source), `handler_timeout` (see daemon section).
The response codes mailqueuer may see are therefore: `200` counted/skipped, `4xx` rejected by this layer,
`503` Redis unavailable or timeout — it ignores all of them (fire-and-forget), they exist for operators and tests.
## Redis (`internal/store`)
Config has a **write** side and a **read** side:
- **Sentinel mode (production)**: `sentinels`, `master_name`, `password` (+ optional `sentinel_password`).
Write = `redis.NewFailoverClient` (follows failover). Read = `NewFailoverClient{ReplicaOnly: true}`; go-redis
picks a random replica and falls back to the master when none is known (`sentinel.go:526`).
- **Static mode (dev/tests, when `sentinels` is empty)**: `write_addr` + `read_addrs` (round-robin; defaults to
`write_addr`).
- Both pooled (`pool_size`); one `timeout_ms` for dial/read/write. `db` must be `0` (validated, startup error otherwise).
Counting = one `TxPipeline` (MULTI/EXEC) per request: `INCRBY key 1` + `EXPIRE key 2678400` for every affected key
(≤ 6 keys). EXEC returns the new values, so no read-after-write and replica lag is irrelevant. Increments are
atomic (fixes the Lua GET-then-INCR race); counting continues past the limit.
**Limit crossing**: for each incremented key with a configured limit `L`, notify iff `new − amount < L ≤ new`
(with amount 1: `new == L`). Exactly one increment can cross, across all workers/instances, with no marker key.
After a web-app reset (DEL) the counter crosses again and notifies again. Accepted gap (same as Lua): a limit
lowered below the current count mid-day produces no crossing.
Redis errors → `503`, log, `redis_errors_total{op}`.
## Notification (`internal/notify`)
- One URL per level in `[notify]`: `org_url` (O), `address_url` (A), `domain_url` (D). Empty URL = that level is
counted and `limit_reached_total` is incremented, but no HTTP call (useful while the web app's `D` branch is a TODO).
Basic auth and timeout are shared.
- `GET <url>` with template vars `$LEVEL $ID $FLOW $PERIOD(=d) $ADDRESS $COUNT $LIMIT`, each `url.QueryEscape`d.
`ADDRESS` = the address that triggered it (sender for outgoing, recipient for incoming). `COUNT` = new value.
- Runs in a background goroutine (semaphore of 8 to not flood the web app): up to 5 attempts, exponential
backoff 1 s → 16 s. Final failure → error log + `notifications_total{level,result="failed"}` (alertable).
Shutdown waits ≤ 10 s for in-flight notifications.
## CLI: `counterctl`
```
counterctl --address me@dimitrijedimitrijevic.com # today (UTC)
counterctl --address me@dimitrijedimitrijevic.com --date 2026-10-12
counterctl --domain dimitrijedimitrijevic.com --json
```
- Flags (Go `flag`, so `-x` and `--x` both work): `--address` | `--domain` (exactly one), `--date YYYY-MM-DD`
(ISO 8601 calendar date; default today UTC; parsed with `time.Parse("2006-01-02")`; future date = error;
older than 31 days = warning on stderr that keys have expired), `--json` (raw API response), `--config PATH`,
`--url` (override), `--version`.
- Talks to the running service's `GET /counters` instead of Redis directly — one code path for key building
and limit resolution, no Redis/CDB access needed from the operator's shell, same approach as `rcptctl`.
Config discovery mirrors the service and `rcptctl` (`cmd/rcptctl/main.go:65-75`): `--config`, else
`/etc/migadu_counters/config.toml`, `/usr/local/etc/migadu_counters/config.toml`; reads `addr` and
`auth_token` only (`MIGADU_COUNTERS_AUTH_TOKEN` overrides); `addr` like `:8095` becomes
`http://localhost:8095` (`rcptctl/api.go:52-60`); warns when the host is not local (token over plaintext).
- Output (`text/tabwriter`), limits shown as `-` when unlimited, `!` appended when value ≥ limit:
```
me@dimitrijedimitrijevic.com date 2026-10-13 (UTC) domain dimitrijedimitrijevic.com org 1234
LEVEL ID IN LIMIT OUT LIMIT
A me@dimitrijedimitrijevic.com 12 500 3 200
D dimitrijedimitrijevic.com 40 1000 9 400
O 1234 120 5000 30 1000
```
`managed: false` prints a `domain is not in rspamd_domains_db (not counted)` note after the table;
`exempt: true` prints `address is exempt from counting`.
- Exit codes: `0` success, `1` usage error / bad date / service unreachable / non-200 (message on stderr).
## Counting transaction log (`internal/txlog`)
A second logger, separate from the slog application log, that writes the Lua plugin's step-by-step lines for
**every** counting transaction (the Lua `infox` lines from `migadu_counters.lua:217/235/266/268/271/283`),
so existing grep habits keep working. rspamd's task id `<ad8525>` is replaced by `<trace_id>` (shared with
mizu/mailqueuer logs via `X-Trace-Id`), and the route is added. `breach=` becomes `reached=` because the
meaning changed (Lua: already over → rejected; here: this increment crossed the limit).
Text format (default), timestamps UTC with milliseconds, one block per request:
```
2026-10-06 01:12:06.412 <ad8525> out; request auth_user=noah@wearemiraclesnow.com recipient=y@example.org mail_from=noah@wearemiraclesnow.com
2026-10-06 01:12:06.412 <ad8525> out; local domain: wearemiraclesnow.com account_id=116681
2026-10-06 01:12:06.412 <ad8525> out; domain: wearemiraclesnow.com limits_map={A/i=500, A/o=100, D/i=10000, D/o=2200, O/i=10000, O/o=2200}
2026-10-06 01:12:06.412 <ad8525> out; determined limit (level=[A] flow=[o] key=[cnt/A/noah@wearemiraclesnow.com/d/2026-10-06/o]) => 100
2026-10-06 01:12:06.412 <ad8525> out; determined limit (level=[D] flow=[o] key=[cnt/D/wearemiraclesnow.com/d/2026-10-06/o]) => 2200
2026-10-06 01:12:06.412 <ad8525> out; determined limit (level=[O] flow=[o] key=[cnt/O/116681/d/2026-10-06/o]) => 2200
2026-10-06 01:12:06.412 <ad8525> out; recipient domain not in domains db: example.org (not counted)
2026-10-06 01:12:06.414 <ad8525> out; tracked address counters for noah@wearemiraclesnow.com => count=12 limit=100 reached=false
2026-10-06 01:12:06.414 <ad8525> out; tracked domain counters for wearemiraclesnow.com => count=1830 limit=2200 reached=false
2026-10-06 01:12:06.414 <ad8525> out; tracked account counters for 116681 => count=2200 limit=2200 reached=true
2026-10-06 01:12:06.414 <ad8525> out; limit reached (level=[O] id=[116681] flow=[o] by=[noah@wearemiraclesnow.com]) => notifying org_url
2026-10-06 01:12:06.903 <ad8525> out; sent limit exceeded notification to url "https://admin.migadu.com/api/limits?level=O&id=116681&flow=o&period=d&by=noah%40wearemiraclesnow.com&count=2200"
```
- A local recipient produces the same `local domain / limits_map / determined limit / tracked …` block for
`flow=[i]` under the recipient's domain row. `/in` requests log the recipient block only.
- `determined limit … => unlimited` when no limit is configured. Skips are logged as one line:
`skipped: exempt address <addr>`, `skipped: junk`, `recipient domain not in domains db: <d> (not counted)`,
`sender domain not in domains db: <d> (not counted)`.
- Notification outcome lines come from the background sender, tagged with the same `<trace_id>`:
`sent limit exceeded notification to url "…"` or
`failed limit exceeded notification to url "…" after 5 attempts: <err>`.
- Redis failure: `redis error: <err> (nothing counted)`.
- `format = "json"` switches the same lines to one JSON object per line (`ts, trace_id, route, event,
level, flow, key, id, count, limit, reached, domain, org_id, limits, url, error`) for later ingestion.
- The "tracked" lines are written after the pipeline EXEC (counts are the returned values); the earlier lines are
written before the Redis call, so a Redis failure still leaves the lookup trail in the log.
- Writer: the same `LogWriter` type as the app log (mutex, unbuffered for `tail -f`, `Reopen` on SIGHUP).
Config: `[logging] counting_output` (default: same destination as `output`; `"off"` disables),
`counting_format` (`text` | `json`). The per-request details live only here; the app log keeps startup,
reloads, warnings and errors, to avoid duplication.
## Process model: standalone daemon
`cmd/migadu_counters` is a long-running **foreground** process:
- Two `http.Server`s: the API (`/in`, `/out`, `/counters`, bearer auth, `http.TimeoutHandler`) on `addr`, and the
admin server (`/health`, `/metrics`, no auth) on `admin_addr`. Server timeouts as in rcptd:
`read_header_timeout=5`, `write_timeout=30`, `idle_timeout=60`, `handler_timeout=4` (seconds).
- Startup: load config → logger → open CDB (fatal if unreadable) → create Redis clients + PING the write side
(warn, not fatal: the service must come up while Redis is briefly away) → start fsnotify watcher → listen.
- Signals: `SIGHUP` → reopen both log files (app + counting) + reload CDB (`atomic.Pointer` swap, old file
closed after 30 s);
`SIGTERM`/`SIGINT` → stop accepting, drain in-flight requests (≤ 10 s), wait for background notifications
(≤ 10 s), close Redis clients, exit 0.
- Supervision on FreeBSD mirrors rcptd (`ansible-freebsd3/roles/rcptd/templates/rcptd.rc.j2`): an rc.d script runs
the binary under `/usr/sbin/daemon -f -H -p <pidfile> -o <logfile>`; daemon(8) owns the pidfile, restarts on
exit and captures stderr; `service migadu_counters reload` = `kill -HUP $(cat pidfile)` (the child's pid).
Therefore **no `daemon`/`pid_file` options** (rcptd's config template warns that writing the same pidfile from
the process breaks the supervisor). Logging defaults to stderr for daemon(8) `-o`; a file path is supported
with `Reopen` on SIGHUP for newsyslog.
- `[logging]` is rcptd's `LoggingConfig` verbatim (`output` = stderr|stdout|path, `format` = console|json,
`level`); copy `rcptd/internal/config/logger.go` (self-contained).
- Binaries carry `version/commit/date` via ldflags (`-version` flag); Makefile mirrors rcptd's: `build`, `test`,
`freebsd` (CGO_ENABLED=0, `migadu_counters-freebsd-amd64`, `counterctl-freebsd-amd64`), `build-linux`.
## Layout
```
migadu_counters/
├── go.mod # module migadu_counters (exists)
├── Makefile # build / test / freebsd / build-linux, ldflags version info
├── cmd/migadu_counters/main.go # config, logger, CDB load/reload, redis clients, HTTP + admin servers, signals
├── cmd/counterctl/main.go # CLI: flags, config discovery, GET /counters, table/json output
├── internal/api/types.go # JSON types shared by the server (/in, /out, /counters) and counterctl
├── internal/api/validate.go # per-route request validators, e-mail syntax, trace_id sanitizing
├── internal/config/config.go # TOML structs, defaults, validation (sentinel vs static, db==0, urls)
├── internal/config/logger.go # LoggingConfig + LogWriter.Reopen (from rcptd), counting_* fields
├── internal/txlog/txlog.go # counting transaction logger: text (Lua-like) / json, per-request block
├── internal/domains/domains.go # CDB open/lookup, ParseRecord/ParseLimits, Limit(level, flow, local)
├── internal/watcher/watcher.go # fsnotify reload of the CDB (rcptd table mode, trimmed)
├── internal/store/redis.go # write/read clients, Incr pipeline, Get for /counters
├── internal/counting/counting.go # /in, /out request → increments (level, id, flow, limit), exempt/junk/unknown
├── internal/notify/notify.go # per-level URL, templating/escaping, background sender with retries
├── internal/server/middleware.go # recover, method, body size, bearer token, allowed_cidrs, content-type, decode, timeout
├── internal/server/server.go # /in, /out, /counters handlers; admin /health, /metrics
├── internal/metrics/metrics.go # prometheus definitions
├── config.toml.example
└── README.md # API contract for mailqueuer, key format, counting rules, ops
```
Example config (`/usr/local/etc/migadu_counters/config.toml`; discovery order `-config`, `/etc/migadu_counters/config.toml`,
`/usr/local/etc/migadu_counters/config.toml`):
```toml
addr = ":5555" # was :8095 before review round 1
admin_addr = ":9095"
auth_token = "…" # or MIGADU_COUNTERS_AUTH_TOKEN; same value in mailqueuer's config
allowed_cidrs = [] # optional source-IP allowlist for /in, /out, /counters, e.g. ["10.0.0.0/8"]
domains_cdb = "/usr/local/etc/rspamd/db/rspamd_domains_db"
exempt = ["support@migadu.com", "contact@migadu.com"]
count_junk = false
read_header_timeout = 5
write_timeout = 30
idle_timeout = 60
handler_timeout = 4
[logging]
output = "stderr" # daemon(8) -o captures it; or a file path (reopened on SIGHUP)
format = "console" # or "json"
level = "info"
counting_output = "/var/log/migadu_counters/counting.log" # default: same as output; "off" disables
counting_format = "text" # Lua-like lines (default) or "json"
[redis]
db = 0 # must stay 0: migadu_web reads message counters from DB 0
password = "…" # or MIGADU_COUNTERS_REDIS_PASSWORD
pool_size = 32
timeout_ms = 500
# Sentinel mode (production): writes → master, reads → replicas
master_name = "mymaster"
sentinels = ["10.0.0.10:26379", "10.0.0.11:26379", "10.0.0.12:26379"]
sentinel_password = ""
# Static mode (used when sentinels is empty)
# write_addr = "127.0.0.1:6379"
# read_addrs = ["127.0.0.1:6379"]
[notify]
# One URL per level; leave empty to disable notifications for that level.
org_url = "https://admin.migadu.com/api/limits?level=$LEVEL&id=$ID&flow=$FLOW&period=$PERIOD&by=$ADDRESS&count=$COUNT"
address_url = "https://admin.migadu.com/api/limits?level=$LEVEL&id=$ID&flow=$FLOW&period=$PERIOD&by=$ADDRESS&count=$COUNT"
domain_url = ""
user = "…"
password = "…" # or MIGADU_COUNTERS_NOTIFY_PASSWORD
timeout_seconds = 5
```
Metrics: `migadu_counters_requests_total{route,status}`, `_request_duration_seconds{route}`,
`_increments_total{level,flow}`, `_limit_reached_total{level,flow}`, `_notifications_total{level,result}`,
`_redis_errors_total{op}`, `_cdb_domains` (gauge), `_cdb_reloads_total{result}`.
`/health`: 200 when the CDB is readable and the write client answers PING within 200 ms; else 503 with reason.
## Behaviour changes vs the Lua plugin (intentional)
- Separate `/in` and `/out` routes, one request per recipient, instead of rspamd's `MIGADU_FLOW` symbol.
- Outgoing counted **per recipient** (Lua: once per message, skipped when all recipients were same-domain).
Existing `*/o` limits therefore become stricter — a 50-recipient message costs 50; limits may need re-tuning
before the web app's `outgoing_limit_exceeded!` is relied upon.
- Local recipients of authenticated mail get incoming counted via `/out`.
- Domain (`D`) level counted; notified only when `domain_url` is set.
- Atomic increments; counters don't stop at the limit; no reject/soft-reject anywhere.
- Notification on the exact crossing increment; re-fires after a web-app reset.
- UTC day keys (matches the web app; Lua used server local time). Malformed CDB rows are skipped, not fatal.
## Deployment prerequisites (not part of this repo)
- An Ansible role in `ansible-freebsd3` cloned from `roles/rcptd`: user/group, `config.toml.j2`, the rc.d script
(daemon(8) with pidfile `/var/run/migadu_counters/migadu_counters.pid`, log `/var/log/migadu_counters/…`),
newsyslog entry, binary `migadu_counters-freebsd-amd64` + `counterctl-freebsd-amd64` in `/usr/local/bin`.
- The CDB must exist on the host running the service (the agent delivers it to rspamd hosts today;
stats-merger pulls the text variant) — same file, so a copy/pull job is enough.
- Firewall: mailqueuer hosts → service port; service → Sentinel hosts :26379/:6379 and admin.migadu.com:443.
- Secrets: bearer token shared with mailqueuer, Redis password, web app basic-auth credentials.
## Implementation order
0. Copy this plan into the repo as `plans/2026-10-13-migadu-counters.md` (requested; kept as the design record).
1. `go.mod` deps, `Makefile`, `config.toml.example`, `internal/config` (structs, defaults, validation, logger from rcptd).
2. `internal/domains` (+ tests with a temp CDB), `internal/watcher` (trimmed from rcptd).
3. `internal/store` (sentinel/static clients, Incr pipeline, MGET) + miniredis tests, incl. crossing detection.
4. `internal/counting` (request → increments, exempt/junk/unknown) + tests for the `/out` and `/in` examples.
5. `internal/txlog` (text + json, golden tests), `internal/notify` (per-level URLs, retries) + httptest tests.
6. `internal/api` types, `internal/server` (`/in`, `/out`, `/counters`, admin), `internal/metrics`.
7. `cmd/migadu_counters/main.go` (wiring, signals, shutdown), `cmd/counterctl/main.go` (+ tests).
8. `README.md` (API contract for mailqueuer, key format, counting rules, ops), manual verification below.
## Verification
1. `go vet ./... && go test -race ./...`
- `domains`: parse/override/unlimited/malformed rows via a temp CDB (`cdb.Create`); unknown domain.
- `counting`: the `/out` example (two calls → out +2 per level, y in +1 per level); external-only recipient;
sender domain not in CDB (only recipient counted); exempt sender/recipient → nothing; `/in` local/unknown/junk.
- `store` + miniredis: key format, TTL set, DB 0, crossing fires exactly once (limit 3: 3rd increment
`reached`, 4th not; after `DEL` the next crossing fires again); concurrent increments → one crossing;
read-side round-robin / fallback; Redis down → 503.
- `notify`: each level hits its own URL, empty URL skips, all vars escaped (`@` → `%40`), retries then failure metric.
- `txlog`: golden-file tests with an injected clock for the `/out` example above (local + external recipient,
one level reached), an `/in` request, each skip reason, a Redis failure, and the `json` format; `Reopen`
after the file is moved writes to the new path.
- `api/validate`: table tests per field and route (missing, empty, too long, two `@`, whitespace, control
chars, uppercase → lowercased, punycode accepted, `mail_from` empty on `/in`, `trace_id` sanitized).
- `server/middleware`: one test per step in chain order — 405, 413 (65 KB body), 401 (missing/wrong/`Basic`),
403 (`allowed_cidrs` miss) vs pass, 415, 400 (syntax error, two objects, unknown field *accepted*), 503
(handler timeout with a blocked Redis), 500 (panic recovered, daemon still serving); each rejection leaves
Redis untouched (miniredis key count unchanged) and increments `requests_total{status}`.
- `server`: 401, 400, 200 with `counted` payload; `/counters` with address/domain, default date = today UTC,
explicit `date`, future date → 400, malformed date → 400, unmanaged domain → `managed:false` + null limits;
notify targets are per-level `httptest.Server`s.
- `counterctl`: date parsing/validation (valid ISO, malformed, future, > 31 days warning), exactly-one-of
`--address`/`--domain`, `addr` → URL normalization, table rendering against a fixed API response
(`httptest.Server`), `--json` passthrough, exit codes.
2. Manual: local Redis in static mode, CDB built from a sample line (`cdb -m -c`, or the test helper),
`go run ./cmd/migadu_counters -config config.toml.example` with `org_url`/`address_url` pointed at a local
listener. `curl -XPOST localhost:8095/out -H 'Authorization: Bearer …' -d '{…}'` repeatedly; check
`redis-cli -n 0 KEYS 'cnt/*'`, `GET /counters?address=…`, exactly one notification per level on crossing,
`redis-cli DEL` the A key → crossing notifies again; edit the CDB + `kill -HUP` → limits reload.
`go run ./cmd/counterctl --config config.toml.example --address x@example.com` shows the same numbers as
`redis-cli`; `--date <yesterday>` shows zeros/older data; `--date 2099-01-01` is rejected.
`tail -f counting.log` while sending the curl requests shows the full block per request, and
`grep '<trace_id>' counting.log` returns one transaction. Daemon behaviour: `kill -TERM` exits 0 after
draining; with `logging.output`/`counting_output` set to files, `kill -HUP` reopens them (rotate with `mv`
first and confirm new lines land in the new files); `make freebsd` produces the two static binaries.
Optionally a local Sentinel (1 master + 1 replica) to confirm reads hit the replica.