dimitrije/counters
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
counters/plans/2026-10-13-migadu-counters.md
T

35 KiB
Raw Permalink Blame History

Plan: migadu_counters — real-time mail counting service (Go)

Context

Daily send/receive limits used to be counted and enforced by the rspamd Lua plugin ansible-freebsd2/roles/rspamd-postfix/templates/rspamd/plugins.d/migadu_counters.lua (Redis counters per address + org, soft-reject/reject on breach, web app notified once). Mail now flows mizu → mailqueuer and bypasses rspamd, so nothing counts in real time. stats-merger/internal/quota recomputes usage from the delivery ledger after each merge, but it is reporting-only today: only fernrohr reads /api/quota/{status,stats,overquota}; nothing calls /check.

migadu_counters is a new standalone Go service that becomes the single counting source and the enforcement trigger:

  • mailqueuer calls it fire-and-forget, once per ingested recipient, from its publish loop;
  • one route per flow: POST /in, POST /out (/rel later, not planned here);
  • it increments Redis counters at Address / Domain / Organization level, in the key format the web app already reads. The domain level is turned back on (Lua defined it but never tracked it);
  • when an increment crosses a limit it notifies the web app (migadu_web GET /api/limits), which decides what to do (notify the user, remove sending/receiving);
  • a small CLI, counterctl, shows an address's incoming/outgoing counters for today or a given date (see "CLI: counterctl" below).

The service returns no verdicts and enforces nothing itself. Out of scope: /rel, the mailqueuer client change, retiring stats-merger quota, the web app's D branch, deployment (see prerequisites at the end).

How it fits together

 sender ──SMTP──▶ mizu ──POST /ingest──▶ mailqueuer
                   X-Trace-Id, X-Mail-From,   │ one queue message per recipient;
                   X-Mail-To, X-Auth-User,    │ after each successful publish:
                   X-Junk                     │ fire-and-forget HTTP call, reply ignored
                                              ▼
            POST /out  {"trace_id","auth_user","mail_from","recipient"}   ← X-Auth-User set
            POST /in   {"trace_id","mail_from","recipient","junk"}        ← X-Auth-User empty
            Authorization: Bearer <token>   (mailqueuer sends it, migadu_counters checks it; 401 if wrong)
                                              │
                                              ▼
                               ┌─────────────────────────────────────┐
                               │ migadu_counters  (daemon :8095)     │
                               │ ┌─────────────────────────────────┐ │   rejected here, nothing counted:
                               │ │ HTTP validation layer           │ │   405 method · 413 body > 64 KB
                               │ │ method → size → bearer token →  │ │   401 bad/missing token · 403 source IP
                               │ │ source IP → content-type → JSON │ │   415 content-type · 400 json / fields /
                               │ │ → fields → e-mail syntax        │ │       e-mail syntax · 503 handler timeout
                               │ └───────────────┬─────────────────┘ │
                               │                 ▼ validated request │
         rspamd_domains_db ───▶│ counting:  1. domain in CDB? → org_id, limits                 │
         (CDB: domain →        │            2. MULTI: INCR A/D/O keys, EXPIRE 31 d             │
          org_id | limits)     │            3. did this INCR cross a limit? → notify           │
                               └───┬─────────────────────────┬───────┘   admin :9095 /health /metrics
                                   │                         │
                 Redis Sentinel ◀──┘                         └──────▶ GET https://admin.migadu.com/api/limits
                 writes → master                             ?level=&id=&flow=&period=d&by=&count=
                 reads  → replicas                           (basic auth)  → migadu_web notifies the user
                 DB 0: cnt/<A|D|O>/<id>/d/<YYYY-MM-DD>/<i|o>                 or disables sending/receiving

 counterctl --address x@y [--date YYYY-MM-DD] ──GET /counters (Bearer)──▶ migadu_counters ──▶ Redis replicas

Existing pieces reused or matched

  • Limits: the CDB the Lua plugin reads, /usr/local/etc/rspamd/db/rspamd_domains_db, built by ansible-freebsd2/roles/agent/templates/extract.sh.j2:478 with tinycdb cdb -m (no NUL terminators, exact-byte lookups like rcptd). Key = domain, value = <org_id>|O/i=..,O/o=..,D/i=..,A/o=..,alice@/o=... Parsing and limit resolution follow stats-merger/internal/quota/quota.go (ParseLimits L66, DomainLimits.limit L116): keys lowercased, malformed pairs skipped, <local>@/<flow> overrides A/<flow>, missing key = unlimited. A domain present in the CDB is "Migadu-hosted".
  • Redis keys — must match the web app's key builder (migadu_web/apps/core/lib/core/redis_sentinel.ex:73-103): cnt/<A|D|O>/<id>/d/<YYYY-MM-DD>/<i|o> with id = address / domain / org_id, date in UTC (web app uses Date.utc_today()), Redis DB 0 (web app reads :redis_sentinel_db0; rspamd uses redis_rspamd_db: 0). TTL 31 days. DB 1 and the /s storage keys belong to the web app; never touched. The web app may DEL today's keys to reset a customer (redis_sentinel.ex:145-153); counting must keep working after that, and a second crossing must notify again.
  • Web app notification: GET https://admin.migadu.com/api/limits?level=&id=&flow=&period=&by=&count= with basic auth (apps/admin_web/lib/admin_web/controllers/internal_api/notification_controller.ex:33, router.ex:99 Plug.BasicAuth). All six params are pattern-matched, so all must be present. id = address for A, domain name for D, org id for O. Credentials: vault_migadu_web.internal_api_basic_auth_* (see rspamd-postfix/templates/rspamd/local.d/migadu_counters.conf.j2).
  • Redis topology: ansible-freebsd2/roles/redis-sentinel, inventory/hosts.ini:160: hosts redisSentinel0-2, Redis :6379 + Sentinel :26379, master name mymaster, 1 master + 2 replicas, one password (vault_redis_sentinel_password) for Redis; Sentinel itself has no requirepass.
  • Service conventions from ~/Work/rcptd (cmd/rcptd/main.go): TOML config via -config + default paths, bearer auth_token with env override, slog, CDB behind atomic.Pointer with a 30 s grace close, SIGHUP + fsnotify reload (internal/watcher/watcher.go table mode), admin port with /health + /metrics, graceful shutdown. No dependency on rcptd's shared module (not present locally).
  • Libraries (in the module cache): github.com/colinmarc/cdb, github.com/BurntSushi/toml v1.6.0, github.com/fsnotify/fsnotify v1.9.0, github.com/redis/go-redis/v9 v9.7.3, github.com/prometheus/client_golang; tests: github.com/alicebob/miniredis/v2 v2.34.0.

API (called by mailqueuer, fire-and-forget, one request per recipient)

Revised 2026-10-14 (review round 1, review/README.md #1–#6): /in and /out now validate, enqueue and answer 202 {"accepted":true}; counting runs in a worker pool (queue_size, workers) under its own time budget, and the counted values appear only in the counting log and GET /counters. A full queue answers 503 {"error":"queue full"}. Only the counted sides are validated (mail_from never rejects; auth_user and recipient independently), IDN domains are converted to punycode, user@dom@SUFFIX logins count under the base address. INCR replaces INCRBY; per-key EXEC failures and lost replies are handled individually (re-read from the master). Parsed CDB records are cached (max_cached_domains). auth_token is mandatory, unknown config keys are fatal, the default API port is :5555. The 200 {"counted":…} contract below is the original design and is superseded by the README.

mailqueuer's ingest handler is already per recipient: it loops over toPublish (mailqueuer/internal/handler/ingest.go:235-306), publishes one queue message per recipient (own uuid id, L237) and marks dedup only after a successful publish (L263-272). Both routes are called from inside that loop, right after a successful Publish, with exactly one recipient. The route is chosen by X-Auth-User: non-empty → /out, empty → /in. mailqueuer never waits for or acts on the response.

Authentication: every request carries Authorization: Bearer <token> — mailqueuer sends it (the same header it already sends to rcptd, internal/routing/routing.go:256-258), migadu_counters validates it (constant-time compare against auth_token; missing/wrong → 401, nothing counted). One shared token, set in both configs; no other auth. All routes take a JSON body ≤ 64 KB. Processing is synchronous: one Redis round trip, then the response. Only web-app notifications run in the background.

  • 200 {"counted":[{"level":"A","id":"x@example.com","flow":"o","count":12,"limit":100,"reached":false},…]} (reached: true on the increment that crossed the limit → notification triggered). Skipped requests return {"counted":[],"skipped":"exempt|unknown_domain|junk"}.
  • 400 invalid body/address, 401 bad token, 503 Redis error (nothing counted).

POST /out — authenticated submission

{ "trace_id": "…", "auth_user": "x@example.com", "mail_from": "x@example.com", "recipient": "y@example.com" }
  1. Sender outgoing +1 on A(auth_user) / D(its domain) / O(its org) — only if auth_user's domain is in the CDB.
  2. Recipient incoming +1 on A / D / O of the recipient's domain row — only if that domain is in the CDB (any Migadu-hosted domain, any org). That mail never passes /in.

Example: x@example.com → y@example.com, s@gmail.com = two /out calls. Call 1: out A x +1, D example.com +1, O +1; in A y +1, D example.com +1, O +1. Call 2: out +1 on each; gmail not in CDB → no incoming. Totals: outgoing +2 per level, incoming for y +1 per level.

POST /in — inbound mail from outside

{ "trace_id": "…", "mail_from": "s@gmail.com", "recipient": "y@example.com", "junk": false }

Recipient incoming +1 on A / D / O if its domain is in the CDB. The sender is never counted. junk = true is skipped unless count_junk = true (Lua only counted no action mail).

GET /counters?address=… or ?domain=… [&date=YYYY-MM-DD]

Read-only view (used by counterctl), served from the read client. date is an ISO 8601 calendar date (YYYY-MM-DD, interpreted as a UTC day); default = today UTC; a future date is 400. Keys live 31 days, so older dates return zeros. Exactly one of address/domain is required.

{ "query": "me@example.com", "mode": "address", "date": "2026-10-13",
  "managed": true, "exempt": false, "domain": "example.com", "org_id": 1234,
  "levels": [
    {"level": "A", "id": "me@example.com", "in": 12,  "in_limit": 500,  "out": 3,  "out_limit": 200},
    {"level": "D", "id": "example.com",    "in": 40,  "in_limit": 1000, "out": 9,  "out_limit": 400},
    {"level": "O", "id": "1234",           "in": 120, "in_limit": 5000, "out": 30, "out_limit": 1000} ] }

in_limit/out_limit are null when unlimited. mode: "domain" returns the D and O rows only. managed: false (domain not in the CDB) still returns the raw counters with null limits — Lua-era keys may exist. Values are read with one MGET of the ≤ 6 keys.

Shared rules

  • Addresses are trimmed and lowercased; validation = one @, non-empty parts, ≤ 254 chars.
  • Exempt list (default support@migadu.com, contact@migadu.com): if auth_user, mail_from or recipient is exempt, nothing is counted for that request (Lua: mail to/from support never costs quota).
  • trace_id (mizu's X-Trace-Id, shared by all recipients of one message) is logged for correlation only; mailqueuer's internal per-recipient queue id is not sent. No Redis idempotency keys: mailqueuer already guarantees at-most-once per (trace, recipient) publish; the residual case (mailqueuer restart + mizu retry) also double-delivers the mail, so the count stays truthful.
  • O and D limits come from the domain row that triggered the increment (sender's row for outgoing, recipient's row for incoming) — same caveat as Lua if domains of one org carry different O/* values.

HTTP validation layer (mailqueuer → migadu_counters)

Everything mailqueuer sends passes this layer before any CDB or Redis access; a request that fails any step is answered with a JSON error {"error":"…"} and nothing is counted. Two parts:

1. Middleware chain (internal/server/middleware.go, applied in this order to /in, /out; /counters gets the same chain with method GET and no body steps):

# Step Rule Failure
1 recover panic in a handler → log with stack, keep the daemon alive 500
2 method POST only (GET for /counters) 405 + Allow header
3 body size http.MaxBytesReader 64 KB 413
4 bearer token Authorization: Bearer <token>, constant-time compare with auth_token 401 + WWW-Authenticate: Bearer
5 source IP optional allowed_cidrs (config); matched on RemoteAddr only — X-Forwarded-For is never trusted (no proxy in front) 403
6 content type Content-Type: application/json (parameters such as charset ignored) 415
7 JSON decode exactly one JSON object; syntax errors rejected; unknown fields ignored so mailqueuer can add fields without breaking counting 400
8 request validation per-route field rules below 400
9 timeout http.TimeoutHandler (handler_timeout, default 4 s) around the counting work 503

Every rejection is logged at warn with route, remote address, status, reason and trace_id (if parsable) and counted in requests_total{route,status} — a burst of 400/401 from mailqueuer means contract drift or a token mismatch and should alert.

2. Request validators (internal/api/validate.go, pure functions, table-tested):

Field /out /in Rule
trace_id optional optional ≤ 128 chars, printable ASCII; control characters/newlines stripped before it reaches any log line (log-injection guard)
auth_user required — (ignored) e-mail syntax
recipient required required e-mail syntax
mail_from optional optional ("" = bounce sender <>) e-mail syntax when non-empty
junk — (ignored) optional, default false boolean

E-mail syntax = trimmed, lowercased, ≤ 254 chars, exactly one @, non-empty local part and domain, no whitespace/control characters; the domain is ASCII (punycode accepted). Deliberately permissive: the CDB lookup decides whether an address is counted, the validator only guards against garbage and log injection. Errors name the field: {"error":"recipient: missing"}, {"error":"auth_user: invalid e-mail address"}.

Config: allowed_cidrs = ["10.0.0.0/8"] (optional; empty = any source), handler_timeout (see daemon section). The response codes mailqueuer may see are therefore: 200 counted/skipped, 4xx rejected by this layer, 503 Redis unavailable or timeout — it ignores all of them (fire-and-forget), they exist for operators and tests.

Redis (internal/store)

Config has a write side and a read side:

  • Sentinel mode (production): sentinels, master_name, password (+ optional sentinel_password). Write = redis.NewFailoverClient (follows failover). Read = NewFailoverClient{ReplicaOnly: true}; go-redis picks a random replica and falls back to the master when none is known (sentinel.go:526).
  • Static mode (dev/tests, when sentinels is empty): write_addr + read_addrs (round-robin; defaults to write_addr).
  • Both pooled (pool_size); one timeout_ms for dial/read/write. db must be 0 (validated, startup error otherwise).

Counting = one TxPipeline (MULTI/EXEC) per request: INCRBY key 1 + EXPIRE key 2678400 for every affected key (≤ 6 keys). EXEC returns the new values, so no read-after-write and replica lag is irrelevant. Increments are atomic (fixes the Lua GET-then-INCR race); counting continues past the limit.

Limit crossing: for each incremented key with a configured limit L, notify iff new − amount < L ≤ new (with amount 1: new == L). Exactly one increment can cross, across all workers/instances, with no marker key. After a web-app reset (DEL) the counter crosses again and notifies again. Accepted gap (same as Lua): a limit lowered below the current count mid-day produces no crossing.

Redis errors → 503, log, redis_errors_total{op}.

Notification (internal/notify)

  • One URL per level in [notify]: org_url (O), address_url (A), domain_url (D). Empty URL = that level is counted and limit_reached_total is incremented, but no HTTP call (useful while the web app's D branch is a TODO). Basic auth and timeout are shared.
  • GET <url> with template vars $LEVEL $ID $FLOW $PERIOD(=d) $ADDRESS $COUNT $LIMIT, each url.QueryEscaped. ADDRESS = the address that triggered it (sender for outgoing, recipient for incoming). COUNT = new value.
  • Runs in a background goroutine (semaphore of 8 to not flood the web app): up to 5 attempts, exponential backoff 1 s → 16 s. Final failure → error log + notifications_total{level,result="failed"} (alertable). Shutdown waits ≤ 10 s for in-flight notifications.

CLI: counterctl

counterctl --address me@dimitrijedimitrijevic.com            # today (UTC)
counterctl --address me@dimitrijedimitrijevic.com --date 2026-10-12
counterctl --domain dimitrijedimitrijevic.com --json
  • Flags (Go flag, so -x and --x both work): --address | --domain (exactly one), --date YYYY-MM-DD (ISO 8601 calendar date; default today UTC; parsed with time.Parse("2006-01-02"); future date = error; older than 31 days = warning on stderr that keys have expired), --json (raw API response), --config PATH, --url (override), --version.
  • Talks to the running service's GET /counters instead of Redis directly — one code path for key building and limit resolution, no Redis/CDB access needed from the operator's shell, same approach as rcptctl. Config discovery mirrors the service and rcptctl (cmd/rcptctl/main.go:65-75): --config, else /etc/migadu_counters/config.toml, /usr/local/etc/migadu_counters/config.toml; reads addr and auth_token only (MIGADU_COUNTERS_AUTH_TOKEN overrides); addr like :8095 becomes http://localhost:8095 (rcptctl/api.go:52-60); warns when the host is not local (token over plaintext).
  • Output (text/tabwriter), limits shown as - when unlimited, ! appended when value ≥ limit: 
    me@dimitrijedimitrijevic.com   date 2026-10-13 (UTC)   domain dimitrijedimitrijevic.com   org 1234
LEVEL  ID                             IN    LIMIT  OUT   LIMIT
A      me@dimitrijedimitrijevic.com   12    500    3     200
D      dimitrijedimitrijevic.com      40    1000   9     400
O      1234                           120   5000   30    1000
    managed: false prints a domain is not in rspamd_domains_db (not counted) note after the table; exempt: true prints address is exempt from counting.
  • Exit codes: 0 success, 1 usage error / bad date / service unreachable / non-200 (message on stderr).

Counting transaction log (internal/txlog)

A second logger, separate from the slog application log, that writes the Lua plugin's step-by-step lines for every counting transaction (the Lua infox lines from migadu_counters.lua:217/235/266/268/271/283), so existing grep habits keep working. rspamd's task id <ad8525> is replaced by <trace_id> (shared with mizu/mailqueuer logs via X-Trace-Id), and the route is added. breach= becomes reached= because the meaning changed (Lua: already over → rejected; here: this increment crossed the limit).

Text format (default), timestamps UTC with milliseconds, one block per request:

2026-10-06 01:12:06.412 <ad8525> out; request auth_user=noah@wearemiraclesnow.com recipient=y@example.org mail_from=noah@wearemiraclesnow.com
2026-10-06 01:12:06.412 <ad8525> out; local domain: wearemiraclesnow.com account_id=116681
2026-10-06 01:12:06.412 <ad8525> out; domain: wearemiraclesnow.com limits_map={A/i=500, A/o=100, D/i=10000, D/o=2200, O/i=10000, O/o=2200}
2026-10-06 01:12:06.412 <ad8525> out; determined limit (level=[A] flow=[o] key=[cnt/A/noah@wearemiraclesnow.com/d/2026-10-06/o]) => 100
2026-10-06 01:12:06.412 <ad8525> out; determined limit (level=[D] flow=[o] key=[cnt/D/wearemiraclesnow.com/d/2026-10-06/o]) => 2200
2026-10-06 01:12:06.412 <ad8525> out; determined limit (level=[O] flow=[o] key=[cnt/O/116681/d/2026-10-06/o]) => 2200
2026-10-06 01:12:06.412 <ad8525> out; recipient domain not in domains db: example.org (not counted)
2026-10-06 01:12:06.414 <ad8525> out; tracked address counters for noah@wearemiraclesnow.com => count=12 limit=100 reached=false
2026-10-06 01:12:06.414 <ad8525> out; tracked domain counters for wearemiraclesnow.com => count=1830 limit=2200 reached=false
2026-10-06 01:12:06.414 <ad8525> out; tracked account counters for 116681 => count=2200 limit=2200 reached=true
2026-10-06 01:12:06.414 <ad8525> out; limit reached (level=[O] id=[116681] flow=[o] by=[noah@wearemiraclesnow.com]) => notifying org_url
2026-10-06 01:12:06.903 <ad8525> out; sent limit exceeded notification to url "https://admin.migadu.com/api/limits?level=O&id=116681&flow=o&period=d&by=noah%40wearemiraclesnow.com&count=2200"
  • A local recipient produces the same local domain / limits_map / determined limit / tracked … block for flow=[i] under the recipient's domain row. /in requests log the recipient block only.
  • determined limit … => unlimited when no limit is configured. Skips are logged as one line: skipped: exempt address <addr>, skipped: junk, recipient domain not in domains db: <d> (not counted), sender domain not in domains db: <d> (not counted).
  • Notification outcome lines come from the background sender, tagged with the same <trace_id>: sent limit exceeded notification to url "…" or failed limit exceeded notification to url "…" after 5 attempts: <err>.
  • Redis failure: redis error: <err> (nothing counted).
  • format = "json" switches the same lines to one JSON object per line (ts, trace_id, route, event, level, flow, key, id, count, limit, reached, domain, org_id, limits, url, error) for later ingestion.
  • The "tracked" lines are written after the pipeline EXEC (counts are the returned values); the earlier lines are written before the Redis call, so a Redis failure still leaves the lookup trail in the log.
  • Writer: the same LogWriter type as the app log (mutex, unbuffered for tail -f, Reopen on SIGHUP). Config: [logging] counting_output (default: same destination as output; "off" disables), counting_format (text | json). The per-request details live only here; the app log keeps startup, reloads, warnings and errors, to avoid duplication.

Process model: standalone daemon

cmd/migadu_counters is a long-running foreground process:

  • Two http.Servers: the API (/in, /out, /counters, bearer auth, http.TimeoutHandler) on addr, and the admin server (/health, /metrics, no auth) on admin_addr. Server timeouts as in rcptd: read_header_timeout=5, write_timeout=30, idle_timeout=60, handler_timeout=4 (seconds).
  • Startup: load config → logger → open CDB (fatal if unreadable) → create Redis clients + PING the write side (warn, not fatal: the service must come up while Redis is briefly away) → start fsnotify watcher → listen.
  • Signals: SIGHUP → reopen both log files (app + counting) + reload CDB (atomic.Pointer swap, old file closed after 30 s); SIGTERM/SIGINT → stop accepting, drain in-flight requests (≤ 10 s), wait for background notifications (≤ 10 s), close Redis clients, exit 0.
  • Supervision on FreeBSD mirrors rcptd (ansible-freebsd3/roles/rcptd/templates/rcptd.rc.j2): an rc.d script runs the binary under /usr/sbin/daemon -f -H -p <pidfile> -o <logfile>; daemon(8) owns the pidfile, restarts on exit and captures stderr; service migadu_counters reload = kill -HUP $(cat pidfile) (the child's pid). Therefore no daemon/pid_file options (rcptd's config template warns that writing the same pidfile from the process breaks the supervisor). Logging defaults to stderr for daemon(8) -o; a file path is supported with Reopen on SIGHUP for newsyslog.
  • [logging] is rcptd's LoggingConfig verbatim (output = stderr|stdout|path, format = console|json, level); copy rcptd/internal/config/logger.go (self-contained).
  • Binaries carry version/commit/date via ldflags (-version flag); Makefile mirrors rcptd's: build, test, freebsd (CGO_ENABLED=0, migadu_counters-freebsd-amd64, counterctl-freebsd-amd64), build-linux.

Layout

migadu_counters/
├── go.mod                         # module migadu_counters (exists)
├── Makefile                       # build / test / freebsd / build-linux, ldflags version info
├── cmd/migadu_counters/main.go    # config, logger, CDB load/reload, redis clients, HTTP + admin servers, signals
├── cmd/counterctl/main.go         # CLI: flags, config discovery, GET /counters, table/json output
├── internal/api/types.go          # JSON types shared by the server (/in, /out, /counters) and counterctl
├── internal/api/validate.go       # per-route request validators, e-mail syntax, trace_id sanitizing
├── internal/config/config.go      # TOML structs, defaults, validation (sentinel vs static, db==0, urls)
├── internal/config/logger.go      # LoggingConfig + LogWriter.Reopen (from rcptd), counting_* fields
├── internal/txlog/txlog.go        # counting transaction logger: text (Lua-like) / json, per-request block
├── internal/domains/domains.go    # CDB open/lookup, ParseRecord/ParseLimits, Limit(level, flow, local)
├── internal/watcher/watcher.go    # fsnotify reload of the CDB (rcptd table mode, trimmed)
├── internal/store/redis.go        # write/read clients, Incr pipeline, Get for /counters
├── internal/counting/counting.go  # /in, /out request → increments (level, id, flow, limit), exempt/junk/unknown
├── internal/notify/notify.go      # per-level URL, templating/escaping, background sender with retries
├── internal/server/middleware.go  # recover, method, body size, bearer token, allowed_cidrs, content-type, decode, timeout
├── internal/server/server.go      # /in, /out, /counters handlers; admin /health, /metrics
├── internal/metrics/metrics.go    # prometheus definitions
├── config.toml.example
└── README.md                      # API contract for mailqueuer, key format, counting rules, ops

Example config (/usr/local/etc/migadu_counters/config.toml; discovery order -config, /etc/migadu_counters/config.toml, /usr/local/etc/migadu_counters/config.toml):

addr = ":5555"                     # was :8095 before review round 1
admin_addr = ":9095"
auth_token = "…"                   # or MIGADU_COUNTERS_AUTH_TOKEN; same value in mailqueuer's config
allowed_cidrs = []                 # optional source-IP allowlist for /in, /out, /counters, e.g. ["10.0.0.0/8"]
domains_cdb = "/usr/local/etc/rspamd/db/rspamd_domains_db"
exempt = ["support@migadu.com", "contact@migadu.com"]
count_junk = false

read_header_timeout = 5
write_timeout = 30
idle_timeout = 60
handler_timeout = 4

[logging]
output = "stderr"                  # daemon(8) -o captures it; or a file path (reopened on SIGHUP)
format = "console"                 # or "json"
level = "info"
counting_output = "/var/log/migadu_counters/counting.log"   # default: same as output; "off" disables
counting_format = "text"           # Lua-like lines (default) or "json"

[redis]
db = 0                             # must stay 0: migadu_web reads message counters from DB 0
password = "…"                     # or MIGADU_COUNTERS_REDIS_PASSWORD
pool_size = 32
timeout_ms = 500
# Sentinel mode (production): writes → master, reads → replicas
master_name = "mymaster"
sentinels = ["10.0.0.10:26379", "10.0.0.11:26379", "10.0.0.12:26379"]
sentinel_password = ""
# Static mode (used when sentinels is empty)
# write_addr = "127.0.0.1:6379"
# read_addrs = ["127.0.0.1:6379"]

[notify]
# One URL per level; leave empty to disable notifications for that level.
org_url     = "https://admin.migadu.com/api/limits?level=$LEVEL&id=$ID&flow=$FLOW&period=$PERIOD&by=$ADDRESS&count=$COUNT"
address_url = "https://admin.migadu.com/api/limits?level=$LEVEL&id=$ID&flow=$FLOW&period=$PERIOD&by=$ADDRESS&count=$COUNT"
domain_url  = ""
user = "…"
password = "…"                     # or MIGADU_COUNTERS_NOTIFY_PASSWORD
timeout_seconds = 5

Metrics: migadu_counters_requests_total{route,status}, _request_duration_seconds{route}, _increments_total{level,flow}, _limit_reached_total{level,flow}, _notifications_total{level,result}, _redis_errors_total{op}, _cdb_domains (gauge), _cdb_reloads_total{result}. /health: 200 when the CDB is readable and the write client answers PING within 200 ms; else 503 with reason.

Behaviour changes vs the Lua plugin (intentional)

  • Separate /in and /out routes, one request per recipient, instead of rspamd's MIGADU_FLOW symbol.
  • Outgoing counted per recipient (Lua: once per message, skipped when all recipients were same-domain). Existing */o limits therefore become stricter — a 50-recipient message costs 50; limits may need re-tuning before the web app's outgoing_limit_exceeded! is relied upon.
  • Local recipients of authenticated mail get incoming counted via /out.
  • Domain (D) level counted; notified only when domain_url is set.
  • Atomic increments; counters don't stop at the limit; no reject/soft-reject anywhere.
  • Notification on the exact crossing increment; re-fires after a web-app reset.
  • UTC day keys (matches the web app; Lua used server local time). Malformed CDB rows are skipped, not fatal.

Deployment prerequisites (not part of this repo)

  • An Ansible role in ansible-freebsd3 cloned from roles/rcptd: user/group, config.toml.j2, the rc.d script (daemon(8) with pidfile /var/run/migadu_counters/migadu_counters.pid, log /var/log/migadu_counters/…), newsyslog entry, binary migadu_counters-freebsd-amd64 + counterctl-freebsd-amd64 in /usr/local/bin.
  • The CDB must exist on the host running the service (the agent delivers it to rspamd hosts today; stats-merger pulls the text variant) — same file, so a copy/pull job is enough.
  • Firewall: mailqueuer hosts → service port; service → Sentinel hosts :26379/:6379 and admin.migadu.com:443.
  • Secrets: bearer token shared with mailqueuer, Redis password, web app basic-auth credentials.

Implementation order

  1. Copy this plan into the repo as plans/2026-10-13-migadu-counters.md (requested; kept as the design record).
  2. go.mod deps, Makefile, config.toml.example, internal/config (structs, defaults, validation, logger from rcptd).
  3. internal/domains (+ tests with a temp CDB), internal/watcher (trimmed from rcptd).
  4. internal/store (sentinel/static clients, Incr pipeline, MGET) + miniredis tests, incl. crossing detection.
  5. internal/counting (request → increments, exempt/junk/unknown) + tests for the /out and /in examples.
  6. internal/txlog (text + json, golden tests), internal/notify (per-level URLs, retries) + httptest tests.
  7. internal/api types, internal/server (/in, /out, /counters, admin), internal/metrics.
  8. cmd/migadu_counters/main.go (wiring, signals, shutdown), cmd/counterctl/main.go (+ tests).
  9. README.md (API contract for mailqueuer, key format, counting rules, ops), manual verification below.

Verification

  1. go vet ./... && go test -race ./...
    • domains: parse/override/unlimited/malformed rows via a temp CDB (cdb.Create); unknown domain.
    • counting: the /out example (two calls → out +2 per level, y in +1 per level); external-only recipient; sender domain not in CDB (only recipient counted); exempt sender/recipient → nothing; /in local/unknown/junk.
    • store + miniredis: key format, TTL set, DB 0, crossing fires exactly once (limit 3: 3rd increment reached, 4th not; after DEL the next crossing fires again); concurrent increments → one crossing; read-side round-robin / fallback; Redis down → 503.
    • notify: each level hits its own URL, empty URL skips, all vars escaped (@ → %40), retries then failure metric.
    • txlog: golden-file tests with an injected clock for the /out example above (local + external recipient, one level reached), an /in request, each skip reason, a Redis failure, and the json format; Reopen after the file is moved writes to the new path.
    • api/validate: table tests per field and route (missing, empty, too long, two @, whitespace, control chars, uppercase → lowercased, punycode accepted, mail_from empty on /in, trace_id sanitized).
    • server/middleware: one test per step in chain order — 405, 413 (65 KB body), 401 (missing/wrong/Basic), 403 (allowed_cidrs miss) vs pass, 415, 400 (syntax error, two objects, unknown field accepted), 503 (handler timeout with a blocked Redis), 500 (panic recovered, daemon still serving); each rejection leaves Redis untouched (miniredis key count unchanged) and increments requests_total{status}.
    • server: 401, 400, 200 with counted payload; /counters with address/domain, default date = today UTC, explicit date, future date → 400, malformed date → 400, unmanaged domain → managed:false + null limits; notify targets are per-level httptest.Servers.
    • counterctl: date parsing/validation (valid ISO, malformed, future, > 31 days warning), exactly-one-of --address/--domain, addr → URL normalization, table rendering against a fixed API response (httptest.Server), --json passthrough, exit codes.
  2. Manual: local Redis in static mode, CDB built from a sample line (cdb -m -c, or the test helper), go run ./cmd/migadu_counters -config config.toml.example with org_url/address_url pointed at a local listener. curl -XPOST localhost:8095/out -H 'Authorization: Bearer …' -d '{…}' repeatedly; check redis-cli -n 0 KEYS 'cnt/*', GET /counters?address=…, exactly one notification per level on crossing, redis-cli DEL the A key → crossing notifies again; edit the CDB + kill -HUP → limits reload. go run ./cmd/counterctl --config config.toml.example --address x@example.com shows the same numbers as redis-cli; --date <yesterday> shows zeros/older data; --date 2099-01-01 is rejected. tail -f counting.log while sending the curl requests shows the full block per request, and grep '<trace_id>' counting.log returns one transaction. Daemon behaviour: kill -TERM exits 0 after draining; with logging.output/counting_output set to files, kill -HUP reopens them (rotate with mv first and confirm new lines land in the new files); make freebsd produces the two static binaries. Optionally a local Sentinel (1 master + 1 replica) to confirm reads hit the replica.
Reference in New Issue View Git Blame Copy Permalink