counters/todo.md

# TODO — migadu_counters

State as of 2026-10-15. Review findings refer to `review/README.md`; the proof tests live in
`review/_proof_tests/` and should become regression tests as each item is fixed (round 1 and items
7–8 are mapped at the end of `review/_proof_tests/README.md`).

## Done

- Review round 1: #1–#6 (queue + 202, lenient validation + IDN, token logins, INCR with per-key
  results and master re-read, CDB parse cache, mandatory token / port 5555), plus
  `contrib/pull-domains.sh` and `counterctl --in/--out`.
- #7 plus-addressing counts against the mailbox (`domains.Mailbox`).
- #8 malformed CDB row is "not hosted" + warning (`domains.ErrMalformed`).

## Review round 2 — remaining findings

Suggested order: #10, #15, #9, then the rest in one sweep.

### Can stop the daemon or lose counts

- [ ] **#10 SIGHUP during startup kills the daemon.** The signal handler is installed after the Redis
      ping (up to 2 s). Install `signal.Notify` before anything slow in `cmd/migadu_counters/main.go`
      (`run`), and queue a reload that arrives before the app is ready. Evidence: `review/_proof_tests/a/run/`.
- [ ] **#15 Sentinel mode starts without a Redis password.** Every write fails; only `/health` and
      metrics show it. Make an empty `redis.password` fatal in `config.Validate` when `sentinels` is set
      (allow an explicit opt-out such as `password_optional = true` for dev). Proof:
      `b/internal/config/review_b_test.go`, case "sentinel without password".
- [ ] **Persistent / recoverable counting on Redis outage** (deferred from round 1, #4). A Redis stall
      longer than a few seconds fills the queue and new requests are dropped (503, `dropped_total`).
      Options: marker key per transaction for exact lost-reply recovery; a disk-backed queue. Decide
      whether the drop-and-alert behaviour is acceptable before building anything.

### Correctness

- [ ] **#9 Notification retries can send duplicate customer emails.** The web app's mailbox-level
      handler (`core/email.ex:3572-3584`) emails on every call; the org handler is guarded once per day.
      In `internal/notify`: treat a slow-but-2xx attempt as success (do not retry after the client timeout
      if the request was sent — or make the retry budget shorter than the web app's handling time), do
      not keep the concurrency slot while backing off, let Shutdown finish pending retries up to its
      deadline, and fix the "after N attempts" count in the failure line.
- [ ] **#12 Override keys are not lowercased.** `Info@/o=5` would be ignored and the domain default
      used instead. Lowercase the key in `domains.ParseLimits`; add a test row. No such keys in today's
      data.
- [ ] **#16 counterctl warns one day too early.** A date exactly 31 days back is reported as expired.
      Compare against the key TTL in whole days (`now.UTC` day minus date > 31). Proof:
      `b/cmd/counterctl/review_b_test.go` `TestReviewB_31DayWarningOffByOne`.
- [ ] **#18 Work keeps running after a handler timeout.** Not reproduced by the reviewer and largely
      covered by the queue change (the synchronous part only validates). Re-check that
      `http.TimeoutHandler` cannot leave a goroutine writing after the 503, then close.

### Operability / security hygiene

- [ ] **#11 Rejected requests leave no trace.** 4xx rejections write nothing to the counting log and no
      `trace_id` to the app log. Decode `trace_id` best-effort before rejecting; add a
      `rejected: <status> <reason>` line to the counting log and `trace_id` to the warn line. Proof:
      `b/internal/server/review_b_test.go` `TestReviewB_RejectLogHasNoTraceID`.
- [ ] **#13 Credentials in the notify URL are written to the log.** Redact `user:pass@` from URLs in
      the txlog `sent/failed notification` lines; warn at startup when a notify URL is plain `http://`
      with basic auth. Proof: `b/internal/server/review_b3_test.go`.
- [ ] **#14 Error responses expose internals.** 503 bodies carry Redis addresses and file paths; keep
      the detail in the log and answer a generic message. Default `admin_addr` to `127.0.0.1:9095`
      (currently `:9095`, all interfaces, no auth). Proof: `TestReviewB_ErrorLeak`.
- [ ] **#17 A recovered panic is not counted in `requests_total`.** The recoverer sits outside
      `observe`; swap the order or record the 500 explicitly. Proof: `TestReviewB_PanicNotInMetrics`.
- [ ] **Test gaps (B F13).** Watcher tests can occasionally pass on a stale signal — make them wait for
      the specific reload they triggered. Add a test for `handler_timeout` vs queue timeout interplay.

## Outside this repo (deployment prerequisites)

- [ ] **mailqueuer client**: `[counters]` section and a fire-and-forget POST `/in`|`/out` per recipient
      inside the publish loop of `mailqueuer/internal/handler/ingest.go` (after a successful publish;
      `/out` when `X-Auth-User` is set). Reply is `202 {"accepted":true}`; counted values only in the
      counting log and `GET /counters`.
- [ ] **Ansible role** in `ansible-freebsd3`, cloned from `roles/rcptd`: user/group, `config.toml.j2`
      (`auth_token` is mandatory, port 5555), rc.d script under `daemon(8)`, newsyslog entries for
      `app.log`, `counting.log`, `pull.log`, templated `contrib/pull-domains.sh` +
      `contrib/migadu_counters-pull.cron` (see `contrib/README.md`), agent host key acceptance,
      firewall: mailqueuer hosts → :5555, service → Sentinel :26379/:6379 and admin.migadu.com:443.
- [ ] **migadu_web** `/api/limits` `D` branch (domain level); until then `notify.domain_url` stays empty.
- [ ] **Limits re-tuning**: outgoing is counted per recipient (Lua counted per message), so existing
      `*/o` limits are stricter. Review before relying on `outgoing_limit_exceeded!`.
- [ ] **Retire stats-merger's quota module** once this service is the source (only fernrohr reads it).
- [ ] **Dependencies**: `golang.org/x/net` is pinned to v0.43.0 because the offline module cache had
      it; run `go get -u golang.org/x/net && go mod tidy` with network access.
- [ ] **git init** the repo (never asked for so far).