Dimitrije DimitrijevicandClaude Opus 5.5
54f9626121
Notifications: - Warn once when a counter reaches warn_percent of its limit, through separate warn_*_url endpoints. - Write ntf/ markers so a lost Redis reply cannot re-send a breach. - Retry only when the request never left or the web app answered 429/5xx. Fixes: - Retry transactions Redis rejected; treat pool timeouts as definite. - Install the signal handler before startup work. - Stop logging the login suffix; redact notify URL credentials. - Reject corrupt or empty CDBs, truncated hash tables included. - Require a Sentinel password, refuse example placeholders, and default admin_addr to localhost. - Normalise /counters domains, count on the arrival day, log rejections, hide internals in errors and exit 1 on a failed drain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
55 lines
3.6 KiB
Markdown
55 lines
3.6 KiB
Markdown
# TODO — migadu_counters
|
||
|
||
State as of 2026-11-09. Review findings refer to `review/README.md`; the proof tests live in
|
||
`review/_proof_tests/` and should become regression tests as each item is fixed (round 1 and items
|
||
7–8 are mapped at the end of `review/_proof_tests/README.md`).
|
||
|
||
## Done
|
||
|
||
- Review round 1: #1–#6 (queue + 202, lenient validation + IDN, token logins, INCR with per-key
|
||
results and master re-read, CDB parse cache, mandatory token / port 5555), plus
|
||
`contrib/pull-domains.sh` and `counterctl --in/--out`.
|
||
- #7 plus-addressing counts against the mailbox (`domains.Mailbox`).
|
||
- #8 malformed CDB row is "not hosted" + warning (`domains.ErrMalformed`).
|
||
|
||
## Review round 2 — done 2026-11-09
|
||
|
||
#9-#17 fixed, plus the new findings from the 2026-11-09 review: repeat notifications after a lost Redis
|
||
reply, pool timeouts treated as unknown outcomes, login suffix in the counting log, no retry of rejected
|
||
transactions, `/counters` punycode, counter day taken from arrival time, CDB corruption checks, context
|
||
deadlines on Redis sockets, example config path and placeholders, non-zero exit on a failed drain.
|
||
Also added: the 80% early warning (`warn_percent`, `warn_*_url`) and per-counter notification markers.
|
||
|
||
## Open
|
||
|
||
- [ ] **Persistent / recoverable counting on Redis outage** (deferred from round 1, #4). A Redis stall
|
||
longer than a few seconds fills the queue and new requests are dropped (503, `dropped_total`).
|
||
Decide whether drop-and-alert is acceptable before building anything.
|
||
- [ ] **Exempt `mail_from` on `/in`.** An outside sender using `MAIL FROM:<support@migadu.com>` avoids the
|
||
recipient's incoming count, unless mizu rejects such senders inbound (not confirmed). Matches the
|
||
old Lua rule; decide whether to keep it.
|
||
- [ ] **Test gap.** Watcher tests can occasionally pass on a stale signal: make them wait for the
|
||
specific reload they triggered.
|
||
|
||
## Outside this repo (deployment prerequisites)
|
||
|
||
- [ ] **mailqueuer client**: `[counters]` section and a fire-and-forget POST `/in`|`/out` per recipient
|
||
inside the publish loop of `mailqueuer/internal/handler/ingest.go` (after a successful publish;
|
||
`/out` when `X-Auth-User` is set). Reply is `202 {"accepted":true}`; counted values only in the
|
||
counting log and `GET /counters`.
|
||
- [ ] **Ansible role** in `ansible-freebsd3`, cloned from `roles/rcptd`: user/group, `config.toml.j2`
|
||
(`auth_token` is mandatory, port 5555), rc.d script under `daemon(8)`, newsyslog entries for
|
||
`app.log`, `counting.log`, `pull.log`, templated `contrib/pull-domains.sh` +
|
||
`contrib/migadu_counters-pull.cron` (see `contrib/README.md`), agent host key acceptance,
|
||
firewall: mailqueuer hosts → :5555, service → Sentinel :26379/:6379 and admin.migadu.com:443.
|
||
- [ ] **migadu_web** `/api/limits` `D` branch (domain level); until then `notify.domain_url` stays empty.
|
||
- [ ] **migadu_web3 branch `limit-warning-notifications`**: review the warning email wording, merge and deploy,
|
||
then set `warn_org_url` and `warn_address_url`. Until then warnings stay off.
|
||
- [ ] **Firewall / monitoring**: `admin_addr` now defaults to localhost; set it explicitly if `/metrics` or
|
||
`/health` are scraped from another host.
|
||
- [ ] **Limits re-tuning**: outgoing is counted per recipient (Lua counted per message), so existing
|
||
`*/o` limits are stricter. Review before relying on `outgoing_limit_exceeded!`.
|
||
- [ ] **Retire stats-merger's quota module** once this service is the source (only fernrohr reads it).
|
||
- [ ] **Dependencies**: `golang.org/x/net` is pinned to v0.43.0 because the offline module cache had
|
||
it; run `go get -u golang.org/x/net && go mod tidy` with network access.
|