Add 80% limit warning, stop repeat notifications, fix review findings

Notifications:
- Warn once when a counter reaches warn_percent of its limit, through
  separate warn_*_url endpoints.
- Write ntf/ markers so a lost Redis reply cannot re-send a breach.
- Retry only when the request never left or the web app answered 429/5xx.

Fixes:
- Retry transactions Redis rejected; treat pool timeouts as definite.
- Install the signal handler before startup work.
- Stop logging the login suffix; redact notify URL credentials.
- Reject corrupt or empty CDBs, truncated hash tables included.
- Require a Sentinel password, refuse example placeholders, and default
  admin_addr to localhost.
- Normalise /counters domains, count on the arrival day, log rejections,
  hide internals in errors and exit 1 on a failed drain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

README.md

@@ -90,20 +90,46 @@ Redis failures are handled per outcome: a key whose own `INCR` fails (it holds
 a non-number) is logged `redis error on key …` while the other keys are counted
 and checked for crossings; when the EXEC reply is lost after the write (timeout,
 reset) the keys are re-read once from the master and the block says `redis
-outcome unknown … re-read from master`, with a crossing reported when
-`count >= limit`; a definite failure (connection refused) logs `redis error: …
-(nothing counted)`.
+outcome unknown … re-read from master`; a transaction Redis rejected as a whole
+(READONLY or EXECABORT during a failover) applied nothing and is retried once
+after 200 ms; a definite failure (connection refused, or a rejection that
+repeats) logs `redis error: … (nothing counted)`.
 
 The CDB is pulled from the agent host by `contrib/pull-domains.sh` (see
 `contrib/README.md`): rsync, compare, verify, atomic `mv`, every 15 s.
 
 A limit is "reached" by the increment that moves the counter from below the
-limit to at or above it (`old < limit ≤ new`). Because INCRBY is atomic exactly
+limit to at or above it (`old < limit ≤ new`). Because INCR is atomic exactly
 one increment crosses, so the web app is notified once per level/id/flow/day —
 `GET <org_url|address_url|domain_url>` with `$LEVEL $ID $FLOW $PERIOD(=d)
 $ADDRESS $COUNT $LIMIT` substituted (URL-escaped), basic auth, in the
-background, 5 attempts with exponential backoff; 4xx is not retried. An empty
-URL disables notifications for that level.
+background. An empty URL disables notifications for that level.
+
+**Early warning.** The same rule at `warn_percent` of the limit (default 80,
+rounded up, `0` = off) calls `warn_org_url` / `warn_address_url` /
+`warn_domain_url`, which also accept `$PERCENT`. No warning URL means the
+feature is off. A limit too small to have a threshold below it (1 or 2 at 80%)
+has no warning. The warning URLs must not use the breach URL's path, because
+the web app treats every call there as a breach; the daemon refuses to start
+if they do. The web app endpoint is `GET /api/limits/warning`.
+
+**No repeats.** A breach or warning is sent once:
+
+- A confirmed crossing writes a marker, `ntf/<level>/<id>/d/<day>/<flow>/<limit|warn>`
+  (TTL 48 h). After a lost Redis reply a notification goes out only if `SET NX`
+  on that marker succeeds, so a counter that already notified stays silent. If
+  Redis cannot answer, nothing is sent and `_redis_errors_total{op="marker"}` rises.
+- An attempt is retried only when the request never left or the web app answered
+  429 or 5xx, up to 5 attempts with exponential backoff. A request that was sent
+  but got no answer may have been acted on, so it is not retried; it is logged
+  as `outcome unknown`.
+- The web app may `DEL` a day key to reset a customer; the next confirmed
+  crossing notifies again, and the web app's once-per-day guard decides whether
+  an email goes out.
+
+Delivery is therefore at most once. A notification that fails, or is in flight
+when the daemon stops past its 30 s deadline, is not sent later that day; look
+for `failed limit` in the counting log and `_notifications_total{result="failed"}`.
 
 ## Counting transaction log
 
@@ -125,8 +151,10 @@ Every request writes one block in the shape of the old Lua plugin's log lines
 `grep '<trace_id>'` returns one transaction; `grep 'skipped:'` every request
 that counted nothing; `grep 'not usable'` every side that failed validation
 (`sender address not usable: nobody (invalid e-mail address, not counted)`);
-`grep 'dropped:'` every request refused by a full queue. A token login shows as
-`… login_suffix=SUPPORTTOKEN` on the request line. `counting_format = "json"` emits the same events as one
+`grep 'dropped:'` every request refused by a full queue; `grep 'rejected:'`
+every `/in` or `/out` body refused by validation. A `user@dom@SUFFIX` login
+shows as `… token_login=true` on the request line; the suffix may be a
+credential and is never logged. `counting_format = "json"` emits the same events as one
 JSON object per line. Domains with thousands of per-mailbox overrides print the
 base keys plus the address's own overrides and a count of the rest.
 
@@ -140,16 +168,19 @@ migadu_counters -version
 Foreground process; two listeners: `addr` (API, default `:5555`) and
 `admin_addr` (`/health`, `/metrics`, no auth). Signals: `SIGHUP` reopens both
 log files and reloads the CDB; `SIGTERM`/`SIGINT` stop accepting, drain the
-counting queue (every accepted request is counted), wait for in-flight
-notifications (≤ 30 s) and exit 0. On FreeBSD run it like rcptd: an rc.d script
+counting queue, wait for in-flight notifications and their retries (≤ 30 s)
+and exit 0. If the deadline passes first, the remaining jobs are lost and the
+exit code is 1. On FreeBSD run it like rcptd: an rc.d script
 with `/usr/sbin/daemon -f -H -p <pidfile> -o <logfile>`; `service
 migadu_counters reload` sends SIGHUP. There is deliberately no self-daemonising
-or pidfile option (daemon(8) owns both). Startup fails without a readable CDB;
-an unreachable Redis is only warned about (`/health` answers 503 until it is
+or pidfile option (daemon(8) owns both). Startup fails without a readable, non-empty
+CDB; an unreachable Redis is only warned about (`/health` answers 503 until it is
 back).
 
-See `config.toml.example` for every option. `auth_token` is mandatory and
-unknown keys (typos such as `auth-token`) are fatal at startup. Secrets may
+See `config.toml.example` for every option. `auth_token` is mandatory, the
+example's `"…"` placeholders are refused, `admin_addr` defaults to
+`127.0.0.1:9095`, `redis.password` is required with Sentinel (unless
+`redis.password_optional = true`) and unknown keys (typos such as `auth-token`) are fatal at startup. Secrets may
 come from the environment: `MIGADU_COUNTERS_AUTH_TOKEN`,
 `MIGADU_COUNTERS_REDIS_PASSWORD`, `MIGADU_COUNTERS_NOTIFY_PASSWORD`. `redis.db` must stay 0. Production uses
 Sentinel (`sentinels`, `master_name = "mymaster"`): writes go to the master,
@@ -158,8 +189,9 @@ reads to a replica (falling back to the master when none is known); without
 
 Metrics (`/metrics` on the admin port): `migadu_counters_requests_total{route,status}`,
 `_request_duration_seconds{route}`, `_increments_total{level,flow}`,
-`_limit_reached_total{level,flow}`, `_notifications_total{level,result}`,
-`_redis_errors_total{op}` (`incr`, `incr_unknown`, `mget`, `ping`),
+`_limit_reached_total{level,flow}`, `_warning_reached_total{level,flow}`,
+`_notifications_total{kind,level,result}` (`kind`: `limit`, `warning`),
+`_redis_errors_total{op}` (`incr`, `incr_unknown`, `mget`, `ping`, `marker`),
 `_dropped_total{route}`, `_queue_depth`, `_queue_wait_seconds`, `_cdb_domains`,
 `_cdb_reloads_total{result}`.
 
@@ -195,12 +227,16 @@ on any error.
 
 ```
 make build      # bin/migadu_counters, bin/counterctl
-make test       # go test -race ./...  (tests use test_files/rspamd_domains_db.cdb and miniredis)
+make test       # go test -race ./...  (tests use miniredis and test_files/rspamd_domains_db.cdb)
 make vet        # go vet + gofmt
 make test-contrib  # shell test for contrib/pull-domains.sh (needs tinycdb cdb + rsync)
 make freebsd    # static FreeBSD amd64 binaries for the Ansible role
 ```
 
+`test_files/` is gitignored because the CDB holds real customer domains, so on
+a fresh clone the tests fail until you copy `rspamd_domains_db` from an agent
+host (`/usr/local/etc/agent/build/rspamd/`) to `test_files/rspamd_domains_db.cdb`.
+
 Manual check with a local Redis: `redis-server --port 6379`, a config in static
 mode (`write_addr = "127.0.0.1:6379"`) pointing `domains_cdb` at
 `test_files/rspamd_domains_db.cdb`, then