Add 80% limit warning, stop repeat notifications, fix review findings

Notifications:
- Warn once when a counter reaches warn_percent of its limit, through
  separate warn_*_url endpoints.
- Write ntf/ markers so a lost Redis reply cannot re-send a breach.
- Retry only when the request never left or the web app answered 429/5xx.

Fixes:
- Retry transactions Redis rejected; treat pool timeouts as definite.
- Install the signal handler before startup work.
- Stop logging the login suffix; redact notify URL credentials.
- Reject corrupt or empty CDBs, truncated hash tables included.
- Require a Sentinel password, refuse example placeholders, and default
  admin_addr to localhost.
- Normalise /counters domains, count on the arrival day, log rejections,
  hide internals in errors and exit 1 on a failed drain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Dimitrije Dimitrijevic
2026-11-09 09:01:56 +01:00
co-authored by Claude Opus 5.5
parent c43ff17627
commit 54f9626121
29 changed files with 1673 additions and 245 deletions
+4 -4
View File
@@ -18,8 +18,8 @@ func TestMetrics(t *testing.T) {
m.RecordIncrement("A", "o")
m.RecordIncrement("O", "i")
m.RecordLimitReached("O", "o")
m.RecordNotification("O", "sent")
m.RecordNotification("A", "failed")
m.RecordNotification("limit", "O", "sent")
m.RecordNotification("warning", "A", "failed")
m.RecordRedisError("incr")
m.SetCDBDomains(93549)
m.RecordReload(true)
@@ -40,8 +40,8 @@ func TestMetrics(t *testing.T) {
{"increments A o", m.Increments.WithLabelValues("A", "o"), 2},
{"increments O i", m.Increments.WithLabelValues("O", "i"), 1},
{"limit reached O o", m.LimitReached.WithLabelValues("O", "o"), 1},
{"notifications O sent", m.Notifications.WithLabelValues("O", "sent"), 1},
{"notifications A failed", m.Notifications.WithLabelValues("A", "failed"), 1},
{"notifications O sent", m.Notifications.WithLabelValues("limit", "O", "sent"), 1},
{"notifications A failed", m.Notifications.WithLabelValues("warning", "A", "failed"), 1},
{"redis errors incr", m.RedisErrors.WithLabelValues("incr"), 1},
{"cdb domains", m.CDBDomains, 93549},
{"reload success", m.CDBReloads.WithLabelValues("success"), 1},