First commit Go counter service
This commit is contained in:
Dimitrije Dimitrijevic
@@ -0,0 +1,4 @@
|
||||
bin/
|
||||
coverage.out
|
||||
*.tmp
|
||||
test_files/
|
||||
@@ -0,0 +1,80 @@
|
||||
.PHONY: all build build-daemon build-ctl test test-contrib vet coverage freebsd build-linux clean help
|
||||
|
||||
# ====================================================================================
|
||||
# Variables
|
||||
# ====================================================================================
|
||||
|
||||
BINARY ?= migadu_counters
|
||||
CTL_BINARY ?= counterctl
|
||||
BIN_DIR ?= ./bin
|
||||
SRC = ./cmd/migadu_counters
|
||||
CTL_SRC = ./cmd/counterctl
|
||||
|
||||
# Version information (falls back when not in a git checkout)
|
||||
VERSION ?= $(shell git describe --tags --always --dirty --match='v*' 2>/dev/null || echo dev)
|
||||
COMMIT ?= $(shell git rev-parse --short HEAD 2>/dev/null || echo none)
|
||||
DATE ?= $(shell date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
LDFLAGS_VARS = -X 'main.version=${VERSION}' -X 'main.commit=${COMMIT}' -X 'main.date=${DATE}'
|
||||
LDFLAGS = -ldflags="${LDFLAGS_VARS}"
|
||||
|
||||
# Cross-compilation (FreeBSD hosts run these under rc.d + daemon(8))
|
||||
FREEBSD_BINARY ?= migadu_counters-freebsd-amd64
|
||||
CTL_FREEBSD_BINARY ?= counterctl-freebsd-amd64
|
||||
LINUX_BINARY ?= migadu_counters-linux-amd64
|
||||
CTL_LINUX_BINARY ?= counterctl-linux-amd64
|
||||
|
||||
# ====================================================================================
|
||||
# Targets
|
||||
# ====================================================================================
|
||||
|
||||
all: build
|
||||
|
||||
build: build-daemon build-ctl
|
||||
|
||||
build-daemon:
|
||||
@mkdir -p $(BIN_DIR)
|
||||
go build $(LDFLAGS) -o $(BIN_DIR)/$(BINARY) $(SRC)
|
||||
|
||||
build-ctl:
|
||||
@mkdir -p $(BIN_DIR)
|
||||
go build $(LDFLAGS) -o $(BIN_DIR)/$(CTL_BINARY) $(CTL_SRC)
|
||||
|
||||
test:
|
||||
go test -race ./...
|
||||
|
||||
# Shell test for contrib/pull-domains.sh (needs tinycdb's cdb and rsync).
|
||||
test-contrib:
|
||||
bash contrib/pull-domains_test.sh
|
||||
|
||||
vet:
|
||||
go vet ./...
|
||||
@test -z "$$(gofmt -l . | grep -v -e '^test_files/' -e '^review/')" || (echo "gofmt needed:" && gofmt -l . | grep -v -e '^test_files/' -e '^review/' && exit 1)
|
||||
|
||||
coverage:
|
||||
go test -coverprofile=coverage.out ./...
|
||||
go tool cover -html=coverage.out
|
||||
|
||||
freebsd:
|
||||
@mkdir -p $(BIN_DIR)
|
||||
CGO_ENABLED=0 GOOS=freebsd GOARCH=amd64 go build -ldflags="${LDFLAGS_VARS}" -o $(BIN_DIR)/$(FREEBSD_BINARY) $(SRC)
|
||||
CGO_ENABLED=0 GOOS=freebsd GOARCH=amd64 go build -ldflags="${LDFLAGS_VARS}" -o $(BIN_DIR)/$(CTL_FREEBSD_BINARY) $(CTL_SRC)
|
||||
|
||||
build-linux:
|
||||
@mkdir -p $(BIN_DIR)
|
||||
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="${LDFLAGS_VARS}" -o $(BIN_DIR)/$(LINUX_BINARY) $(SRC)
|
||||
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="${LDFLAGS_VARS}" -o $(BIN_DIR)/$(CTL_LINUX_BINARY) $(CTL_SRC)
|
||||
|
||||
clean:
|
||||
rm -rf $(BIN_DIR) coverage.out
|
||||
|
||||
help:
|
||||
@echo "Targets:"
|
||||
@echo " build Build migadu_counters and counterctl into $(BIN_DIR)"
|
||||
@echo " test go test -race ./..."
|
||||
@echo " test-contrib Shell test for contrib/pull-domains.sh"
|
||||
@echo " vet go vet + gofmt check"
|
||||
@echo " coverage Coverage report"
|
||||
@echo " freebsd Cross-compile static FreeBSD amd64 binaries"
|
||||
@echo " build-linux Cross-compile static Linux amd64 binaries"
|
||||
@echo " clean Remove build artifacts"
|
||||
@@ -0,0 +1,213 @@
|
||||
# migadu_counters
|
||||
|
||||
Real-time mail counting for Migadu. mailqueuer tells it about every ingested
|
||||
recipient; it increments per-address / per-domain / per-organisation daily
|
||||
counters in Redis (the keys the web app already charts) and, when an increment
|
||||
reaches a configured limit, notifies the web app, which decides what happens to
|
||||
the customer. It replaces the rspamd Lua plugin `migadu_counters.lua` and the
|
||||
ledger-derived quota in stats-merger. It returns no verdicts and enforces
|
||||
nothing itself.
|
||||
|
||||
Design record: `plans/2026-10-13-migadu-counters.md`.
|
||||
|
||||
```
|
||||
sender ──SMTP──▶ mizu ──POST /ingest──▶ mailqueuer
|
||||
│ per recipient, after a successful publish:
|
||||
│ fire-and-forget HTTP call, reply ignored
|
||||
▼
|
||||
POST /out {"trace_id","auth_user","mail_from","recipient"} ← X-Auth-User set
|
||||
POST /in {"trace_id","mail_from","recipient","junk"} ← X-Auth-User empty
|
||||
Authorization: Bearer <token>
|
||||
▼
|
||||
┌─────────────────────────────────────┐
|
||||
rspamd_domains_db ───▶│ migadu_counters (:5555, admin :9095)│
|
||||
(domain → org|limits) │ validate → 202 → queue → workers: │
|
||||
│ lookup domain → MULTI/INCR → notify │──▶ GET admin.migadu.com/api/limits
|
||||
└──────────────────┬──────────────────┘ (on the increment that crosses a limit)
|
||||
▼
|
||||
Redis Sentinel: writes → master, reads → replicas
|
||||
DB 0: cnt/<A|D|O>/<id>/d/<YYYY-MM-DD>/<i|o> (TTL 31 d)
|
||||
|
||||
counterctl --address x@y [--date YYYY-MM-DD] ──GET /counters──▶ migadu_counters ──▶ replicas
|
||||
```
|
||||
|
||||
## API (for mailqueuer)
|
||||
|
||||
Every request: `Authorization: Bearer <auth_token>` (same value in mailqueuer's
|
||||
config; the daemon refuses to start without one), `Content-Type:
|
||||
application/json`, body ≤ 64 KB, one JSON object. Unknown fields are ignored.
|
||||
mailqueuer never waits for or acts on the reply.
|
||||
|
||||
Counting is **decoupled from the request**: `/in` and `/out` validate the body,
|
||||
enqueue a job and answer `202` at once; a worker pool does the CDB lookup, the
|
||||
Redis transaction and the notification under its own time budget, so a client
|
||||
that hangs up or times out can no longer cancel a count. The counted values
|
||||
appear in the counting log and in `GET /counters`, not in the reply.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| `POST /out` | `{"trace_id":"…","auth_user":"x@example.com","mail_from":"x@example.com","recipient":"y@example.com"}` — one call per recipient of an authenticated submission. Sender A/D/O **outgoing** +1 (if the sender's domain is hosted), recipient A/D/O **incoming** +1 (if the recipient's domain is hosted). `auth_user` and `recipient` are validated independently: whichever side is a usable address is counted, the other is logged as `not usable`; `400` only when neither is. A support/token login `user@dom@SUFFIX` counts under `user@dom` (the suffix is logged). |
|
||||
| `POST /in` | `{"trace_id":"…","mail_from":"s@gmail.com","recipient":"y@example.com","junk":false}` — one call per recipient of inbound mail. Recipient A/D/O **incoming** +1 if hosted. `junk:true` is skipped unless `count_junk = true`. `mail_from` may be `""` (bounce) or anything mizu accepted — it only feeds the exempt check and never rejects the request. |
|
||||
| `GET /counters?address=…` or `?domain=…` `[&date=YYYY-MM-DD]` | Today's (UTC) or a given day's values and limits, read from the replicas. |
|
||||
|
||||
Responses: `202 {"accepted":true}` (validated and queued; skips such as
|
||||
exempt/junk/unknown domain are decided by the worker and appear only in the
|
||||
counting log), `4xx {"error":"…"}` from the validation layer (`405` method,
|
||||
`413` size, `401` token, `403` source IP, `415` content type, `400` JSON /
|
||||
required fields / e-mail syntax), `503 {"error":"queue full"}` when the
|
||||
counting queue (`queue_size`) is full — the drop is counted in
|
||||
`dropped_total{route}` and logged — or `503` on a handler timeout of the
|
||||
synchronous part.
|
||||
|
||||
Rules: addresses are trimmed and lowercased, Unicode domains are converted to
|
||||
punycode (the CDB key), and a plus-address tag is dropped so `b+tag@…` counts
|
||||
against the mailbox `b@…` (the counter the web app reads); mail to or from an exempt address (`support@migadu.com`,
|
||||
`contact@migadu.com`) is never counted; a domain that is not in
|
||||
`rspamd_domains_db` is not hosted here and not counted; the sender of `/in`
|
||||
mail is never counted; there is no reject/soft-reject — counting continues past
|
||||
the limit.
|
||||
|
||||
## Limits and keys
|
||||
|
||||
Limits come from the rspamd domains CDB (tinycdb, built by the agent):
|
||||
`example.com → 1049|O/i=200,O/o=20,D/i=200,D/o=20,A/i=200,A/o=20,info@/o=5`.
|
||||
`<local>@/<flow>` overrides the `A/<flow>` default for one mailbox; a missing
|
||||
key means unlimited. The file is reloaded on SIGHUP and automatically when it
|
||||
changes on disk (atomic `mv` is detected); a file that cannot be read or is
|
||||
empty is rejected and the current table kept. A row that exists but cannot be
|
||||
parsed (non-numeric org id) is treated as "not hosted" for that domain and
|
||||
warned about in the app log; the other side of the request is still counted.
|
||||
Always replace the file with
|
||||
`mv` (as the agent's `cdb -c` + `mv` do) — rewriting it in place truncates the
|
||||
inode the daemon holds open and breaks lookups until the next reload.
|
||||
|
||||
Redis keys are exactly the web app's (`migadu_web` `core/redis_sentinel.ex`):
|
||||
`cnt/<A|D|O>/<id>/d/<YYYY-MM-DD>/<i|o>` in **DB 0**, UTC day, TTL 31 days,
|
||||
`INCR`+`EXPIRE` in one MULTI/EXEC. The web app may `DEL` a day key to reset a
|
||||
customer; the next crossing notifies again.
|
||||
|
||||
Redis failures are handled per outcome: a key whose own `INCR` fails (it holds
|
||||
a non-number) is logged `redis error on key …` while the other keys are counted
|
||||
and checked for crossings; when the EXEC reply is lost after the write (timeout,
|
||||
reset) the keys are re-read once from the master and the block says `redis
|
||||
outcome unknown … re-read from master`, with a crossing reported when
|
||||
`count >= limit`; a definite failure (connection refused) logs `redis error: …
|
||||
(nothing counted)`.
|
||||
|
||||
The CDB is pulled from the agent host by `contrib/pull-domains.sh` (see
|
||||
`contrib/README.md`): rsync, compare, verify, atomic `mv`, every 15 s.
|
||||
|
||||
A limit is "reached" by the increment that moves the counter from below the
|
||||
limit to at or above it (`old < limit ≤ new`). Because INCRBY is atomic exactly
|
||||
one increment crosses, so the web app is notified once per level/id/flow/day —
|
||||
`GET <org_url|address_url|domain_url>` with `$LEVEL $ID $FLOW $PERIOD(=d)
|
||||
$ADDRESS $COUNT $LIMIT` substituted (URL-escaped), basic auth, in the
|
||||
background, 5 attempts with exponential backoff; 4xx is not retried. An empty
|
||||
URL disables notifications for that level.
|
||||
|
||||
## Counting transaction log
|
||||
|
||||
Every request writes one block in the shape of the old Lua plugin's log lines
|
||||
(rspamd's task id replaced by the trace id) to `logging.counting_output`:
|
||||
|
||||
```
|
||||
2026-10-06 01:12:06.412 <ad8525> out; request auth_user=noah@example.com recipient=y@example.org mail_from=noah@example.com
|
||||
2026-10-06 01:12:06.412 <ad8525> out; local domain: example.com account_id=116681
|
||||
2026-10-06 01:12:06.412 <ad8525> out; domain: example.com limits_map={A/i=500, A/o=100, D/i=10000, D/o=2200, O/i=10000, O/o=2200}
|
||||
2026-10-06 01:12:06.412 <ad8525> out; determined limit (level=[A] flow=[o] key=[cnt/A/noah@example.com/d/2026-10-06/o]) => 100
|
||||
2026-10-06 01:12:06.412 <ad8525> out; recipient domain not in domains db: example.org (not counted)
|
||||
2026-10-06 01:12:06.414 <ad8525> out; tracked address counters for noah@example.com => count=12 limit=100 reached=false
|
||||
2026-10-06 01:12:06.414 <ad8525> out; tracked account counters for 116681 => count=2200 limit=2200 reached=true
|
||||
2026-10-06 01:12:06.414 <ad8525> out; limit reached (level=[O] id=[116681] flow=[o] by=[noah@example.com]) => notifying org_url
|
||||
2026-10-06 01:12:06.903 <ad8525> out; sent limit exceeded notification to url "https://admin.migadu.com/api/limits?…"
|
||||
```
|
||||
|
||||
`grep '<trace_id>'` returns one transaction; `grep 'skipped:'` every request
|
||||
that counted nothing; `grep 'not usable'` every side that failed validation
|
||||
(`sender address not usable: nobody (invalid e-mail address, not counted)`);
|
||||
`grep 'dropped:'` every request refused by a full queue. A token login shows as
|
||||
`… login_suffix=SUPPORTTOKEN` on the request line. `counting_format = "json"` emits the same events as one
|
||||
JSON object per line. Domains with thousands of per-mailbox overrides print the
|
||||
base keys plus the address's own overrides and a count of the rest.
|
||||
|
||||
## Running
|
||||
|
||||
```
|
||||
migadu_counters -config /usr/local/etc/migadu_counters/config.toml # default paths: /etc/…, /usr/local/etc/…
|
||||
migadu_counters -version
|
||||
```
|
||||
|
||||
Foreground process; two listeners: `addr` (API, default `:5555`) and
|
||||
`admin_addr` (`/health`, `/metrics`, no auth). Signals: `SIGHUP` reopens both
|
||||
log files and reloads the CDB; `SIGTERM`/`SIGINT` stop accepting, drain the
|
||||
counting queue (every accepted request is counted), wait for in-flight
|
||||
notifications (≤ 30 s) and exit 0. On FreeBSD run it like rcptd: an rc.d script
|
||||
with `/usr/sbin/daemon -f -H -p <pidfile> -o <logfile>`; `service
|
||||
migadu_counters reload` sends SIGHUP. There is deliberately no self-daemonising
|
||||
or pidfile option (daemon(8) owns both). Startup fails without a readable CDB;
|
||||
an unreachable Redis is only warned about (`/health` answers 503 until it is
|
||||
back).
|
||||
|
||||
See `config.toml.example` for every option. `auth_token` is mandatory and
|
||||
unknown keys (typos such as `auth-token`) are fatal at startup. Secrets may
|
||||
come from the environment: `MIGADU_COUNTERS_AUTH_TOKEN`,
|
||||
`MIGADU_COUNTERS_REDIS_PASSWORD`, `MIGADU_COUNTERS_NOTIFY_PASSWORD`. `redis.db` must stay 0. Production uses
|
||||
Sentinel (`sentinels`, `master_name = "mymaster"`): writes go to the master,
|
||||
reads to a replica (falling back to the master when none is known); without
|
||||
`sentinels` the static `write_addr`/`read_addrs` are used.
|
||||
|
||||
Metrics (`/metrics` on the admin port): `migadu_counters_requests_total{route,status}`,
|
||||
`_request_duration_seconds{route}`, `_increments_total{level,flow}`,
|
||||
`_limit_reached_total{level,flow}`, `_notifications_total{level,result}`,
|
||||
`_redis_errors_total{op}` (`incr`, `incr_unknown`, `mget`, `ping`),
|
||||
`_dropped_total{route}`, `_queue_depth`, `_queue_wait_seconds`, `_cdb_domains`,
|
||||
`_cdb_reloads_total{result}`.
|
||||
|
||||
## counterctl
|
||||
|
||||
```
|
||||
counterctl --address me@example.com # today, UTC
|
||||
counterctl --address me@example.com --date 2026-10-12
|
||||
counterctl --address me@example.com --out # outgoing columns only (--in: incoming)
|
||||
counterctl --domain example.com --json
|
||||
```
|
||||
|
||||
Reads `addr`/`auth_token` from the daemon's config (`--config`, default paths,
|
||||
`MIGADU_COUNTERS_AUTH_TOKEN` overrides the token; `--url` overrides the
|
||||
address) and calls `GET /counters`. Output:
|
||||
|
||||
```
|
||||
me@example.com date 2026-10-13 (UTC) domain example.com org 1234
|
||||
LEVEL ID IN LIMIT OUT LIMIT
|
||||
A me@example.com 12 500 200! 200
|
||||
D example.com 40 1000 9 -
|
||||
O 1234 120 5000 30 1000
|
||||
```
|
||||
|
||||
`!` marks a value at or above its limit, `-` an unlimited level. `--in` /
|
||||
`--out` restrict the table to one flow (`LEVEL ID IN LIMIT` or `LEVEL ID OUT
|
||||
LIMIT`; both or neither show the full table; `--json` ignores them). `--date` takes
|
||||
an ISO 8601 calendar date (`YYYY-MM-DD`); future dates are refused and dates
|
||||
older than 31 days are warned about (keys have expired). Exit 0 on success, 1
|
||||
on any error.
|
||||
|
||||
## Development
|
||||
|
||||
```
|
||||
make build # bin/migadu_counters, bin/counterctl
|
||||
make test # go test -race ./... (tests use test_files/rspamd_domains_db.cdb and miniredis)
|
||||
make vet # go vet + gofmt
|
||||
make test-contrib # shell test for contrib/pull-domains.sh (needs tinycdb cdb + rsync)
|
||||
make freebsd # static FreeBSD amd64 binaries for the Ansible role
|
||||
```
|
||||
|
||||
Manual check with a local Redis: `redis-server --port 6379`, a config in static
|
||||
mode (`write_addr = "127.0.0.1:6379"`) pointing `domains_cdb` at
|
||||
`test_files/rspamd_domains_db.cdb`, then
|
||||
|
||||
```
|
||||
curl -s -XPOST localhost:5555/out -H 'Authorization: Bearer <token>' -H 'Content-Type: application/json' \
|
||||
-d '{"trace_id":"t1","auth_user":"a@yagolivas.com","recipient":"x@gmail.com"}' # -> {"accepted":true}
|
||||
redis-cli -n 0 KEYS 'cnt/*'
|
||||
go run ./cmd/counterctl -config config.toml --address a@yagolivas.com --out
|
||||
```
|
||||
@@ -0,0 +1,255 @@
|
||||
// counterctl shows today's (or a given day's) incoming/outgoing counters for
|
||||
// an address or a domain by asking the running migadu_counters daemon
|
||||
// (GET /counters), the same way rcptctl talks to rcptd.
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"text/tabwriter"
|
||||
"time"
|
||||
|
||||
"github.com/BurntSushi/toml"
|
||||
|
||||
"migadu_counters/internal/api"
|
||||
"migadu_counters/internal/config"
|
||||
)
|
||||
|
||||
// Set by -ldflags at build time.
|
||||
var (
|
||||
version = "dev"
|
||||
commit = "none"
|
||||
date = "unknown"
|
||||
)
|
||||
|
||||
func main() {
|
||||
os.Exit(run(os.Args[1:], os.Stdout, os.Stderr, os.Getenv, time.Now))
|
||||
}
|
||||
|
||||
// cliConfig mirrors the daemon config fields the CLI needs; TOML ignores the
|
||||
// rest, so the daemon's own config file can be used as is.
|
||||
type cliConfig struct {
|
||||
Addr string `toml:"addr"`
|
||||
AuthToken string `toml:"auth_token"`
|
||||
}
|
||||
|
||||
func run(args []string, stdout, stderr io.Writer, getenv func(string) string, now func() time.Time) int {
|
||||
fs := flag.NewFlagSet("counterctl", flag.ContinueOnError)
|
||||
fs.SetOutput(stderr)
|
||||
address := fs.String("address", "", "mailbox address to show (A, D and O levels)")
|
||||
domain := fs.String("domain", "", "domain to show (D and O levels)")
|
||||
dateFlag := fs.String("date", "", "ISO 8601 calendar day YYYY-MM-DD, UTC (default: today)")
|
||||
jsonOut := fs.Bool("json", false, "print the raw API response")
|
||||
inOnly := fs.Bool("in", false, "show only the incoming counters")
|
||||
outOnly := fs.Bool("out", false, "show only the outgoing counters")
|
||||
configPath := fs.String("config", "", "config file (default: "+strings.Join(config.DefaultPaths, " or ")+")")
|
||||
urlFlag := fs.String("url", "", "service base URL (default: derived from addr in the config)")
|
||||
showVersion := fs.Bool("version", false, "print version and exit")
|
||||
fs.Usage = func() {
|
||||
fmt.Fprintf(stderr, "Usage: counterctl --address me@example.com | --domain example.com [--date YYYY-MM-DD] [--in|--out] [--json] [--config PATH] [--url URL]\n\n")
|
||||
fs.PrintDefaults()
|
||||
}
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return 1
|
||||
}
|
||||
if *showVersion {
|
||||
fmt.Fprintf(stdout, "counterctl %s (commit %s, built %s)\n", version, commit, date)
|
||||
return 0
|
||||
}
|
||||
if (*address == "") == (*domain == "") {
|
||||
fmt.Fprintln(stderr, "exactly one of --address or --domain is required")
|
||||
fs.Usage()
|
||||
return 1
|
||||
}
|
||||
|
||||
q := url.Values{}
|
||||
if *address != "" {
|
||||
addr, err := api.NormalizeEmail(*address)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "invalid address %q: %v\n", *address, err)
|
||||
return 1
|
||||
}
|
||||
q.Set("address", addr)
|
||||
} else {
|
||||
q.Set("domain", strings.ToLower(strings.TrimSpace(*domain)))
|
||||
}
|
||||
if *dateFlag != "" {
|
||||
day, err := api.ParseDate(*dateFlag, now())
|
||||
if err != nil {
|
||||
fmt.Fprintln(stderr, err)
|
||||
return 1
|
||||
}
|
||||
q.Set("date", day)
|
||||
if t, _ := time.Parse("2006-01-02", day); now().UTC().Sub(t) > 31*24*time.Hour {
|
||||
fmt.Fprintf(stderr, "warning: %s is more than 31 days ago; counter keys expire after 31 days and will read as 0\n", day)
|
||||
}
|
||||
}
|
||||
|
||||
cfg, err := loadConfig(*configPath, *urlFlag != "")
|
||||
if err != nil {
|
||||
fmt.Fprintln(stderr, err)
|
||||
return 1
|
||||
}
|
||||
token := cfg.AuthToken
|
||||
if v := getenv(config.EnvAuthToken); v != "" {
|
||||
token = v
|
||||
}
|
||||
base := *urlFlag
|
||||
if base == "" {
|
||||
base = baseURL(cfg.Addr)
|
||||
if host := strings.TrimPrefix(strings.TrimPrefix(base, "http://"), "https://"); !strings.HasPrefix(host, "localhost") && !strings.HasPrefix(host, "127.0.0.1") && !strings.HasPrefix(host, "[::1]") {
|
||||
fmt.Fprintf(stderr, "warning: %s is not local; the auth token travels over plaintext HTTP\n", base)
|
||||
}
|
||||
}
|
||||
|
||||
req, err := http.NewRequest(http.MethodGet, base+"/counters?"+q.Encode(), nil)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "bad URL: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
if token != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
}
|
||||
resp, err := (&http.Client{Timeout: 10 * time.Second}).Do(req)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "request failed: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
body, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "reading response: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
msg := strings.TrimSpace(string(body))
|
||||
var e api.ErrorResponse
|
||||
if json.Unmarshal(body, &e) == nil && e.Error != "" {
|
||||
msg = e.Error
|
||||
}
|
||||
fmt.Fprintf(stderr, "server answered %d: %s\n", resp.StatusCode, msg)
|
||||
return 1
|
||||
}
|
||||
if *jsonOut {
|
||||
stdout.Write(body)
|
||||
if len(body) == 0 || body[len(body)-1] != '\n' {
|
||||
io.WriteString(stdout, "\n")
|
||||
}
|
||||
return 0
|
||||
}
|
||||
var r api.CountersResponse
|
||||
if err := json.Unmarshal(body, &r); err != nil {
|
||||
fmt.Fprintf(stderr, "unexpected response: %v\n%s\n", err, body)
|
||||
return 1
|
||||
}
|
||||
showIn, showOut := flows(*inOnly, *outOnly)
|
||||
printTable(stdout, r, showIn, showOut)
|
||||
return 0
|
||||
}
|
||||
|
||||
// flows picks the table columns: --in or --out alone restrict the table to
|
||||
// that flow; neither or both show both flows.
|
||||
func flows(in, out bool) (showIn, showOut bool) {
|
||||
if in == out {
|
||||
return true, true
|
||||
}
|
||||
return in, out
|
||||
}
|
||||
|
||||
// loadConfig reads addr/auth_token from the daemon config. Without -config
|
||||
// the default paths are tried; when none exists the CLI can still run with
|
||||
// -url and the token from the environment.
|
||||
func loadConfig(path string, haveURL bool) (cliConfig, error) {
|
||||
var cfg cliConfig
|
||||
if path == "" {
|
||||
found, err := config.Discover("")
|
||||
if err != nil {
|
||||
if haveURL {
|
||||
return cfg, nil
|
||||
}
|
||||
return cfg, err
|
||||
}
|
||||
path = found
|
||||
}
|
||||
if _, err := toml.DecodeFile(path, &cfg); err != nil {
|
||||
return cfg, fmt.Errorf("read config %s: %v", path, err)
|
||||
}
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
// baseURL turns the daemon's listen address into a URL: ":5555" →
|
||||
// "http://localhost:5555"; a full URL is used as is.
|
||||
func baseURL(addr string) string {
|
||||
if addr == "" {
|
||||
addr = ":5555"
|
||||
}
|
||||
if strings.HasPrefix(addr, "http://") || strings.HasPrefix(addr, "https://") {
|
||||
return strings.TrimRight(addr, "/")
|
||||
}
|
||||
if strings.HasPrefix(addr, ":") {
|
||||
addr = "localhost" + addr
|
||||
}
|
||||
return "http://" + addr
|
||||
}
|
||||
|
||||
func printTable(w io.Writer, r api.CountersResponse, showIn, showOut bool) {
|
||||
dom, org := "-", "-"
|
||||
if r.Domain != "" {
|
||||
dom = r.Domain
|
||||
}
|
||||
if r.OrgID != nil {
|
||||
org = strconv.FormatInt(*r.OrgID, 10)
|
||||
}
|
||||
fmt.Fprintf(w, "%s date %s (UTC) domain %s org %s\n", r.Query, r.Date, dom, org)
|
||||
|
||||
tw := tabwriter.NewWriter(w, 0, 8, 2, ' ', 0)
|
||||
head := []string{"LEVEL", "ID"}
|
||||
if showIn {
|
||||
head = append(head, "IN", "LIMIT")
|
||||
}
|
||||
if showOut {
|
||||
head = append(head, "OUT", "LIMIT")
|
||||
}
|
||||
fmt.Fprintln(tw, strings.Join(head, "\t"))
|
||||
for _, l := range r.Levels {
|
||||
cols := []string{l.Level, l.ID}
|
||||
if showIn {
|
||||
cols = append(cols, count(l.In, l.InLimit), limit(l.InLimit))
|
||||
}
|
||||
if showOut {
|
||||
cols = append(cols, count(l.Out, l.OutLimit), limit(l.OutLimit))
|
||||
}
|
||||
fmt.Fprintln(tw, strings.Join(cols, "\t"))
|
||||
}
|
||||
tw.Flush()
|
||||
|
||||
if !r.Managed {
|
||||
fmt.Fprintln(w, "note: domain is not in rspamd_domains_db (not counted)")
|
||||
}
|
||||
if r.Exempt {
|
||||
fmt.Fprintln(w, "note: address is exempt from counting")
|
||||
}
|
||||
}
|
||||
|
||||
// count renders a value, marked with "!" once it reached its limit.
|
||||
func count(n int64, lim *int64) string {
|
||||
s := strconv.FormatInt(n, 10)
|
||||
if lim != nil && n >= *lim {
|
||||
s += "!"
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func limit(lim *int64) string {
|
||||
if lim == nil {
|
||||
return "-"
|
||||
}
|
||||
return strconv.FormatInt(*lim, 10)
|
||||
}
|
||||
@@ -0,0 +1,335 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"migadu_counters/internal/api"
|
||||
)
|
||||
|
||||
var now = time.Date(2026, 10, 13, 12, 0, 0, 0, time.UTC)
|
||||
|
||||
func i64(n int64) *int64 { return &n }
|
||||
|
||||
func sampleResponse() api.CountersResponse {
|
||||
return api.CountersResponse{
|
||||
Query: "me@example.com", Mode: "address", Date: "2026-10-13", Managed: true,
|
||||
Domain: "example.com", OrgID: i64(1234),
|
||||
Levels: []api.CountersLevel{
|
||||
{Level: "A", ID: "me@example.com", In: 12, InLimit: i64(500), Out: 200, OutLimit: i64(200)},
|
||||
{Level: "D", ID: "example.com", In: 40, InLimit: i64(1000), Out: 9, OutLimit: nil},
|
||||
{Level: "O", ID: "1234", In: 120, InLimit: i64(5000), Out: 30, OutLimit: i64(1000)},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// fakeAPI serves GET /counters and records what it was asked.
|
||||
type fakeAPI struct {
|
||||
mu sync.Mutex
|
||||
requests []*http.Request
|
||||
status int
|
||||
body any
|
||||
}
|
||||
|
||||
func (f *fakeAPI) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
f.mu.Lock()
|
||||
f.requests = append(f.requests, r)
|
||||
f.mu.Unlock()
|
||||
if f.status == 0 {
|
||||
f.status = 200
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(f.status)
|
||||
_ = json.NewEncoder(w).Encode(f.body)
|
||||
}
|
||||
|
||||
func (f *fakeAPI) last() *http.Request {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return f.requests[len(f.requests)-1]
|
||||
}
|
||||
|
||||
func writeConfig(t *testing.T, body string) string {
|
||||
t.Helper()
|
||||
p := filepath.Join(t.TempDir(), "config.toml")
|
||||
if err := os.WriteFile(p, []byte(body), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
func noEnv(string) string { return "" }
|
||||
|
||||
func runCLI(t *testing.T, env func(string) string, args ...string) (int, string, string) {
|
||||
t.Helper()
|
||||
var out, errb bytes.Buffer
|
||||
code := run(args, &out, &errb, env, func() time.Time { return now })
|
||||
return code, out.String(), errb.String()
|
||||
}
|
||||
|
||||
func TestRun_Table(t *testing.T) {
|
||||
f := &fakeAPI{body: sampleResponse()}
|
||||
s := httptest.NewServer(f)
|
||||
defer s.Close()
|
||||
cfg := writeConfig(t, "addr = \":8095\"\nauth_token = \"filetoken\"\n")
|
||||
|
||||
code, out, errOut := runCLI(t, noEnv, "--config", cfg, "--url", s.URL, "--address", "Me@Example.com")
|
||||
if code != 0 {
|
||||
t.Fatalf("exit %d, stderr %q", code, errOut)
|
||||
}
|
||||
req := f.last()
|
||||
if req.URL.Path != "/counters" || req.URL.Query().Get("address") != "me@example.com" || req.URL.Query().Has("date") || req.URL.Query().Has("domain") {
|
||||
t.Errorf("request = %s", req.URL)
|
||||
}
|
||||
if req.Header.Get("Authorization") != "Bearer filetoken" {
|
||||
t.Errorf("Authorization = %q", req.Header.Get("Authorization"))
|
||||
}
|
||||
|
||||
lines := strings.Split(strings.TrimSpace(out), "\n")
|
||||
if len(lines) != 5 {
|
||||
t.Fatalf("want header + column line + 3 rows, got %d lines:\n%s", len(lines), out)
|
||||
}
|
||||
for _, want := range []string{"me@example.com", "date 2026-10-13 (UTC)", "domain example.com", "org 1234"} {
|
||||
if !strings.Contains(lines[0], want) {
|
||||
t.Errorf("header %q lacks %q", lines[0], want)
|
||||
}
|
||||
}
|
||||
if cols := strings.Fields(lines[1]); strings.Join(cols, " ") != "LEVEL ID IN LIMIT OUT LIMIT" {
|
||||
t.Errorf("column line = %q", lines[1])
|
||||
}
|
||||
rows := [][]string{strings.Fields(lines[2]), strings.Fields(lines[3]), strings.Fields(lines[4])}
|
||||
want := [][]string{
|
||||
{"A", "me@example.com", "12", "500", "200!", "200"}, // out reached its limit
|
||||
{"D", "example.com", "40", "1000", "9", "-"}, // unlimited out
|
||||
{"O", "1234", "120", "5000", "30", "1000"},
|
||||
}
|
||||
for i := range want {
|
||||
if strings.Join(rows[i], " ") != strings.Join(want[i], " ") {
|
||||
t.Errorf("row %d = %v, want %v", i, rows[i], want[i])
|
||||
}
|
||||
}
|
||||
if errOut != "" {
|
||||
t.Errorf("unexpected stderr: %q", errOut)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRun_UnmanagedAndExemptNotes(t *testing.T) {
|
||||
f := &fakeAPI{body: api.CountersResponse{Query: "x@gmail.com", Mode: "address", Date: "2026-10-13",
|
||||
Levels: []api.CountersLevel{{Level: "A", ID: "x@gmail.com"}, {Level: "D", ID: "gmail.com"}}}}
|
||||
s := httptest.NewServer(f)
|
||||
defer s.Close()
|
||||
cfg := writeConfig(t, "auth_token = \"t\"\n")
|
||||
code, out, _ := runCLI(t, noEnv, "--config", cfg, "--url", s.URL, "--address", "x@gmail.com")
|
||||
if code != 0 || !strings.Contains(out, "not in rspamd_domains_db") || !strings.Contains(out, "org -") {
|
||||
t.Errorf("exit %d output:\n%s", code, out)
|
||||
}
|
||||
|
||||
f.body = api.CountersResponse{Query: "support@migadu.com", Mode: "address", Date: "2026-10-13", Managed: true, Exempt: true,
|
||||
Domain: "migadu.com", OrgID: i64(3743), Levels: []api.CountersLevel{}}
|
||||
code, out, _ = runCLI(t, noEnv, "--config", cfg, "--url", s.URL, "--address", "support@migadu.com")
|
||||
if code != 0 || !strings.Contains(out, "exempt from counting") {
|
||||
t.Errorf("exit %d output:\n%s", code, out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRun_JSON(t *testing.T) {
|
||||
f := &fakeAPI{body: sampleResponse()}
|
||||
s := httptest.NewServer(f)
|
||||
defer s.Close()
|
||||
cfg := writeConfig(t, "auth_token = \"t\"\n")
|
||||
code, out, _ := runCLI(t, noEnv, "--config", cfg, "--url", s.URL, "--address", "me@example.com", "--json")
|
||||
if code != 0 {
|
||||
t.Fatalf("exit %d", code)
|
||||
}
|
||||
var got api.CountersResponse
|
||||
if err := json.Unmarshal([]byte(out), &got); err != nil {
|
||||
t.Fatalf("output is not the API JSON: %v\n%s", err, out)
|
||||
}
|
||||
if got.Query != "me@example.com" || len(got.Levels) != 3 || *got.Levels[0].InLimit != 500 {
|
||||
t.Errorf("decoded = %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRun_DomainMode(t *testing.T) {
|
||||
f := &fakeAPI{body: api.CountersResponse{Query: "example.com", Mode: "domain", Date: "2026-10-13", Managed: true,
|
||||
Domain: "example.com", OrgID: i64(1234),
|
||||
Levels: []api.CountersLevel{{Level: "D", ID: "example.com", In: 1, InLimit: i64(5), Out: 2, OutLimit: i64(6)}, {Level: "O", ID: "1234"}}}}
|
||||
s := httptest.NewServer(f)
|
||||
defer s.Close()
|
||||
cfg := writeConfig(t, "auth_token = \"t\"\n")
|
||||
code, out, _ := runCLI(t, noEnv, "--config", cfg, "--url", s.URL, "--domain", "Example.COM")
|
||||
if code != 0 {
|
||||
t.Fatalf("exit %d\n%s", code, out)
|
||||
}
|
||||
if q := f.last().URL.Query(); q.Get("domain") != "example.com" || q.Has("address") {
|
||||
t.Errorf("query = %v", q)
|
||||
}
|
||||
if !strings.Contains(out, "D example.com") && !strings.Contains(out, "D example.com") {
|
||||
t.Errorf("table lacks the D row:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRun_Date(t *testing.T) {
|
||||
f := &fakeAPI{body: sampleResponse()}
|
||||
s := httptest.NewServer(f)
|
||||
defer s.Close()
|
||||
cfg := writeConfig(t, "auth_token = \"t\"\n")
|
||||
|
||||
code, _, errOut := runCLI(t, noEnv, "--config", cfg, "--url", s.URL, "--address", "me@example.com", "--date", "2026-10-01")
|
||||
if code != 0 || f.last().URL.Query().Get("date") != "2026-10-01" {
|
||||
t.Errorf("exit %d stderr %q query %v", code, errOut, f.last().URL.Query())
|
||||
}
|
||||
if errOut != "" {
|
||||
t.Errorf("12 days back should not warn: %q", errOut)
|
||||
}
|
||||
// Older than the 31-day key TTL: still queried, but warned about.
|
||||
code, _, errOut = runCLI(t, noEnv, "--config", cfg, "--url", s.URL, "--address", "me@example.com", "--date", "2026-09-01")
|
||||
if code != 0 || !strings.Contains(errOut, "31 days") {
|
||||
t.Errorf("exit %d stderr %q, want a 31-day warning", code, errOut)
|
||||
}
|
||||
for _, bad := range []struct{ date, want string }{{"2099-01-01", "future"}, {"13.10.2026", "YYYY-MM-DD"}, {"2026-1-5", "YYYY-MM-DD"}} {
|
||||
before := len(f.requests)
|
||||
code, _, errOut = runCLI(t, noEnv, "--config", cfg, "--url", s.URL, "--address", "me@example.com", "--date", bad.date)
|
||||
if code != 1 || !strings.Contains(errOut, bad.want) {
|
||||
t.Errorf("--date %s: exit %d stderr %q, want exit 1 mentioning %q", bad.date, code, errOut, bad.want)
|
||||
}
|
||||
if len(f.requests) != before {
|
||||
t.Errorf("--date %s: request was sent despite the invalid date", bad.date)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRun_Usage(t *testing.T) {
|
||||
cfg := writeConfig(t, "auth_token = \"t\"\n")
|
||||
for _, args := range [][]string{
|
||||
{"--config", cfg},
|
||||
{"--config", cfg, "--address", "a@b.c", "--domain", "b.c"},
|
||||
{"--config", cfg, "--address", "not-an-address"},
|
||||
{"--config", cfg, "--bogus"},
|
||||
} {
|
||||
code, _, errOut := runCLI(t, noEnv, args...)
|
||||
if code != 1 || errOut == "" {
|
||||
t.Errorf("%v: exit %d stderr %q, want 1 with a message", args, code, errOut)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRun_ServerErrors(t *testing.T) {
|
||||
f := &fakeAPI{status: 401, body: api.ErrorResponse{Error: "unauthorized"}}
|
||||
s := httptest.NewServer(f)
|
||||
defer s.Close()
|
||||
cfg := writeConfig(t, "auth_token = \"wrong\"\n")
|
||||
code, _, errOut := runCLI(t, noEnv, "--config", cfg, "--url", s.URL, "--address", "me@example.com")
|
||||
if code != 1 || !strings.Contains(errOut, "401") || !strings.Contains(errOut, "unauthorized") {
|
||||
t.Errorf("exit %d stderr %q", code, errOut)
|
||||
}
|
||||
code, _, errOut = runCLI(t, noEnv, "--config", cfg, "--url", "http://127.0.0.1:1", "--address", "me@example.com")
|
||||
if code != 1 || errOut == "" {
|
||||
t.Errorf("unreachable: exit %d stderr %q", code, errOut)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRun_ConfigAndEnv(t *testing.T) {
|
||||
f := &fakeAPI{body: sampleResponse()}
|
||||
s := httptest.NewServer(f)
|
||||
defer s.Close()
|
||||
cfg := writeConfig(t, "addr = \":8095\"\nauth_token = \"filetoken\"\n")
|
||||
env := func(k string) string {
|
||||
if k == "MIGADU_COUNTERS_AUTH_TOKEN" {
|
||||
return "envtoken"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
if code, _, errOut := runCLI(t, env, "--config", cfg, "--url", s.URL, "--address", "me@example.com"); code != 0 {
|
||||
t.Fatalf("exit %d %q", code, errOut)
|
||||
}
|
||||
if got := f.last().Header.Get("Authorization"); got != "Bearer envtoken" {
|
||||
t.Errorf("env token not used: %q", got)
|
||||
}
|
||||
// Missing config file is an error naming the path.
|
||||
if code, _, errOut := runCLI(t, noEnv, "--config", "/nonexistent/x.toml", "--address", "me@example.com"); code != 1 || !strings.Contains(errOut, "/nonexistent/x.toml") {
|
||||
t.Errorf("exit %d stderr %q", code, errOut)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBaseURL(t *testing.T) {
|
||||
for in, want := range map[string]string{
|
||||
":5555": "http://localhost:5555",
|
||||
"10.0.0.5:5555": "http://10.0.0.5:5555",
|
||||
"localhost:1": "http://localhost:1",
|
||||
"http://x:1": "http://x:1",
|
||||
"https://x.example/": "https://x.example",
|
||||
"": "http://localhost:5555",
|
||||
} {
|
||||
if got := baseURL(in); got != want {
|
||||
t.Errorf("baseURL(%q) = %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// --in / --out restrict the table to one flow; both or neither give the full table.
|
||||
func TestRun_FlowFlags(t *testing.T) {
|
||||
f := &fakeAPI{body: sampleResponse()}
|
||||
s := httptest.NewServer(f)
|
||||
defer s.Close()
|
||||
cfg := writeConfig(t, "auth_token = \"t\"\n")
|
||||
base := []string{"--config", cfg, "--url", s.URL, "--address", "me@example.com"}
|
||||
|
||||
code, out, errOut := runCLI(t, noEnv, append(base, "--in")...)
|
||||
if code != 0 || errOut != "" {
|
||||
t.Fatalf("--in: exit %d stderr %q", code, errOut)
|
||||
}
|
||||
lines := strings.Split(strings.TrimSpace(out), "\n")
|
||||
if len(lines) != 5 || strings.Join(strings.Fields(lines[1]), " ") != "LEVEL ID IN LIMIT" {
|
||||
t.Errorf("--in table:\n%s", out)
|
||||
}
|
||||
wantIn := [][]string{{"A", "me@example.com", "12", "500"}, {"D", "example.com", "40", "1000"}, {"O", "1234", "120", "5000"}}
|
||||
for i, w := range wantIn {
|
||||
if got := strings.Join(strings.Fields(lines[2+i]), " "); got != strings.Join(w, " ") {
|
||||
t.Errorf("--in row %d = %q, want %v", i, got, w)
|
||||
}
|
||||
}
|
||||
|
||||
code, out, _ = runCLI(t, noEnv, append(base, "--out")...)
|
||||
if code != 0 {
|
||||
t.Fatalf("--out: exit %d", code)
|
||||
}
|
||||
lines = strings.Split(strings.TrimSpace(out), "\n")
|
||||
if len(lines) != 5 || strings.Join(strings.Fields(lines[1]), " ") != "LEVEL ID OUT LIMIT" {
|
||||
t.Errorf("--out table:\n%s", out)
|
||||
}
|
||||
wantOut := [][]string{{"A", "me@example.com", "200!", "200"}, {"D", "example.com", "9", "-"}, {"O", "1234", "30", "1000"}}
|
||||
for i, w := range wantOut {
|
||||
if got := strings.Join(strings.Fields(lines[2+i]), " "); got != strings.Join(w, " ") {
|
||||
t.Errorf("--out row %d = %q, want %v", i, got, w)
|
||||
}
|
||||
}
|
||||
|
||||
_, full, _ := runCLI(t, noEnv, base...)
|
||||
_, both, _ := runCLI(t, noEnv, append(base, "--in", "--out")...)
|
||||
if both != full {
|
||||
t.Errorf("--in --out should equal the full table:\n%s\nvs\n%s", both, full)
|
||||
}
|
||||
|
||||
// --json is the raw response whatever the flow flags say.
|
||||
code, out, _ = runCLI(t, noEnv, append(base, "--out", "--json")...)
|
||||
var got api.CountersResponse
|
||||
if code != 0 || json.Unmarshal([]byte(out), &got) != nil || len(got.Levels) != 3 || got.Levels[0].In != 12 {
|
||||
t.Errorf("--out --json: exit %d output %s", code, out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRun_Version(t *testing.T) {
|
||||
code, out, _ := runCLI(t, noEnv, "--version")
|
||||
if code != 0 || !strings.HasPrefix(out, "counterctl ") {
|
||||
t.Errorf("exit %d output %q", code, out)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,349 @@
|
||||
// migadu_counters counts mail per address / domain / organisation in Redis and
|
||||
// notifies the web app when a daily limit is reached. mailqueuer calls it
|
||||
// fire-and-forget once per ingested recipient (POST /in, POST /out).
|
||||
//
|
||||
// It is a foreground process: on FreeBSD it runs under rc.d + daemon(8)
|
||||
// (pidfile, log capture, restart). SIGHUP reopens the log files and reloads
|
||||
// the domains CDB; SIGTERM/SIGINT drain and exit.
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"github.com/prometheus/client_golang/prometheus/collectors"
|
||||
|
||||
"migadu_counters/internal/config"
|
||||
"migadu_counters/internal/counting"
|
||||
"migadu_counters/internal/domains"
|
||||
"migadu_counters/internal/metrics"
|
||||
"migadu_counters/internal/notify"
|
||||
"migadu_counters/internal/server"
|
||||
"migadu_counters/internal/store"
|
||||
"migadu_counters/internal/txlog"
|
||||
"migadu_counters/internal/watcher"
|
||||
)
|
||||
|
||||
// Set by -ldflags at build time.
|
||||
var (
|
||||
version = "dev"
|
||||
commit = "none"
|
||||
date = "unknown"
|
||||
)
|
||||
|
||||
func main() {
|
||||
os.Exit(run(os.Args[1:], os.Stdout, os.Stderr, nil))
|
||||
}
|
||||
|
||||
// run parses flags, starts the daemon and serves until SIGTERM/SIGINT. sigs
|
||||
// may be nil (then the real process signals are used); tests inject one.
|
||||
func run(args []string, stdout, stderr io.Writer, sigs <-chan os.Signal) int {
|
||||
fs := flag.NewFlagSet("migadu_counters", flag.ContinueOnError)
|
||||
fs.SetOutput(stderr)
|
||||
configPath := fs.String("config", "", "path to config file (TOML)")
|
||||
showVersion := fs.Bool("version", false, "print version and exit")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return 1
|
||||
}
|
||||
if *showVersion {
|
||||
fmt.Fprintf(stdout, "migadu_counters %s (commit %s, built %s)\n", version, commit, date)
|
||||
return 0
|
||||
}
|
||||
|
||||
path, err := config.Discover(*configPath)
|
||||
if err != nil {
|
||||
fmt.Fprintln(stderr, err)
|
||||
return 1
|
||||
}
|
||||
cfg, err := config.Load(path)
|
||||
if err != nil {
|
||||
fmt.Fprintln(stderr, err)
|
||||
return 1
|
||||
}
|
||||
a, err := newApp(cfg)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "startup failed: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
a.logger.Info("migadu_counters starting", "version", version, "config", path)
|
||||
|
||||
apiLn, err := net.Listen("tcp", cfg.Addr)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "listen %s: %v\n", cfg.Addr, err)
|
||||
return 1
|
||||
}
|
||||
adminLn, err := net.Listen("tcp", cfg.AdminAddr)
|
||||
if err != nil {
|
||||
apiLn.Close()
|
||||
fmt.Fprintf(stderr, "listen %s: %v\n", cfg.AdminAddr, err)
|
||||
return 1
|
||||
}
|
||||
a.Serve(apiLn, adminLn)
|
||||
|
||||
if sigs == nil {
|
||||
ch := make(chan os.Signal, 2)
|
||||
signal.Notify(ch, syscall.SIGHUP, syscall.SIGTERM, syscall.SIGINT)
|
||||
defer signal.Stop(ch)
|
||||
sigs = ch
|
||||
}
|
||||
for {
|
||||
select {
|
||||
case sig := <-sigs:
|
||||
if sig == syscall.SIGHUP {
|
||||
a.logger.Info("SIGHUP received: reopening logs and reloading domains db")
|
||||
if err := a.Reload(); err != nil {
|
||||
a.logger.Warn("reload failed", "error", err)
|
||||
}
|
||||
continue
|
||||
}
|
||||
a.logger.Info("shutting down", "signal", sig.String())
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
err := a.Shutdown(ctx)
|
||||
cancel()
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "shutdown: %v\n", err)
|
||||
}
|
||||
return 0
|
||||
case err := <-a.fatal:
|
||||
a.logger.Error("fatal server error", "error", err)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
_ = a.Shutdown(ctx)
|
||||
cancel()
|
||||
return 1
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// app is the assembled daemon.
|
||||
type app struct {
|
||||
cfg *config.Config
|
||||
logger *slog.Logger
|
||||
logWriter *config.LogWriter
|
||||
txWriter *config.LogWriter // nil when the counting log shares the app log or is off
|
||||
|
||||
db atomic.Pointer[domains.DB]
|
||||
st *store.Store
|
||||
notifier *notify.Notifier
|
||||
reg *prometheus.Registry
|
||||
m *metrics.Metrics
|
||||
srv *server.Server
|
||||
|
||||
apiSrv *http.Server
|
||||
adminSrv *http.Server
|
||||
w *watcher.Watcher
|
||||
fatal chan error
|
||||
reloadMu sync.Mutex
|
||||
}
|
||||
|
||||
// newApp opens everything the daemon needs. A missing CDB is fatal (nothing
|
||||
// could be counted); an unreachable Redis is only a warning (the service must
|
||||
// come up while Redis is briefly away).
|
||||
func newApp(cfg *config.Config) (*app, error) {
|
||||
logger, logWriter, err := config.NewLogger(cfg.Logging)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, w := range cfg.Warnings() {
|
||||
logger.Warn(w)
|
||||
}
|
||||
a := &app{cfg: cfg, logger: logger, logWriter: logWriter, fatal: make(chan error, 2)}
|
||||
|
||||
db, err := domains.OpenWithCache(cfg.DomainsCDB, cfg.MaxCachedDomains)
|
||||
if err != nil {
|
||||
logWriter.Close()
|
||||
return nil, err
|
||||
}
|
||||
a.db.Store(db)
|
||||
logger.Info("domains db loaded", "path", cfg.DomainsCDB, "domains", db.Count(), "max_cached_domains", cfg.MaxCachedDomains)
|
||||
|
||||
a.reg = prometheus.NewRegistry()
|
||||
a.reg.MustRegister(collectors.NewGoCollector(), collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}))
|
||||
a.m = metrics.New(a.reg)
|
||||
a.m.SetCDBDomains(db.Count())
|
||||
|
||||
a.st, err = store.New(cfg.Redis)
|
||||
if err != nil {
|
||||
db.Close()
|
||||
logWriter.Close()
|
||||
return nil, err
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
|
||||
if err := a.st.Ping(ctx); err != nil {
|
||||
logger.Warn("redis not reachable at startup; commands will be retried on demand", "error", err)
|
||||
} else {
|
||||
mode := "static"
|
||||
if cfg.Redis.SentinelMode() {
|
||||
mode = "sentinel " + cfg.Redis.MasterName
|
||||
}
|
||||
logger.Info("redis reachable", "mode", mode)
|
||||
}
|
||||
cancel()
|
||||
|
||||
var tx *txlog.Logger
|
||||
if cfg.Logging.CountingEnabled() {
|
||||
w := logWriter
|
||||
if cfg.Logging.CountingOutput != cfg.Logging.Output {
|
||||
if a.txWriter, err = config.NewWriter(cfg.Logging.CountingOutput); err != nil {
|
||||
a.st.Close()
|
||||
db.Close()
|
||||
logWriter.Close()
|
||||
return nil, err
|
||||
}
|
||||
w = a.txWriter
|
||||
}
|
||||
tx = txlog.New(w, cfg.Logging.CountingFormat, nil)
|
||||
}
|
||||
|
||||
a.notifier = notify.New(notify.Config{
|
||||
OrgURL: cfg.Notify.OrgURL, AddressURL: cfg.Notify.AddressURL, DomainURL: cfg.Notify.DomainURL,
|
||||
User: cfg.Notify.User, Password: cfg.Notify.Password, Timeout: cfg.Notify.Timeout(),
|
||||
}, server.NotifyHooks(tx, a.m))
|
||||
|
||||
planner := &counting.Planner{
|
||||
Lookup: counting.LookupFunc(func(domain string) (*domains.Domain, error) {
|
||||
db := a.db.Load()
|
||||
if db == nil {
|
||||
return nil, errors.New("domains db not loaded")
|
||||
}
|
||||
return db.Lookup(domain)
|
||||
}),
|
||||
Exempt: cfg.ExemptSet(),
|
||||
CountJunk: cfg.CountJunk,
|
||||
}
|
||||
a.srv = server.New(server.Deps{
|
||||
AuthToken: cfg.AuthToken,
|
||||
AllowedCIDRs: cfg.AllowedPrefixes(),
|
||||
HandlerTimeout: cfg.HandlerTimeoutDuration(),
|
||||
QueueSize: cfg.QueueSize,
|
||||
Workers: cfg.Workers,
|
||||
Exempt: cfg.ExemptSet(),
|
||||
Counter: &counting.Counter{Planner: planner, Store: a.st},
|
||||
Store: a.st,
|
||||
Domains: func() *domains.DB { return a.db.Load() },
|
||||
Notifier: a.notifier,
|
||||
TxLog: tx,
|
||||
Metrics: a.m,
|
||||
Gatherer: a.reg,
|
||||
Logger: logger,
|
||||
})
|
||||
return a, nil
|
||||
}
|
||||
|
||||
// Serve starts the API and admin servers on the listeners and the CDB file
|
||||
// watcher. Listener failures are reported on a.fatal.
|
||||
func (a *app) Serve(apiLn, adminLn net.Listener) {
|
||||
newServer := func(h http.Handler) *http.Server {
|
||||
return &http.Server{
|
||||
Handler: h,
|
||||
ReadHeaderTimeout: time.Duration(a.cfg.ReadHeaderTimeout) * time.Second,
|
||||
WriteTimeout: time.Duration(a.cfg.WriteTimeout) * time.Second,
|
||||
IdleTimeout: time.Duration(a.cfg.IdleTimeout) * time.Second,
|
||||
MaxHeaderBytes: 1 << 20,
|
||||
}
|
||||
}
|
||||
a.apiSrv = newServer(a.srv.Handler())
|
||||
a.adminSrv = newServer(a.srv.AdminHandler())
|
||||
serve := func(name string, s *http.Server, ln net.Listener) {
|
||||
if err := s.Serve(ln); err != nil && !errors.Is(err, http.ErrServerClosed) {
|
||||
a.fatal <- fmt.Errorf("%s server: %w", name, err)
|
||||
}
|
||||
}
|
||||
go serve("api", a.apiSrv, apiLn)
|
||||
go serve("admin", a.adminSrv, adminLn)
|
||||
|
||||
w, err := watcher.New([]string{a.cfg.DomainsCDB}, a.reloadDomains)
|
||||
if err != nil {
|
||||
a.logger.Error("domains db watcher not started; reload with SIGHUP", "error", err)
|
||||
} else {
|
||||
w.Start()
|
||||
a.w = w
|
||||
}
|
||||
a.logger.Info("listening", "addr", apiLn.Addr().String(), "admin_addr", adminLn.Addr().String())
|
||||
}
|
||||
|
||||
// Reload is the SIGHUP action: reopen both log files, then reload the CDB.
|
||||
func (a *app) Reload() error {
|
||||
if err := a.logWriter.Reopen(); err != nil {
|
||||
a.logger.Error("reopen log failed", "error", err)
|
||||
}
|
||||
if a.txWriter != nil {
|
||||
if err := a.txWriter.Reopen(); err != nil {
|
||||
a.logger.Error("reopen counting log failed", "error", err)
|
||||
}
|
||||
}
|
||||
return a.reloadDomains()
|
||||
}
|
||||
|
||||
// reloadDomains swaps in a freshly opened CDB. A file that cannot be opened,
|
||||
// or holds no domains at all, is rejected and the current table kept: a
|
||||
// half-deployed file must not silently stop all counting.
|
||||
func (a *app) reloadDomains() error {
|
||||
a.reloadMu.Lock()
|
||||
defer a.reloadMu.Unlock()
|
||||
db, err := domains.OpenWithCache(a.cfg.DomainsCDB, a.cfg.MaxCachedDomains)
|
||||
if err == nil && db.Count() == 0 {
|
||||
db.Close()
|
||||
err = fmt.Errorf("domains db %s holds no records", a.cfg.DomainsCDB)
|
||||
}
|
||||
if err != nil {
|
||||
a.m.RecordReload(false)
|
||||
a.logger.Warn("domains db reload failed, keeping current table", "error", err)
|
||||
return err
|
||||
}
|
||||
old := a.db.Swap(db)
|
||||
a.m.SetCDBDomains(db.Count())
|
||||
a.m.RecordReload(true)
|
||||
a.logger.Info("domains db reloaded", "path", a.cfg.DomainsCDB, "domains", db.Count())
|
||||
if old != nil {
|
||||
// Grace period for lookups still holding the old table.
|
||||
time.AfterFunc(30*time.Second, func() { old.Close() })
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Shutdown stops the listeners, drains the counting queue (every accepted
|
||||
// request is counted), waits for notifications, then closes Redis and the
|
||||
// log files — in that order, so nothing a worker needs disappears under it.
|
||||
func (a *app) Shutdown(ctx context.Context) error {
|
||||
if a.w != nil {
|
||||
a.w.Stop()
|
||||
}
|
||||
var errs []error
|
||||
if a.apiSrv != nil {
|
||||
errs = append(errs, a.apiSrv.Shutdown(ctx))
|
||||
}
|
||||
if a.adminSrv != nil {
|
||||
errs = append(errs, a.adminSrv.Shutdown(ctx))
|
||||
}
|
||||
if pending := a.srv.Pending(); pending > 0 {
|
||||
a.logger.Info("draining counting queue", "pending", pending)
|
||||
}
|
||||
if err := a.srv.Drain(ctx); err != nil {
|
||||
a.logger.Error("counting queue not fully drained", "error", err)
|
||||
errs = append(errs, err)
|
||||
}
|
||||
errs = append(errs, a.notifier.Shutdown(ctx))
|
||||
errs = append(errs, a.st.Close())
|
||||
if db := a.db.Swap(nil); db != nil {
|
||||
errs = append(errs, db.Close())
|
||||
}
|
||||
a.logger.Info("stopped")
|
||||
if a.txWriter != nil {
|
||||
errs = append(errs, a.txWriter.Close())
|
||||
}
|
||||
errs = append(errs, a.logWriter.Close())
|
||||
return errors.Join(errs...)
|
||||
}
|
||||
@@ -0,0 +1,417 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/alicebob/miniredis/v2"
|
||||
"github.com/colinmarc/cdb"
|
||||
|
||||
"migadu_counters/internal/api"
|
||||
"migadu_counters/internal/config"
|
||||
)
|
||||
|
||||
const realCDB = "../../test_files/rspamd_domains_db.cdb"
|
||||
|
||||
// env is one configured daemon with its own temp dir, miniredis and notify capture.
|
||||
type env struct {
|
||||
t *testing.T
|
||||
dir string
|
||||
cdbPath string
|
||||
cfgPath string
|
||||
logPath string
|
||||
txPath string
|
||||
redis *miniredis.Miniredis
|
||||
notified *capture
|
||||
}
|
||||
|
||||
type capture struct {
|
||||
mu sync.Mutex
|
||||
urls []string
|
||||
}
|
||||
|
||||
func (c *capture) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
c.mu.Lock()
|
||||
c.urls = append(c.urls, r.URL.String())
|
||||
c.mu.Unlock()
|
||||
w.WriteHeader(200)
|
||||
}
|
||||
|
||||
func copyFile(t *testing.T, src, dst string) {
|
||||
t.Helper()
|
||||
in, err := os.Open(src)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer in.Close()
|
||||
out, err := os.Create(dst)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := io.Copy(out, in); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := out.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// writeSmallCDB atomically replaces path with a CDB holding one domain.
|
||||
func writeSmallCDB(t *testing.T, path string) {
|
||||
t.Helper()
|
||||
tmp := path + ".tmp"
|
||||
w, err := cdb.Create(tmp)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w.Put([]byte("newdomain.test"), []byte("5|A/i=1,A/o=1,D/i=2,D/o=2,O/i=3,O/o=3")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Rename(tmp, path); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func newEnv(t *testing.T, extra string) *env {
|
||||
t.Helper()
|
||||
e := &env{t: t, dir: t.TempDir(), redis: miniredis.RunT(t), notified: &capture{}}
|
||||
e.cdbPath = filepath.Join(e.dir, "rspamd_domains_db")
|
||||
copyFile(t, realCDB, e.cdbPath)
|
||||
ns := httptest.NewServer(e.notified)
|
||||
t.Cleanup(ns.Close)
|
||||
e.logPath = filepath.Join(e.dir, "app.log")
|
||||
e.txPath = filepath.Join(e.dir, "counting.log")
|
||||
e.cfgPath = filepath.Join(e.dir, "config.toml")
|
||||
q := "/api/limits?level=$LEVEL&id=$ID&flow=$FLOW&period=$PERIOD&by=$ADDRESS&count=$COUNT"
|
||||
body := fmt.Sprintf(`
|
||||
addr = "127.0.0.1:0"
|
||||
admin_addr = "127.0.0.1:0"
|
||||
auth_token = "t"
|
||||
domains_cdb = %q
|
||||
%s
|
||||
[logging]
|
||||
output = %q
|
||||
counting_output = %q
|
||||
[redis]
|
||||
write_addr = %q
|
||||
pool_size = 4
|
||||
timeout_ms = 500
|
||||
[notify]
|
||||
org_url = %q
|
||||
address_url = %q
|
||||
user = "u"
|
||||
password = "p"
|
||||
timeout_seconds = 2
|
||||
`, e.cdbPath, extra, e.logPath, e.txPath, e.redis.Addr(), ns.URL+q, ns.URL+q)
|
||||
if err := os.WriteFile(e.cfgPath, []byte(body), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return e
|
||||
}
|
||||
|
||||
func (e *env) logs() string {
|
||||
b, _ := os.ReadFile(e.logPath)
|
||||
return string(b)
|
||||
}
|
||||
|
||||
func listen(t *testing.T) net.Listener {
|
||||
t.Helper()
|
||||
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return ln
|
||||
}
|
||||
|
||||
func httpDo(t *testing.T, method, url, body string) (int, string) {
|
||||
t.Helper()
|
||||
req, _ := http.NewRequest(method, url, strings.NewReader(body))
|
||||
req.Header.Set("Authorization", "Bearer t")
|
||||
if body != "" {
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatalf("%s %s: %v", method, url, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, _ := io.ReadAll(resp.Body)
|
||||
return resp.StatusCode, string(b)
|
||||
}
|
||||
|
||||
func counters(t *testing.T, base, address string) api.CountersResponse {
|
||||
t.Helper()
|
||||
st, body := httpDo(t, http.MethodGet, base+"/counters?address="+address, "")
|
||||
if st != 200 {
|
||||
t.Fatalf("/counters %s: %d %s", address, st, body)
|
||||
}
|
||||
var r api.CountersResponse
|
||||
if err := json.Unmarshal([]byte(body), &r); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
func TestApp_EndToEnd(t *testing.T) {
|
||||
e := newEnv(t, "")
|
||||
cfg, err := config.Load(e.cfgPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
a, err := newApp(cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("newApp: %v", err)
|
||||
}
|
||||
apiLn, adminLn := listen(t), listen(t)
|
||||
a.Serve(apiLn, adminLn)
|
||||
base := "http://" + apiLn.Addr().String()
|
||||
admin := "http://" + adminLn.Addr().String()
|
||||
|
||||
// Count through the real stack: accepted immediately, counted by a worker.
|
||||
st, body := httpDo(t, http.MethodPost, base+"/in", `{"trace_id":"e2e","mail_from":"s@gmail.com","recipient":"b@yagolivas.com"}`)
|
||||
if st != 202 || strings.TrimSpace(body) != `{"accepted":true}` {
|
||||
t.Fatalf("/in = %d %s", st, body)
|
||||
}
|
||||
key := "cnt/A/b@yagolivas.com/d/" + time.Now().UTC().Format("2006-01-02") + "/i"
|
||||
for deadline := time.Now().Add(5 * time.Second); !e.redis.Exists(key); {
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("counter key missing; keys: %v", e.redis.Keys())
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
if r := counters(t, base, "b@yagolivas.com"); r.Levels[0].In != 1 || *r.Levels[0].InLimit != 200 {
|
||||
t.Errorf("/counters after /in = %+v", r.Levels[0])
|
||||
}
|
||||
if st, body := httpDo(t, http.MethodGet, admin+"/health", ""); st != 200 {
|
||||
t.Errorf("/health = %d %s", st, body)
|
||||
}
|
||||
if st, body := httpDo(t, http.MethodGet, admin+"/metrics", ""); st != 200 || !strings.Contains(body, "migadu_counters_cdb_domains 93549") {
|
||||
t.Errorf("/metrics = %d (cdb_domains gauge missing)", st)
|
||||
}
|
||||
for deadline := time.Now().Add(5 * time.Second); ; {
|
||||
tx, _ := os.ReadFile(e.txPath)
|
||||
if strings.Contains(string(tx), "<e2e> in; tracked address counters for b@yagolivas.com => count=1 limit=200 reached=false") {
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("counting log:\n%s", tx)
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
if !strings.Contains(e.logs(), "listening") {
|
||||
t.Errorf("app log should announce listening:\n%s", e.logs())
|
||||
}
|
||||
|
||||
// SIGHUP path: the CDB was replaced by a different table.
|
||||
writeSmallCDB(t, e.cdbPath)
|
||||
if err := a.Reload(); err != nil {
|
||||
t.Fatalf("Reload: %v", err)
|
||||
}
|
||||
if r := counters(t, base, "x@newdomain.test"); !r.Managed || r.OrgID == nil || *r.OrgID != 5 {
|
||||
t.Errorf("after reload newdomain.test = %+v", r)
|
||||
}
|
||||
if r := counters(t, base, "a@yagolivas.com"); r.Managed {
|
||||
t.Errorf("after reload yagolivas.com should be unmanaged: %+v", r)
|
||||
}
|
||||
if st, body := httpDo(t, http.MethodGet, admin+"/metrics", ""); !strings.Contains(body, "migadu_counters_cdb_domains 1") || !strings.Contains(body, `migadu_counters_cdb_reloads_total{result="success"} 1`) {
|
||||
t.Errorf("/metrics after reload = %d:\n%s", st, body)
|
||||
}
|
||||
|
||||
// A corrupt replacement (deployed atomically, as the agent's mv does) must
|
||||
// be rejected and the old table kept. Never overwrite the CDB in place:
|
||||
// that truncates the inode the daemon holds open.
|
||||
if err := os.WriteFile(e.cdbPath+".bad", []byte("garbage"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Rename(e.cdbPath+".bad", e.cdbPath); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := a.Reload(); err == nil {
|
||||
t.Error("Reload with a corrupt CDB should fail")
|
||||
}
|
||||
if r := counters(t, base, "x@newdomain.test"); !r.Managed {
|
||||
t.Errorf("old table should survive a failed reload: %+v", r)
|
||||
}
|
||||
|
||||
// Watcher path: an atomic replacement reloads without a signal.
|
||||
tmp := e.cdbPath + ".new"
|
||||
copyFile(t, realCDB, tmp)
|
||||
if err := os.Rename(tmp, e.cdbPath); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
deadline := time.Now().Add(8 * time.Second)
|
||||
for {
|
||||
if r := counters(t, base, "a@yagolivas.com"); r.Managed {
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("watcher did not reload the replaced CDB; log:\n%s", e.logs())
|
||||
}
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
|
||||
// Log rotation: after renaming the app log, Reload reopens it.
|
||||
rotated := e.logPath + ".0"
|
||||
if err := os.Rename(e.logPath, rotated); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := a.Reload(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
httpDo(t, http.MethodPost, base+"/in", `{}`) // a rejection -> warn line
|
||||
if !strings.Contains(e.logs(), "request rejected") {
|
||||
t.Errorf("new log file should receive lines after rotation:\n%s", e.logs())
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
if err := a.Shutdown(ctx); err != nil {
|
||||
t.Fatalf("Shutdown: %v", err)
|
||||
}
|
||||
if _, err := http.Get(base + "/counters"); err == nil {
|
||||
t.Error("API still answering after Shutdown")
|
||||
}
|
||||
if !strings.Contains(e.logs(), "stopped") {
|
||||
t.Errorf("log should record the stop:\n%s", e.logs())
|
||||
}
|
||||
}
|
||||
|
||||
func TestApp_ReachedNotifies(t *testing.T) {
|
||||
e := newEnv(t, "")
|
||||
cfg, _ := config.Load(e.cfgPath)
|
||||
a, err := newApp(cfg)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
apiLn, adminLn := listen(t), listen(t)
|
||||
a.Serve(apiLn, adminLn)
|
||||
base := "http://" + apiLn.Addr().String()
|
||||
for i := 0; i < 20; i++ { // A/o=20 for yagolivas.com
|
||||
if st, body := httpDo(t, http.MethodPost, base+"/out", `{"auth_user":"a@yagolivas.com","recipient":"x@gmail.com"}`); st != 202 {
|
||||
t.Fatalf("/out = %d %s", st, body)
|
||||
}
|
||||
}
|
||||
// Shutdown drains the queue, then the notifier: every notification lands.
|
||||
if err := a.Shutdown(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v, _ := e.redis.Get("cnt/A/a@yagolivas.com/d/" + time.Now().UTC().Format("2006-01-02") + "/o"); v != "20" {
|
||||
t.Errorf("after Shutdown A/o = %q, want 20 (queue drained)", v)
|
||||
}
|
||||
e.notified.mu.Lock()
|
||||
n := len(e.notified.urls)
|
||||
e.notified.mu.Unlock()
|
||||
if n != 2 { // address_url + org_url (domain_url not configured)
|
||||
t.Errorf("notifications = %d, want 2: %v", n, e.notified.urls)
|
||||
}
|
||||
if tx, _ := os.ReadFile(e.txPath); strings.Count(string(tx), "sent limit exceeded notification") != 2 {
|
||||
t.Errorf("counting log should have two sent lines:\n%s", tx)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewApp_MissingCDBIsFatal(t *testing.T) {
|
||||
e := newEnv(t, "")
|
||||
os.Remove(e.cdbPath)
|
||||
cfg, _ := config.Load(e.cfgPath)
|
||||
if _, err := newApp(cfg); err == nil || !strings.Contains(err.Error(), "rspamd_domains_db") {
|
||||
t.Errorf("newApp without CDB: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewApp_RedisDownIsNotFatal(t *testing.T) {
|
||||
e := newEnv(t, "")
|
||||
e.redis.Close()
|
||||
cfg, _ := config.Load(e.cfgPath)
|
||||
a, err := newApp(cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("Redis being down must not prevent startup: %v", err)
|
||||
}
|
||||
apiLn, adminLn := listen(t), listen(t)
|
||||
a.Serve(apiLn, adminLn)
|
||||
defer a.Shutdown(context.Background())
|
||||
if st, body := httpDo(t, http.MethodGet, "http://"+adminLn.Addr().String()+"/health", ""); st != 503 || !strings.Contains(body, "redis") {
|
||||
t.Errorf("/health with redis down = %d %s", st, body)
|
||||
}
|
||||
if !strings.Contains(e.logs(), "redis") {
|
||||
t.Errorf("startup should warn about redis:\n%s", e.logs())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRun_FlagsAndExitCodes(t *testing.T) {
|
||||
var out, errb bytes.Buffer
|
||||
if code := run([]string{"-version"}, &out, &errb, nil); code != 0 || !strings.HasPrefix(out.String(), "migadu_counters ") {
|
||||
t.Errorf("-version: %d %q", code, out.String())
|
||||
}
|
||||
out.Reset()
|
||||
errb.Reset()
|
||||
if code := run([]string{"-config", "/nonexistent/config.toml"}, &out, &errb, nil); code != 1 || !strings.Contains(errb.String(), "/nonexistent/config.toml") {
|
||||
t.Errorf("missing config: %d %q", code, errb.String())
|
||||
}
|
||||
errb.Reset()
|
||||
bad := filepath.Join(t.TempDir(), "bad.toml")
|
||||
os.WriteFile(bad, []byte("domains_cdb = \"x\"\nauth_token = \"t\"\n[redis]\ndb = 1\nwrite_addr = \"127.0.0.1:1\"\n"), 0o600)
|
||||
if code := run([]string{"-config", bad}, &out, &errb, nil); code != 1 || !strings.Contains(errb.String(), "redis.db") {
|
||||
t.Errorf("invalid config: %d %q", code, errb.String())
|
||||
}
|
||||
// Review #6: an empty / mistyped token never starts the daemon.
|
||||
for _, body := range []string{"domains_cdb = \"x\"\n[redis]\nwrite_addr = \"127.0.0.1:1\"\n", "domains_cdb = \"x\"\nauth-token = \"t\"\n[redis]\nwrite_addr = \"127.0.0.1:1\"\n"} {
|
||||
errb.Reset()
|
||||
os.WriteFile(bad, []byte(body), 0o600)
|
||||
if code := run([]string{"-config", bad}, &out, &errb, nil); code != 1 || !strings.Contains(errb.String(), "auth") {
|
||||
t.Errorf("config %q: exit %d stderr %q, want 1 mentioning the token", body, code, errb.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// run() serves until SIGTERM; SIGHUP in between reloads.
|
||||
func TestRun_SignalLoop(t *testing.T) {
|
||||
e := newEnv(t, "")
|
||||
sigs := make(chan os.Signal, 2)
|
||||
done := make(chan int, 1)
|
||||
var out, errb bytes.Buffer
|
||||
go func() { done <- run([]string{"-config", e.cfgPath}, &out, &errb, sigs) }()
|
||||
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for !strings.Contains(e.logs(), "listening") {
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("daemon did not start; stderr %q log:\n%s", errb.String(), e.logs())
|
||||
}
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
}
|
||||
sigs <- syscall.SIGHUP
|
||||
deadline = time.Now().Add(5 * time.Second)
|
||||
for !strings.Contains(e.logs(), "reloaded") {
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("SIGHUP did not reload; log:\n%s", e.logs())
|
||||
}
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
}
|
||||
sigs <- syscall.SIGTERM
|
||||
select {
|
||||
case code := <-done:
|
||||
if code != 0 {
|
||||
t.Errorf("exit code %d, stderr %q", code, errb.String())
|
||||
}
|
||||
case <-time.After(15 * time.Second):
|
||||
t.Fatal("run did not return after SIGTERM")
|
||||
}
|
||||
if !strings.Contains(e.logs(), "stopped") {
|
||||
t.Errorf("log:\n%s", e.logs())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
# migadu_counters configuration
|
||||
# Install as /usr/local/etc/migadu_counters/config.toml (or pass -config PATH).
|
||||
|
||||
addr = ":5555" # API: POST /in, POST /out, GET /counters
|
||||
admin_addr = ":9095" # /health and /metrics, no auth
|
||||
auth_token = "…" # REQUIRED (or MIGADU_COUNTERS_AUTH_TOKEN); same value in mailqueuer's config
|
||||
allowed_cidrs = [] # optional source-IP allowlist for the API, e.g. ["10.0.0.0/8"]
|
||||
domains_cdb = "/usr/local/etc/rspamd/db/rspamd_domains_db"
|
||||
exempt = ["support@migadu.com", "contact@migadu.com"] # mail to/from these never counts
|
||||
count_junk = false # count /in requests flagged junk by mizu
|
||||
|
||||
# Counting runs off the request path: /in and /out validate, enqueue and
|
||||
# answer 202; workers drain the queue. A full queue answers 503 (dropped_total).
|
||||
queue_size = 10000
|
||||
workers = 8
|
||||
# Parsed CDB records cached per loaded table (some domains carry thousands of
|
||||
# per-mailbox overrides); 0 disables the cache.
|
||||
max_cached_domains = 20000
|
||||
|
||||
# HTTP server timeouts (seconds); handler_timeout bounds the synchronous part
|
||||
# of a request and each queued counting job.
|
||||
read_header_timeout = 5
|
||||
write_timeout = 30
|
||||
idle_timeout = 60
|
||||
handler_timeout = 4
|
||||
|
||||
[logging]
|
||||
output = "stderr" # daemon(8) -o captures it; or a file path (reopened on SIGHUP)
|
||||
format = "console" # or "json"
|
||||
level = "info"
|
||||
counting_output = "/var/log/migadu_counters/counting.log" # default: same as output; "off" disables
|
||||
counting_format = "text" # Lua-like lines (default) or "json"
|
||||
|
||||
[redis]
|
||||
db = 0 # must stay 0: migadu_web reads message counters from DB 0
|
||||
password = "…" # or MIGADU_COUNTERS_REDIS_PASSWORD
|
||||
pool_size = 32
|
||||
timeout_ms = 500
|
||||
# Sentinel mode (production): writes -> master, reads -> replicas
|
||||
master_name = "mymaster"
|
||||
sentinels = ["10.0.0.10:26379", "10.0.0.11:26379", "10.0.0.12:26379"]
|
||||
sentinel_password = ""
|
||||
# Static mode (used when sentinels is empty)
|
||||
# write_addr = "127.0.0.1:6379"
|
||||
# read_addrs = ["127.0.0.1:6379"]
|
||||
|
||||
[notify]
|
||||
# One URL per level; leave empty to disable notifications for that level.
|
||||
# Placeholders: $LEVEL $ID $FLOW $PERIOD $ADDRESS $COUNT $LIMIT (all URL-escaped).
|
||||
org_url = "https://admin.migadu.com/api/limits?level=$LEVEL&id=$ID&flow=$FLOW&period=$PERIOD&by=$ADDRESS&count=$COUNT"
|
||||
address_url = "https://admin.migadu.com/api/limits?level=$LEVEL&id=$ID&flow=$FLOW&period=$PERIOD&by=$ADDRESS&count=$COUNT"
|
||||
domain_url = "" # web app's D branch is still a TODO
|
||||
user = "…"
|
||||
password = "…" # or MIGADU_COUNTERS_NOTIFY_PASSWORD
|
||||
timeout_seconds = 5
|
||||
@@ -0,0 +1,43 @@
|
||||
# contrib: getting the domains CDB onto the service host
|
||||
|
||||
migadu_counters reads limits from the rspamd domains CDB that the agent builds
|
||||
(`extract.sh` → `/usr/local/etc/agent/build/rspamd/rspamd_domains_db`). The
|
||||
files here fetch it the way rcptd fetches its maps (`ansible-freebsd3`
|
||||
`roles/rcptd/templates/{build.sh.j2,rcptd-maps.cron.j2}`); the future Ansible
|
||||
role templates them.
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `pull-domains.sh` | rsync the CDB from the agent, `cmp` against the installed `.orig`, and only on change verify (`cdb -s`) and install it with an atomic `mv`; `.pull-lock` with a 120 s max age; log to `/var/log/migadu_counters/pull.log` via `tee -a`. |
|
||||
| `migadu_counters-pull.cron` | FreeBSD `@15` crontab fragment (15 s after the previous run completes) wrapped in `timeout -k 30 300`. |
|
||||
| `pull-domains_test.sh` | Test against a local source directory (`RSYNC_SRC` override): unchanged → no `mv`, no reload; changed → atomic install, `.orig` updated, reload; corrupt or empty CDB → not installed, non-zero exit, old file untouched; lock handling. Run with `make test-contrib` (needs tinycdb's `cdb` and `rsync`). |
|
||||
|
||||
## What the Ansible role must template
|
||||
|
||||
Environment of `pull-domains.sh` (every variable has the production default
|
||||
shown in the script):
|
||||
|
||||
- `AGENT_HOST` (`agent@<agent host>`) and `AGENT_PATH`; ssh is forced to IPv4
|
||||
(`RSYNC_RSH="ssh -4"`), the agent's host key must be accepted beforehand
|
||||
(rcptd has `accept-agent-hostkey.sh.j2` for that).
|
||||
- `MAPS_DIR` (pulled copy), `DOMAINS_DIR` / `DOMAINS_CDB` (what `domains_cdb`
|
||||
in `config.toml` points at), `LOG_FILE`, `LOCK_FILE`.
|
||||
- `RELOAD_CMD` (default `service migadu_counters reload`, i.e. SIGHUP). The
|
||||
daemon's fsnotify watcher already reloads on the `mv`; the reload is belt
|
||||
and braces.
|
||||
- The cron user (`migadu_counters`) must own `MAPS_DIR`, `DOMAINS_DIR` and the
|
||||
log directory, and have the ssh key for the agent host. `timeout`, `rsync`
|
||||
and `cdb` (tinycdb) must be installed.
|
||||
|
||||
## Rules the daemon relies on
|
||||
|
||||
- **Never rewrite the CDB in place.** The daemon keeps the file open; an
|
||||
in-place rewrite truncates the inode under it and breaks every lookup until
|
||||
the next reload. The script only ever `mv`s a verified dotfile over the
|
||||
target, which is atomic on the same filesystem (`DOMAINS_DIR` must hold both).
|
||||
- A file that does not open as a CDB or holds no records is refused and the
|
||||
installed file kept; the daemon applies the same rule on reload.
|
||||
- Memory stays flat across the 15-second cycles: a reload replaces the whole
|
||||
in-memory DB object (2 KB index plus a bounded parse cache,
|
||||
`max_cached_domains`, default 20 000 records) and the old one is closed after
|
||||
a 30 s grace period. Unchanged pulls are a `cmp` and nothing else.
|
||||
@@ -0,0 +1,23 @@
|
||||
# migadu_counters: pull the rspamd domains CDB from the agent host.
|
||||
# Install as /etc/cron.d/migadu_counters-pull (system crontab fragment:
|
||||
# five time/date fields OR an @ spec, then the user, then the command).
|
||||
# The Ansible role templates the user and the script path.
|
||||
#
|
||||
# @15 reruns pull-domains.sh 15 seconds after the PREVIOUS run COMPLETES (a
|
||||
# FreeBSD cron extension), not every 15s of wall clock — the same arrangement
|
||||
# as rcptd's maps job:
|
||||
# - cron cannot overlap this job with itself, so no flock wrapper is needed;
|
||||
# the script's own .pull-lock still guards a manual or ansible-triggered
|
||||
# run racing the scheduled one.
|
||||
# - start times drift with run duration instead of landing on
|
||||
# :00/:15/:30/:45 simultaneously across the fleet, which is what bursts
|
||||
# ssh connections at the agent host (pf max-src-conn-rate, MaxStartups).
|
||||
#
|
||||
# timeout is what guarantees the schedule resumes: cron waits for completion
|
||||
# before counting the 15s, so a hung run (ssh stall) would otherwise stall
|
||||
# this job forever with nothing retrying. The budget only has to exceed the
|
||||
# worst legitimate run, since overlap is no longer possible.
|
||||
#
|
||||
# pull-domains.sh appends to pull.log itself via tee -a, so only stderr is
|
||||
# captured here; redirecting stdout to that same file would truncate it.
|
||||
@15 migadu_counters /usr/bin/timeout -k 30 300 /usr/local/etc/migadu_counters/pull-domains.sh >/dev/null 2>>/var/log/migadu_counters/pull.log
|
||||
Executable
+93
@@ -0,0 +1,93 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# pull-domains.sh — fetch the rspamd domains CDB from the agent host and
|
||||
# install it for migadu_counters, the way rcptd's build.sh pulls its maps.
|
||||
#
|
||||
# The agent already produces the finished CDB (extract.sh: cdb -m ->
|
||||
# /usr/local/etc/agent/build/rspamd/rspamd_domains_db), so this is
|
||||
# pull → compare → install:
|
||||
#
|
||||
# 1. rsync the file from the agent into $MAPS_DIR (ssh forced to IPv4, see
|
||||
# rcptd: agenthost also has an IPv6 address not reachable from every host).
|
||||
# 2. cmp against the installed baseline $DOMAINS_CDB.orig; identical → done.
|
||||
# Most runs end here: the agent only rewrites the file when its content
|
||||
# changed, so the 15-second cron cycle is a no-op cmp.
|
||||
# 3. Changed: copy to a dotfile next to the target, verify it opens and has
|
||||
# records (cdb -s), then mv it over $DOMAINS_CDB — atomic, so the daemon
|
||||
# never sees a half-written file — and keep the pulled copy as .orig.
|
||||
# The daemon's fsnotify watcher reloads on the mv; the explicit reload is
|
||||
# belt and braces.
|
||||
#
|
||||
# Never rewrite $DOMAINS_CDB in place: the daemon keeps the file open and an
|
||||
# in-place rewrite truncates the inode under it. This script only ever mv's.
|
||||
#
|
||||
# Environment (templated by the Ansible role; defaults below are the
|
||||
# production paths). Tests override RSYNC_SRC with a local path.
|
||||
#
|
||||
set -u
|
||||
|
||||
AGENT_HOST=${AGENT_HOST:-agent@agenthost}
|
||||
AGENT_PATH=${AGENT_PATH:-/usr/local/etc/agent/build/rspamd/rspamd_domains_db}
|
||||
RSYNC_SRC=${RSYNC_SRC:-$AGENT_HOST:$AGENT_PATH} # a local path skips ssh (tests, co-located agent)
|
||||
RSYNC_RSH=${RSYNC_RSH:-ssh -4}
|
||||
MAPS_DIR=${MAPS_DIR:-/usr/local/etc/migadu_counters/maps}
|
||||
DOMAINS_DIR=${DOMAINS_DIR:-/usr/local/etc/migadu_counters/db}
|
||||
DOMAINS_CDB=${DOMAINS_CDB:-$DOMAINS_DIR/rspamd_domains_db}
|
||||
LOG_FILE=${LOG_FILE:-/var/log/migadu_counters/pull.log}
|
||||
LOCK_FILE=${LOCK_FILE:-$DOMAINS_DIR/.pull-lock}
|
||||
LOCK_MAX_AGE=${LOCK_MAX_AGE:-120} # seconds; older locks are stale (crashed run)
|
||||
RELOAD_CMD=${RELOAD_CMD:-service migadu_counters reload}
|
||||
|
||||
NAME=$(basename "$DOMAINS_CDB")
|
||||
PULLED=$MAPS_DIR/$NAME
|
||||
TMP=$DOMAINS_DIR/.$NAME.tmp
|
||||
|
||||
log() { echo "$(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE"; }
|
||||
mtime() { stat -f %m "$1" 2>/dev/null || stat -c %Y "$1"; }
|
||||
|
||||
cleanup() { rm -f "$TMP" "$LOCK_FILE"; }
|
||||
fail() { log "FAIL: $*"; cleanup; exit 1; }
|
||||
|
||||
mkdir -p "$MAPS_DIR" "$DOMAINS_DIR" "$(dirname "$LOG_FILE")" || { echo "cannot create directories" >&2; exit 1; }
|
||||
|
||||
# Lock: cron (@15) cannot overlap this job with itself; the lock guards a
|
||||
# manual or ansible-triggered run racing the scheduled one.
|
||||
if [ -f "$LOCK_FILE" ] && [ "$(mtime "$LOCK_FILE")" -ge $(( $(date +%s) - LOCK_MAX_AGE )) ]; then
|
||||
log "previous instance is still running, skipping"
|
||||
exit 0
|
||||
fi
|
||||
touch "$LOCK_FILE" || { echo "cannot create $LOCK_FILE" >&2; exit 1; }
|
||||
trap 'rm -f "$TMP" "$LOCK_FILE"' EXIT
|
||||
|
||||
# 1. pull. --checksum: the agent can rebuild the file with the same size
|
||||
# within the same second, which rsync's size+mtime quick check would miss;
|
||||
# checksumming one ~9 MB file every cycle is cheap.
|
||||
if ! rsync -a --checksum -e "$RSYNC_RSH" "$RSYNC_SRC" "$PULLED" >>"$LOG_FILE" 2>&1; then
|
||||
fail "rsync from $RSYNC_SRC failed"
|
||||
fi
|
||||
[ -s "$PULLED" ] || fail "pulled $PULLED is missing or empty"
|
||||
|
||||
# 2. compare with the installed baseline
|
||||
if [ -f "$DOMAINS_CDB" ] && cmp -s "$PULLED" "$DOMAINS_CDB.orig"; then
|
||||
log "$NAME unchanged"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# 3. verify, then install atomically
|
||||
rm -f "$TMP"
|
||||
cp "$PULLED" "$TMP" || fail "copy to $TMP failed"
|
||||
records=$(cdb -s "$TMP" 2>/dev/null | awk '/number of records/ { print $4 }')
|
||||
if [ -z "$records" ]; then
|
||||
fail "$PULLED is not a valid CDB (cdb -s failed), keeping the installed file"
|
||||
fi
|
||||
if [ "$records" -eq 0 ]; then
|
||||
fail "$PULLED holds no records, keeping the installed file"
|
||||
fi
|
||||
chmod 0644 "$TMP"
|
||||
mv "$TMP" "$DOMAINS_CDB" || fail "mv into place failed"
|
||||
cp "$PULLED" "$DOMAINS_CDB.orig" || log "warning: could not update $DOMAINS_CDB.orig (next run will reinstall)"
|
||||
log "$NAME installed ($records records)"
|
||||
|
||||
# The fsnotify watcher already picked up the mv; make sure anyway.
|
||||
$RELOAD_CMD >>"$LOG_FILE" 2>&1 || true
|
||||
exit 0
|
||||
Executable
+102
@@ -0,0 +1,102 @@
|
||||
#!/usr/bin/env bash
|
||||
# Tests for pull-domains.sh against a local "agent" directory (RSYNC_SRC
|
||||
# override) instead of ssh. Needs tinycdb's `cdb` and rsync in PATH.
|
||||
# Run with `make test-contrib`.
|
||||
set -u
|
||||
|
||||
HERE=$(cd "$(dirname "$0")" && pwd)
|
||||
SCRIPT=$HERE/pull-domains.sh
|
||||
TMP=$(mktemp -d)
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
|
||||
export RSYNC_SRC=$TMP/agent/rspamd_domains_db
|
||||
export MAPS_DIR=$TMP/maps
|
||||
export DOMAINS_DIR=$TMP/db
|
||||
export DOMAINS_CDB=$DOMAINS_DIR/rspamd_domains_db
|
||||
export LOG_FILE=$TMP/pull.log
|
||||
export LOCK_FILE=$TMP/.pull-lock
|
||||
export RELOAD_CMD="touch $TMP/reloaded"
|
||||
mkdir -p "$TMP/agent" "$DOMAINS_DIR"
|
||||
|
||||
fails=0
|
||||
pass() { echo "ok - $1"; }
|
||||
fail() { echo "FAIL - $1"; fails=$((fails + 1)); }
|
||||
check() { if eval "$2"; then pass "$1"; else fail "$1"; fi; }
|
||||
|
||||
# make_cdb FILE KEY VALUE: build a one-record CDB the way the agent does (cdb -c).
|
||||
make_cdb() {
|
||||
printf '+%d,%d:%s->%s\n\n' "${#2}" "${#3}" "$2" "$3" | cdb -c "$1"
|
||||
}
|
||||
|
||||
inode() { stat -f %i "$1" 2>/dev/null || stat -c %i "$1"; }
|
||||
|
||||
# --- 1. first install -------------------------------------------------------
|
||||
make_cdb "$RSYNC_SRC" example.com "1049|A/i=200,A/o=20"
|
||||
rm -f "$TMP/reloaded"
|
||||
"$SCRIPT" >/dev/null 2>&1; rc=$?
|
||||
check "first run exits 0" "[ $rc -eq 0 ]"
|
||||
check "first run installs the CDB" "[ -f '$DOMAINS_CDB' ] && cmp -s '$RSYNC_SRC' '$DOMAINS_CDB'"
|
||||
check "first run keeps the .orig baseline" "cmp -s '$RSYNC_SRC' '$DOMAINS_CDB.orig'"
|
||||
check "first run reloads the daemon" "[ -f '$TMP/reloaded' ]"
|
||||
check "no temp file left behind" "! ls '$DOMAINS_DIR'/.rspamd_domains_db.* >/dev/null 2>&1"
|
||||
check "lock released" "[ ! -f '$LOCK_FILE' ]"
|
||||
check "log written" "grep -q 'rspamd_domains_db' '$LOG_FILE'"
|
||||
|
||||
# --- 2. unchanged source ----------------------------------------------------
|
||||
before=$(inode "$DOMAINS_CDB")
|
||||
rm -f "$TMP/reloaded"
|
||||
"$SCRIPT" >/dev/null 2>&1; rc=$?
|
||||
check "unchanged run exits 0" "[ $rc -eq 0 ]"
|
||||
check "unchanged run does not replace the file (same inode)" "[ '$(inode "$DOMAINS_CDB")' = '$before' ]"
|
||||
check "unchanged run does not reload" "[ ! -f '$TMP/reloaded' ]"
|
||||
check "unchanged run logs the skip" "grep -q 'unchanged' '$LOG_FILE'"
|
||||
|
||||
# --- 3. changed source -------------------------------------------------------
|
||||
make_cdb "$RSYNC_SRC" example.com "1049|A/i=500,A/o=50"
|
||||
rm -f "$TMP/reloaded"
|
||||
"$SCRIPT" >/dev/null 2>&1; rc=$?
|
||||
check "changed run exits 0" "[ $rc -eq 0 ]"
|
||||
check "changed run installs atomically (new inode)" "[ '$(inode "$DOMAINS_CDB")' != '$before' ]"
|
||||
check "changed run installs the new content" "cmp -s '$RSYNC_SRC' '$DOMAINS_CDB'"
|
||||
check "changed run updates .orig" "cmp -s '$RSYNC_SRC' '$DOMAINS_CDB.orig'"
|
||||
check "changed run reloads the daemon" "[ -f '$TMP/reloaded' ]"
|
||||
check "changed run logs the install" "grep -q 'installed' '$LOG_FILE'"
|
||||
|
||||
# --- 4. corrupt source -------------------------------------------------------
|
||||
good_inode=$(inode "$DOMAINS_CDB")
|
||||
cp "$DOMAINS_CDB" "$TMP/good.copy"
|
||||
printf 'garbage' > "$RSYNC_SRC"
|
||||
rm -f "$TMP/reloaded"
|
||||
"$SCRIPT" >/dev/null 2>&1; rc=$?
|
||||
check "corrupt source exits non-zero" "[ $rc -ne 0 ]"
|
||||
check "corrupt source leaves the installed file untouched" "[ '$(inode "$DOMAINS_CDB")' = '$good_inode' ] && cmp -s '$DOMAINS_CDB' '$TMP/good.copy'"
|
||||
check "corrupt source does not touch .orig" "cmp -s '$DOMAINS_CDB.orig' '$TMP/good.copy'"
|
||||
check "corrupt source does not reload" "[ ! -f '$TMP/reloaded' ]"
|
||||
check "corrupt source leaves no temp file" "! ls '$DOMAINS_DIR'/.rspamd_domains_db.* >/dev/null 2>&1"
|
||||
check "corrupt source releases the lock" "[ ! -f '$LOCK_FILE' ]"
|
||||
check "corrupt source is logged" "grep -q 'not a valid CDB' '$LOG_FILE'"
|
||||
|
||||
# An empty CDB (zero records) is just as useless: refused too.
|
||||
printf '\n' | cdb -c "$RSYNC_SRC"
|
||||
"$SCRIPT" >/dev/null 2>&1; rc=$?
|
||||
check "empty CDB is refused" "[ $rc -ne 0 ] && cmp -s '$DOMAINS_CDB' '$TMP/good.copy'"
|
||||
|
||||
# --- 5. lock file -------------------------------------------------------------
|
||||
make_cdb "$RSYNC_SRC" example.com "1049|A/i=1,A/o=1"
|
||||
touch "$LOCK_FILE"
|
||||
"$SCRIPT" >/dev/null 2>&1; rc=$?
|
||||
check "fresh lock skips the run (exit 0)" "[ $rc -eq 0 ]"
|
||||
check "fresh lock leaves the file alone" "cmp -s '$DOMAINS_CDB' '$TMP/good.copy'"
|
||||
check "fresh lock is logged" "grep -q 'still running' '$LOG_FILE'"
|
||||
touch -t 200001010000 "$LOCK_FILE" # stale
|
||||
"$SCRIPT" >/dev/null 2>&1; rc=$?
|
||||
check "stale lock is ignored" "[ $rc -eq 0 ] && cmp -s '$RSYNC_SRC' '$DOMAINS_CDB'"
|
||||
|
||||
# --- 6. the pull itself fails ------------------------------------------------
|
||||
rm -f "$RSYNC_SRC"
|
||||
"$SCRIPT" >/dev/null 2>&1; rc=$?
|
||||
check "missing source exits non-zero and keeps the installed file" "[ $rc -ne 0 ] && [ -f '$DOMAINS_CDB' ]"
|
||||
check "failed pull releases the lock" "[ ! -f '$LOCK_FILE' ]"
|
||||
|
||||
echo
|
||||
if [ $fails -eq 0 ]; then echo "all contrib tests passed"; else echo "$fails contrib test(s) FAILED"; exit 1; fi
|
||||
@@ -0,0 +1,31 @@
|
||||
module migadu_counters
|
||||
|
||||
go 1.27.2
|
||||
|
||||
require (
|
||||
github.com/BurntSushi/toml v1.6.0
|
||||
github.com/alicebob/miniredis/v2 v2.34.0
|
||||
github.com/colinmarc/cdb v0.0.0-20190223170904-60f317823f70
|
||||
github.com/fsnotify/fsnotify v1.9.0
|
||||
github.com/prometheus/client_golang v1.23.2
|
||||
github.com/redis/go-redis/v9 v9.7.3
|
||||
golang.org/x/net v0.43.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/alicebob/gopher-json v0.0.0-20230218143504-906a9b012302 // indirect
|
||||
github.com/beorn7/perks v1.0.1 // indirect
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
|
||||
github.com/kr/text v0.2.0 // indirect
|
||||
github.com/kylelemons/godebug v1.1.0 // indirect
|
||||
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
|
||||
github.com/prometheus/client_model v0.6.2 // indirect
|
||||
github.com/prometheus/common v0.66.1 // indirect
|
||||
github.com/prometheus/procfs v0.16.1 // indirect
|
||||
github.com/yuin/gopher-lua v1.1.1 // indirect
|
||||
go.yaml.in/yaml/v2 v2.4.2 // indirect
|
||||
golang.org/x/sys v0.47.0 // indirect
|
||||
golang.org/x/text v0.28.0 // indirect
|
||||
google.golang.org/protobuf v1.36.8 // indirect
|
||||
)
|
||||
@@ -0,0 +1,75 @@
|
||||
github.com/BurntSushi/toml v1.6.0 h1:dRaEfpa2VI55EwlIW72hMRHdWouJeRF7TPYhI+AUQjk=
|
||||
github.com/BurntSushi/toml v1.6.0/go.mod h1:ukJfTF/6rtPPRCnwkur4qwRxa8vTRFBF0uk2lLoLwho=
|
||||
github.com/Pallinder/go-randomdata v1.1.0 h1:gUubB1IEUliFmzjqjhf+bgkg1o6uoFIkRsP3VrhEcx8=
|
||||
github.com/Pallinder/go-randomdata v1.1.0/go.mod h1:yHmJgulpD2Nfrm0cR9tI/+oAgRqCQQixsA8HyRZfV9Y=
|
||||
github.com/alicebob/gopher-json v0.0.0-20230218143504-906a9b012302 h1:uvdUDbHQHO85qeSydJtItA4T55Pw6BtAejd0APRJOCE=
|
||||
github.com/alicebob/gopher-json v0.0.0-20230218143504-906a9b012302/go.mod h1:SGnFV6hVsYE877CKEZ6tDNTjaSXYUk6QqoIK6PrAtcc=
|
||||
github.com/alicebob/miniredis/v2 v2.34.0 h1:mBFWMaJSNL9RwdGRyEDoAAv8OQc5UlEhLDQggTglU/0=
|
||||
github.com/alicebob/miniredis/v2 v2.34.0/go.mod h1:kWShP4b58T1CW0Y5dViCd5ztzrDqRWqM3nksiyXk5s8=
|
||||
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
|
||||
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
|
||||
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
|
||||
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
|
||||
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
|
||||
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
|
||||
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/colinmarc/cdb v0.0.0-20190223170904-60f317823f70 h1:1uCY1nJQwssamFp/L2rk8rRycjBn0l2nYIrP/pPBRgE=
|
||||
github.com/colinmarc/cdb v0.0.0-20190223170904-60f317823f70/go.mod h1:lZuNMoMtkGwujKDy0EndRQBl7owNIHwRq1ycvQeaWqg=
|
||||
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
|
||||
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
|
||||
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
|
||||
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
||||
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
||||
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
|
||||
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
|
||||
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
|
||||
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
|
||||
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
|
||||
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
|
||||
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
|
||||
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
|
||||
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
|
||||
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o=
|
||||
github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg=
|
||||
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
|
||||
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
|
||||
github.com/prometheus/common v0.66.1 h1:h5E0h5/Y8niHc5DlaLlWLArTQI7tMrsfQjHV+d9ZoGs=
|
||||
github.com/prometheus/common v0.66.1/go.mod h1:gcaUsgf3KfRSwHY4dIMXLPV0K/Wg1oZ8+SbZk/HH/dA=
|
||||
github.com/prometheus/procfs v0.16.1 h1:hZ15bTNuirocR6u0JZ6BAHHmwS1p8B4P6MRqxtzMyRg=
|
||||
github.com/prometheus/procfs v0.16.1/go.mod h1:teAbpZRB1iIAJYREa1LsoWUXykVXA1KlTmWl8x/U+Is=
|
||||
github.com/redis/go-redis/v9 v9.7.3 h1:YpPyAayJV+XErNsatSElgRZZVCwXX9QzkKYNvO7x0wM=
|
||||
github.com/redis/go-redis/v9 v9.7.3/go.mod h1:bGUrSggJ9X9GUmZpZNEOQKaANxSGgOEBRltRTZHSvrA=
|
||||
github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ=
|
||||
github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
github.com/yuin/gopher-lua v1.1.1 h1:kYKnWBjvbNP4XLT3+bPEwAXJx262OhaHDWDVOPjL46M=
|
||||
github.com/yuin/gopher-lua v1.1.1/go.mod h1:GBR0iDaNXjAgGg9zfCvksxSRnQx76gclCIb7kdAd1Pw=
|
||||
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
|
||||
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
|
||||
go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI=
|
||||
go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU=
|
||||
golang.org/x/net v0.43.0 h1:lat02VYK2j4aLzMzecihNvTlJNQUq316m2Mr9rnM6YE=
|
||||
golang.org/x/net v0.43.0/go.mod h1:vhO1fvI4dGsIjh73sWfUVjj3N7CA9WkKJNQm2svM6Jg=
|
||||
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
|
||||
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
golang.org/x/text v0.28.0 h1:rhazDwis8INMIwQ4tpjLDzUhx6RlXqZNPEM0huQojng=
|
||||
golang.org/x/text v0.28.0/go.mod h1:U8nCwOR8jO/marOQ0QbDiOngZVEBB7MAiitBuMjXiNU=
|
||||
google.golang.org/protobuf v1.36.8 h1:xHScyCOEuuwZEc6UtSOvPbAT4zRh0xcNRYekJwfqyMc=
|
||||
google.golang.org/protobuf v1.36.8/go.mod h1:fuxRtAxBytpl4zzqUh6/eyUujkJdNiuEkXntxiD/uRU=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
|
||||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
@@ -0,0 +1,255 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestValidateOut(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
in OutRequest
|
||||
wantErr string // substring; "" = accepted
|
||||
want ValidatedOut
|
||||
}{
|
||||
{"valid", OutRequest{TraceID: "tr", AuthUser: "x@example.com", MailFrom: "x@example.com", Recipient: "y@example.com"}, "",
|
||||
ValidatedOut{TraceID: "tr", AuthUser: Address{Value: "x@example.com"}, MailFrom: "x@example.com", MailFromUsable: true, Recipient: Address{Value: "y@example.com"}}},
|
||||
{"normalised", OutRequest{AuthUser: " X@Example.COM ", Recipient: "Y@EXAMPLE.com"}, "",
|
||||
ValidatedOut{AuthUser: Address{Value: "x@example.com"}, Recipient: Address{Value: "y@example.com"}}},
|
||||
// Support/token logins (review #3): mizu forwards the raw SASL login
|
||||
// user@dom@SUFFIX; the base address is counted, the suffix logged.
|
||||
{"login suffix", OutRequest{AuthUser: "x@yagolivas.com@SUPPORTTOKEN", Recipient: "y@example.com"}, "",
|
||||
ValidatedOut{AuthUser: Address{Value: "x@yagolivas.com"}, LoginSuffix: "SUPPORTTOKEN", Recipient: Address{Value: "y@example.com"}}},
|
||||
// mail_from never rejects (review #2): kept normalised when possible,
|
||||
// otherwise as a sanitised string for the log.
|
||||
{"mail_from empty", OutRequest{AuthUser: "x@example.com", Recipient: "y@example.com", MailFrom: ""}, "",
|
||||
ValidatedOut{AuthUser: Address{Value: "x@example.com"}, Recipient: Address{Value: "y@example.com"}}},
|
||||
{"mail_from garbage kept for log", OutRequest{AuthUser: "x@example.com", Recipient: "y@example.com", MailFrom: "john smith@x.com\x1b[1m"}, "",
|
||||
ValidatedOut{AuthUser: Address{Value: "x@example.com"}, MailFrom: "john smith@x.com[1m", Recipient: Address{Value: "y@example.com"}}},
|
||||
// One unusable side: the other is still counted.
|
||||
{"auth_user missing, recipient ok", OutRequest{Recipient: "y@example.com"}, "",
|
||||
ValidatedOut{AuthUser: Address{Reason: "missing"}, Recipient: Address{Value: "y@example.com"}}},
|
||||
{"auth_user bad, recipient ok", OutRequest{AuthUser: "nobody", Recipient: "y@example.com"}, "",
|
||||
ValidatedOut{AuthUser: Address{Raw: "nobody", Reason: "invalid e-mail address"}, Recipient: Address{Value: "y@example.com"}}},
|
||||
{"recipient literal, sender ok", OutRequest{AuthUser: "x@example.com", Recipient: "u@[192.0.2.1]"}, "",
|
||||
ValidatedOut{AuthUser: Address{Value: "x@example.com"}, Recipient: Address{Raw: "u@[192.0.2.1]", Reason: "invalid e-mail address"}}},
|
||||
{"recipient underscore host, sender ok", OutRequest{AuthUser: "x@example.com", Recipient: "u@host_name.example.com"}, "",
|
||||
ValidatedOut{AuthUser: Address{Value: "x@example.com"}, Recipient: Address{Raw: "u@host_name.example.com", Reason: "invalid e-mail address"}}},
|
||||
// IDN domains are counted under their punycode CDB key.
|
||||
{"idn recipient", OutRequest{AuthUser: "x@example.com", Recipient: "info@Bücher.de"}, "",
|
||||
ValidatedOut{AuthUser: Address{Value: "x@example.com"}, Recipient: Address{Value: "info@xn--bcher-kva.de"}}},
|
||||
{"trailing dot stripped", OutRequest{AuthUser: "x@example.com", Recipient: "u@example.com."}, "",
|
||||
ValidatedOut{AuthUser: Address{Value: "x@example.com"}, Recipient: Address{Value: "u@example.com"}}},
|
||||
// Neither side usable: 400 naming both.
|
||||
{"both missing", OutRequest{}, "auth_user: missing; recipient: missing", ValidatedOut{}},
|
||||
{"both bad", OutRequest{AuthUser: "nobody", Recipient: " "}, "auth_user: invalid e-mail address; recipient: missing", ValidatedOut{}},
|
||||
{"trace sanitised", OutRequest{TraceID: "ab\ncd\x1b[31m" + strings.Repeat("z", 200), AuthUser: "x@example.com", Recipient: "y@example.com"}, "",
|
||||
ValidatedOut{TraceID: "abcd[31m" + strings.Repeat("z", 120), AuthUser: Address{Value: "x@example.com"}, Recipient: Address{Value: "y@example.com"}}},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got, err := ValidateOut(tt.in)
|
||||
if tt.wantErr != "" {
|
||||
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Fatalf("err = %v, want containing %q", err, tt.wantErr)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if got != tt.want {
|
||||
t.Errorf("got %+v\nwant %+v", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateIn(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
in InRequest
|
||||
wantErr string
|
||||
want ValidatedIn
|
||||
}{
|
||||
{"bounce sender", InRequest{TraceID: "t", MailFrom: "", Recipient: "Y@Example.com", Junk: true}, "",
|
||||
ValidatedIn{TraceID: "t", Recipient: "y@example.com", Junk: true}},
|
||||
{"valid sender", InRequest{MailFrom: "S@Gmail.com", Recipient: "y@example.com"}, "",
|
||||
ValidatedIn{MailFrom: "s@gmail.com", MailFromUsable: true, Recipient: "y@example.com"}},
|
||||
// Every odd-but-accepted-by-mizu sender keeps the recipient counted.
|
||||
{"sender with space", InRequest{MailFrom: "john smith@example.com", Recipient: "y@example.com"}, "",
|
||||
ValidatedIn{MailFrom: "john smith@example.com", Recipient: "y@example.com"}},
|
||||
{"sender two ats", InRequest{MailFrom: "a@b@example.com", Recipient: "y@example.com"}, "",
|
||||
ValidatedIn{MailFrom: "a@b@example.com", Recipient: "y@example.com"}},
|
||||
{"sender idn", InRequest{MailFrom: "info@bücher.de", Recipient: "y@example.com"}, "",
|
||||
ValidatedIn{MailFrom: "info@xn--bcher-kva.de", MailFromUsable: true, Recipient: "y@example.com"}},
|
||||
{"sender literal", InRequest{MailFrom: "postmaster@[192.0.2.1]", Recipient: "y@example.com"}, "",
|
||||
ValidatedIn{MailFrom: "postmaster@[192.0.2.1]", Recipient: "y@example.com"}},
|
||||
{"sender too long", InRequest{MailFrom: strings.Repeat("v", 250) + "@example.com", Recipient: "y@example.com"}, "",
|
||||
ValidatedIn{MailFrom: strings.Repeat("v", 250) + "@exa", Recipient: "y@example.com"}},
|
||||
{"sender control chars stripped", InRequest{MailFrom: "a\r\nb@x.com", Recipient: "y@example.com"}, "",
|
||||
ValidatedIn{MailFrom: "ab@x.com", Recipient: "y@example.com"}}, // sanitised for the log, not repaired into an address
|
||||
{"recipient idn", InRequest{Recipient: "info@Bücher.de"}, "", ValidatedIn{Recipient: "info@xn--bcher-kva.de"}},
|
||||
{"recipient missing", InRequest{MailFrom: "s@gmail.com"}, "recipient: missing", ValidatedIn{}},
|
||||
{"recipient bad", InRequest{Recipient: "nobody"}, "recipient: invalid e-mail address", ValidatedIn{}},
|
||||
{"recipient literal", InRequest{Recipient: "u@[192.0.2.1]"}, "recipient: invalid e-mail address", ValidatedIn{}},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got, err := ValidateIn(tt.in)
|
||||
if tt.wantErr != "" {
|
||||
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Fatalf("err = %v, want containing %q", err, tt.wantErr)
|
||||
}
|
||||
var fe *FieldError
|
||||
if !errors.As(err, &fe) {
|
||||
t.Errorf("error should be a *FieldError, got %T", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if got != tt.want {
|
||||
t.Errorf("got %+v\nwant %+v", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeAuthUser(t *testing.T) {
|
||||
tests := []struct {
|
||||
in, addr, suffix, wantErr string
|
||||
}{
|
||||
{"x@yagolivas.com", "x@yagolivas.com", "", ""},
|
||||
{" X@Yagolivas.com@SUPPORTTOKEN ", "x@yagolivas.com", "SUPPORTTOKEN", ""},
|
||||
{"x@yagolivas.com@tok@en", "x@yagolivas.com", "tok@en", ""},
|
||||
{"x@yagolivas.com@", "x@yagolivas.com", "", ""},
|
||||
{"nobody", "", "", "exactly one @"},
|
||||
{"@dom@SUFFIX", "", "", "exactly one @"},
|
||||
{"", "", "", "empty"},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
addr, suffix, err := NormalizeAuthUser(tt.in)
|
||||
if tt.wantErr != "" {
|
||||
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Errorf("NormalizeAuthUser(%q) err = %v, want %q", tt.in, err, tt.wantErr)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if err != nil || addr != tt.addr || suffix != tt.suffix {
|
||||
t.Errorf("NormalizeAuthUser(%q) = %q, %q, %v; want %q, %q", tt.in, addr, suffix, err, tt.addr, tt.suffix)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeEmail(t *testing.T) {
|
||||
good := map[string]string{
|
||||
" A.B+tag@Sub.Example.COM ": "a.b+tag@sub.example.com",
|
||||
"x@example.com\t": "x@example.com",
|
||||
"x@xn--exmple-cua.com": "x@xn--exmple-cua.com",
|
||||
"jürgen@example.com": "jürgen@example.com", // SMTPUTF8 local part
|
||||
"x@exämple.com": "x@xn--exmple-cua.com",
|
||||
"u@münchen.de": "u@xn--mnchen-3ya.de",
|
||||
"u@日本.jp": "u@xn--wgv71a.jp",
|
||||
"u@example.com.": "u@example.com",
|
||||
}
|
||||
for in, want := range good {
|
||||
if got, err := NormalizeEmail(in); err != nil || got != want {
|
||||
t.Errorf("NormalizeEmail(%q) = %q, %v; want %q", in, got, err, want)
|
||||
}
|
||||
}
|
||||
bad := []string{"", "nobody", "@x.com", "x@", "a@b@c.com", "x@exam ple.com", "x y@example.com", "x\x01@example.com",
|
||||
strings.Repeat("a", 250) + "@e.com", "x@.example.com", "x@a..b.com", "x@[192.0.2.1]", "x@host_name.example.com", "x@-bad.com", "x@example.com.."}
|
||||
for _, in := range bad {
|
||||
if got, err := NormalizeEmail(in); err == nil {
|
||||
t.Errorf("NormalizeEmail(%q) = %q, want error", in, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSanitizeForLog(t *testing.T) {
|
||||
if got := SanitizeForLog("a\r\nb\x1b[31m" + strings.Repeat("z", 300)); !strings.HasPrefix(got, "ab[31m") || len(got) != MaxEmailLen {
|
||||
t.Errorf("got %q (len %d)", got, len(got))
|
||||
}
|
||||
}
|
||||
|
||||
func TestSanitizeTraceID(t *testing.T) {
|
||||
if got := SanitizeTraceID(" a-b_c.d:e "); got != "a-b_c.d:e" {
|
||||
t.Errorf("got %q", got)
|
||||
}
|
||||
if got := SanitizeTraceID("x\r\ny\x00z\x7f"); got != "xyz" {
|
||||
t.Errorf("got %q", got)
|
||||
}
|
||||
if got := SanitizeTraceID(""); got != "" {
|
||||
t.Errorf("got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseDate(t *testing.T) {
|
||||
now := time.Date(2026, 10, 13, 23, 30, 0, 0, time.FixedZone("CEST", 2*3600)) // 21:30 UTC on the 13th
|
||||
tests := []struct {
|
||||
in, want, wantErr string
|
||||
}{
|
||||
{"", "2026-10-13", ""},
|
||||
{"2026-10-13", "2026-10-13", ""},
|
||||
{"2026-10-12", "2026-10-12", ""},
|
||||
{"2026-10-14", "", "future"},
|
||||
{"2026-1-5", "", "YYYY-MM-DD"},
|
||||
{"2026-13-01", "", "YYYY-MM-DD"},
|
||||
{"13.10.2026", "", "YYYY-MM-DD"},
|
||||
{"2026-10-13T00:00:00Z", "", "YYYY-MM-DD"},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
got, err := ParseDate(tt.in, now)
|
||||
if tt.wantErr != "" {
|
||||
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Errorf("ParseDate(%q) err = %v, want containing %q", tt.in, err, tt.wantErr)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if err != nil || got != tt.want {
|
||||
t.Errorf("ParseDate(%q) = %q, %v; want %q", tt.in, got, err, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCountResponseJSON(t *testing.T) {
|
||||
b, err := json.Marshal(CountResponse{Accepted: true})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if string(b) != `{"accepted":true}` {
|
||||
t.Errorf("JSON = %s", b)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCountersResponseJSON(t *testing.T) {
|
||||
org := int64(1234)
|
||||
in := int64(500)
|
||||
resp := CountersResponse{Query: "me@example.com", Mode: "address", Date: "2026-10-13", Managed: true,
|
||||
Domain: "example.com", OrgID: &org,
|
||||
Levels: []CountersLevel{{Level: "A", ID: "me@example.com", In: 12, InLimit: &in, Out: 3, OutLimit: nil}}}
|
||||
b, _ := json.Marshal(resp)
|
||||
s := string(b)
|
||||
for _, want := range []string{`"query":"me@example.com"`, `"mode":"address"`, `"date":"2026-10-13"`, `"managed":true`, `"exempt":false`,
|
||||
`"domain":"example.com"`, `"org_id":1234`, `"levels":[`, `"in":12`, `"in_limit":500`, `"out":3`, `"out_limit":null`} {
|
||||
if !strings.Contains(s, want) {
|
||||
t.Errorf("JSON %s lacks %s", s, want)
|
||||
}
|
||||
}
|
||||
b, _ = json.Marshal(CountersResponse{Query: "x@nowhere.tld", Mode: "address", Date: "2026-10-13", Levels: []CountersLevel{}})
|
||||
if strings.Contains(string(b), "org_id") || strings.Contains(string(b), `"domain"`) {
|
||||
t.Errorf("unmanaged response should omit org_id and domain: %s", b)
|
||||
}
|
||||
}
|
||||
|
||||
func TestErrorResponseJSON(t *testing.T) {
|
||||
b, _ := json.Marshal(ErrorResponse{Error: "recipient: missing"})
|
||||
if string(b) != `{"error":"recipient: missing"}` {
|
||||
t.Errorf("got %s", b)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
// Package api holds the JSON contract between mailqueuer, the daemon and
|
||||
// counterctl, plus the request validators (the "fields" step of the HTTP
|
||||
// validation layer).
|
||||
package api
|
||||
|
||||
// OutRequest is the body of POST /out: one recipient of an authenticated
|
||||
// submission.
|
||||
type OutRequest struct {
|
||||
TraceID string `json:"trace_id"` // mizu's X-Trace-Id; log correlation only
|
||||
AuthUser string `json:"auth_user"` // SMTP AUTH user (mizu's X-Auth-User)
|
||||
MailFrom string `json:"mail_from"` // envelope sender, optional
|
||||
Recipient string `json:"recipient"`
|
||||
}
|
||||
|
||||
// InRequest is the body of POST /in: one recipient of an inbound message.
|
||||
type InRequest struct {
|
||||
TraceID string `json:"trace_id"`
|
||||
MailFrom string `json:"mail_from"` // "" for a bounce sender <>
|
||||
Recipient string `json:"recipient"`
|
||||
Junk bool `json:"junk"` // mizu's X-Junk: yes
|
||||
}
|
||||
|
||||
// CountResponse answers /in and /out with 202: the request was validated and
|
||||
// queued; counting happens off the request path. The counted values appear in
|
||||
// the counting log and in GET /counters.
|
||||
type CountResponse struct {
|
||||
Accepted bool `json:"accepted"`
|
||||
}
|
||||
|
||||
// ErrorResponse is the body of every non-2xx answer.
|
||||
type ErrorResponse struct {
|
||||
Error string `json:"error"`
|
||||
}
|
||||
|
||||
// CountersLevel is one row of GET /counters.
|
||||
type CountersLevel struct {
|
||||
Level string `json:"level"`
|
||||
ID string `json:"id"`
|
||||
In int64 `json:"in"`
|
||||
InLimit *int64 `json:"in_limit"` // null when unlimited or unmanaged
|
||||
Out int64 `json:"out"`
|
||||
OutLimit *int64 `json:"out_limit"`
|
||||
}
|
||||
|
||||
// CountersResponse answers GET /counters?address=|domain=[&date=].
|
||||
type CountersResponse struct {
|
||||
Query string `json:"query"`
|
||||
Mode string `json:"mode"` // address | domain
|
||||
Date string `json:"date"` // UTC day
|
||||
Managed bool `json:"managed"`
|
||||
Exempt bool `json:"exempt"`
|
||||
Domain string `json:"domain,omitempty"`
|
||||
OrgID *int64 `json:"org_id,omitempty"`
|
||||
Levels []CountersLevel `json:"levels"`
|
||||
}
|
||||
@@ -0,0 +1,223 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
|
||||
"golang.org/x/net/idna"
|
||||
)
|
||||
|
||||
// Limits applied by the validators.
|
||||
const (
|
||||
MaxEmailLen = 254
|
||||
MaxTraceIDLen = 128
|
||||
)
|
||||
|
||||
// FieldError names the offending field: "recipient: missing".
|
||||
type FieldError struct {
|
||||
Field string
|
||||
Reason string
|
||||
}
|
||||
|
||||
func (e *FieldError) Error() string { return e.Field + ": " + e.Reason }
|
||||
|
||||
// Address is one validated address side of a request. A side that is not
|
||||
// usable is logged (Raw, Reason) but never rejects the request on its own:
|
||||
// the other side is still counted.
|
||||
type Address struct {
|
||||
Value string // normalised address; "" when not usable
|
||||
Raw string // sanitised input, for the log, when not usable
|
||||
Reason string // "" when usable; "missing" or "invalid e-mail address"
|
||||
}
|
||||
|
||||
// Usable reports whether the side can be counted.
|
||||
func (a Address) Usable() bool { return a.Value != "" }
|
||||
|
||||
// ValidatedOut is an accepted /out body.
|
||||
type ValidatedOut struct {
|
||||
TraceID string
|
||||
AuthUser Address
|
||||
LoginSuffix string // "SUPPORTTOKEN" of a user@dom@SUPPORTTOKEN login
|
||||
MailFrom string // normalised, or the sanitised input when not an address
|
||||
MailFromUsable bool
|
||||
Recipient Address
|
||||
}
|
||||
|
||||
// ValidatedIn is an accepted /in body.
|
||||
type ValidatedIn struct {
|
||||
TraceID string
|
||||
MailFrom string // normalised, or the sanitised input when not an address
|
||||
MailFromUsable bool
|
||||
Recipient string
|
||||
Junk bool
|
||||
}
|
||||
|
||||
// ValidateOut checks an /out body. auth_user and recipient are validated
|
||||
// independently and the request is rejected only when neither is usable;
|
||||
// mail_from only feeds the exempt check and never rejects.
|
||||
func ValidateOut(r OutRequest) (ValidatedOut, error) {
|
||||
v := ValidatedOut{TraceID: SanitizeTraceID(r.TraceID)}
|
||||
v.AuthUser, v.LoginSuffix = validateAuthUser(r.AuthUser)
|
||||
v.Recipient = validateAddress(r.Recipient)
|
||||
v.MailFrom, v.MailFromUsable = lenient(r.MailFrom)
|
||||
if !v.AuthUser.Usable() && !v.Recipient.Usable() {
|
||||
return ValidatedOut{}, fmt.Errorf("%w; %w", &FieldError{"auth_user", v.AuthUser.Reason}, &FieldError{"recipient", v.Recipient.Reason})
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
|
||||
// ValidateIn checks an /in body: recipient is required and must be an
|
||||
// address; mail_from never rejects (it may be "" for a bounce sender).
|
||||
func ValidateIn(r InRequest) (ValidatedIn, error) {
|
||||
v := ValidatedIn{TraceID: SanitizeTraceID(r.TraceID), Junk: r.Junk}
|
||||
rcpt := validateAddress(r.Recipient)
|
||||
if !rcpt.Usable() {
|
||||
return ValidatedIn{}, &FieldError{"recipient", rcpt.Reason}
|
||||
}
|
||||
v.Recipient = rcpt.Value
|
||||
v.MailFrom, v.MailFromUsable = lenient(r.MailFrom)
|
||||
return v, nil
|
||||
}
|
||||
|
||||
func validateAddress(s string) Address {
|
||||
if strings.TrimSpace(s) == "" {
|
||||
return Address{Reason: "missing"}
|
||||
}
|
||||
n, err := NormalizeEmail(s)
|
||||
if err != nil {
|
||||
return Address{Raw: SanitizeForLog(s), Reason: "invalid e-mail address"}
|
||||
}
|
||||
return Address{Value: n}
|
||||
}
|
||||
|
||||
func validateAuthUser(s string) (Address, string) {
|
||||
if strings.TrimSpace(s) == "" {
|
||||
return Address{Reason: "missing"}, ""
|
||||
}
|
||||
n, suffix, err := NormalizeAuthUser(s)
|
||||
if err != nil {
|
||||
return Address{Raw: SanitizeForLog(s), Reason: "invalid e-mail address"}, ""
|
||||
}
|
||||
return Address{Value: n}, suffix
|
||||
}
|
||||
|
||||
// lenient normalises when possible and otherwise keeps a sanitised copy for
|
||||
// the log; "" stays "" (bounce sender).
|
||||
func lenient(s string) (string, bool) {
|
||||
if strings.TrimSpace(s) == "" {
|
||||
return "", false
|
||||
}
|
||||
if n, err := NormalizeEmail(s); err == nil {
|
||||
return n, true
|
||||
}
|
||||
return SanitizeForLog(s), false
|
||||
}
|
||||
|
||||
// NormalizeAuthUser normalises an SMTP AUTH login. mizu forwards the raw SASL
|
||||
// user name, and support/token logins have the form user@domain@SUFFIX: the
|
||||
// base address is everything up to the second "@", the rest is returned as
|
||||
// the suffix for the log.
|
||||
func NormalizeAuthUser(s string) (addr, suffix string, err error) {
|
||||
s = strings.TrimSpace(s)
|
||||
if first := strings.Index(s, "@"); first >= 0 {
|
||||
if second := strings.Index(s[first+1:], "@"); second >= 0 {
|
||||
base := s[:first+1+second]
|
||||
suffix = s[first+1+second+1:]
|
||||
s = base
|
||||
}
|
||||
}
|
||||
addr, err = NormalizeEmail(s)
|
||||
if err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
return addr, suffix, nil
|
||||
}
|
||||
|
||||
// NormalizeEmail trims, lowercases and syntax-checks an address. The check
|
||||
// is deliberately permissive — the domains CDB decides what is counted — it
|
||||
// only rejects garbage: no "@" (or more than one), empty parts, whitespace or
|
||||
// control characters, over-long input. Non-ASCII local parts are accepted
|
||||
// (SMTPUTF8); a Unicode domain is converted to its punycode form, which is
|
||||
// how the CDB keys it. A trailing dot on the domain is dropped. Address
|
||||
// literals ([192.0.2.1]) and underscores in the domain are rejected: such
|
||||
// domains are never hosted here.
|
||||
func NormalizeEmail(s string) (string, error) {
|
||||
s = strings.ToLower(strings.TrimSpace(s))
|
||||
if s == "" || len(s) > MaxEmailLen {
|
||||
return "", errors.New("empty or longer than 254 characters")
|
||||
}
|
||||
at := strings.Index(s, "@")
|
||||
if at <= 0 || at != strings.LastIndex(s, "@") || at == len(s)-1 {
|
||||
return "", errors.New("must be local@domain with exactly one @")
|
||||
}
|
||||
local, domain := s[:at], s[at+1:]
|
||||
for _, r := range local {
|
||||
if r <= ' ' || r == 0x7f || unicode.IsSpace(r) || unicode.IsControl(r) {
|
||||
return "", errors.New("local part contains whitespace or control characters")
|
||||
}
|
||||
}
|
||||
domain = strings.TrimSuffix(domain, ".")
|
||||
if domain == "" || strings.HasPrefix(domain, ".") || strings.HasSuffix(domain, ".") || strings.Contains(domain, "..") {
|
||||
return "", errors.New("domain has an empty label")
|
||||
}
|
||||
ascii, err := idna.Lookup.ToASCII(domain)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("domain is not a valid host name: %v", err)
|
||||
}
|
||||
for i := 0; i < len(ascii); i++ {
|
||||
c := ascii[i]
|
||||
if !(c >= 'a' && c <= 'z' || c >= '0' && c <= '9' || c == '-' || c == '.') {
|
||||
return "", errors.New("domain contains characters other than a-z, 0-9, '-' and '.'")
|
||||
}
|
||||
}
|
||||
out := local + "@" + ascii
|
||||
if len(out) > MaxEmailLen {
|
||||
return "", errors.New("longer than 254 characters")
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// SanitizeTraceID trims the id, strips control characters (it ends up in log
|
||||
// lines) and caps it at MaxTraceIDLen bytes.
|
||||
func SanitizeTraceID(s string) string { return sanitize(s, MaxTraceIDLen) }
|
||||
|
||||
// SanitizeForLog strips control characters from an arbitrary field value and
|
||||
// caps it at MaxEmailLen bytes, so unusable addresses can be logged safely.
|
||||
func SanitizeForLog(s string) string { return sanitize(s, MaxEmailLen) }
|
||||
|
||||
func sanitize(s string, max int) string {
|
||||
var b strings.Builder
|
||||
for _, r := range strings.TrimSpace(s) {
|
||||
if r < 0x20 || r == 0x7f || unicode.IsControl(r) {
|
||||
continue
|
||||
}
|
||||
if b.Len()+len(string(r)) > max {
|
||||
break
|
||||
}
|
||||
b.WriteRune(r)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// ParseDate validates a /counters or counterctl date: "" means today (UTC),
|
||||
// otherwise an ISO 8601 calendar date YYYY-MM-DD that is not in the future.
|
||||
// It returns the normalised YYYY-MM-DD string.
|
||||
func ParseDate(s string, now time.Time) (string, error) {
|
||||
y, m, d := now.UTC().Date()
|
||||
today := time.Date(y, m, d, 0, 0, 0, 0, time.UTC)
|
||||
s = strings.TrimSpace(s)
|
||||
if s == "" {
|
||||
return today.Format("2006-01-02"), nil
|
||||
}
|
||||
day, err := time.Parse("2006-01-02", s)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("invalid date %q: use ISO 8601 YYYY-MM-DD", s)
|
||||
}
|
||||
if day.After(today) {
|
||||
return "", fmt.Errorf("date %s is in the future (today is %s UTC)", s, today.Format("2006-01-02"))
|
||||
}
|
||||
return day.Format("2006-01-02"), nil
|
||||
}
|
||||
@@ -0,0 +1,369 @@
|
||||
// Package config loads and validates the migadu_counters TOML configuration,
|
||||
// shared by the daemon (cmd/migadu_counters) and the CLI (cmd/counterctl).
|
||||
package config
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/netip"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/BurntSushi/toml"
|
||||
)
|
||||
|
||||
// DefaultPaths are tried in order when no -config flag is given.
|
||||
var DefaultPaths = []string{
|
||||
"/etc/migadu_counters/config.toml",
|
||||
"/usr/local/etc/migadu_counters/config.toml",
|
||||
}
|
||||
|
||||
// Environment variables that override the secrets from the file.
|
||||
const (
|
||||
EnvAuthToken = "MIGADU_COUNTERS_AUTH_TOKEN"
|
||||
EnvRedisPassword = "MIGADU_COUNTERS_REDIS_PASSWORD"
|
||||
EnvNotifyPassword = "MIGADU_COUNTERS_NOTIFY_PASSWORD"
|
||||
)
|
||||
|
||||
// NotifyPlaceholders are the template variables every notify URL is expected
|
||||
// to carry: the web app's /api/limits action pattern-matches all six query
|
||||
// parameters and answers 500 when one is missing.
|
||||
var NotifyPlaceholders = []string{"$LEVEL", "$ID", "$FLOW", "$PERIOD", "$ADDRESS", "$COUNT"}
|
||||
|
||||
// Config is the full daemon configuration.
|
||||
type Config struct {
|
||||
Addr string `toml:"addr"`
|
||||
AdminAddr string `toml:"admin_addr"`
|
||||
AuthToken string `toml:"auth_token"`
|
||||
AllowedCIDRs []string `toml:"allowed_cidrs"`
|
||||
DomainsCDB string `toml:"domains_cdb"`
|
||||
Exempt []string `toml:"exempt"`
|
||||
CountJunk bool `toml:"count_junk"`
|
||||
|
||||
// Counting runs off the request path: /in and /out enqueue a job and
|
||||
// answer 202; Workers goroutines drain a queue of QueueSize jobs.
|
||||
QueueSize int `toml:"queue_size"`
|
||||
Workers int `toml:"workers"`
|
||||
// Parsed CDB records are cached per loaded table up to this many
|
||||
// domains; beyond it lookups parse without caching. 0 disables the cache.
|
||||
MaxCachedDomains int `toml:"max_cached_domains"`
|
||||
|
||||
// HTTP server timeouts in seconds.
|
||||
ReadHeaderTimeout int `toml:"read_header_timeout"`
|
||||
WriteTimeout int `toml:"write_timeout"`
|
||||
IdleTimeout int `toml:"idle_timeout"`
|
||||
HandlerTimeout int `toml:"handler_timeout"`
|
||||
|
||||
Logging LoggingConfig `toml:"logging"`
|
||||
Redis RedisConfig `toml:"redis"`
|
||||
Notify NotifyConfig `toml:"notify"`
|
||||
|
||||
prefixes []netip.Prefix
|
||||
exempt map[string]bool
|
||||
}
|
||||
|
||||
// LoggingConfig holds the application log and the counting transaction log
|
||||
// settings (the first three fields match the rcptd/mailqueuer/mizu pattern).
|
||||
type LoggingConfig struct {
|
||||
Output string `toml:"output"` // "stderr", "stdout", or a file path
|
||||
Format string `toml:"format"` // "console" or "json"
|
||||
Level string `toml:"level"` // "debug", "info", "warn", "error"
|
||||
|
||||
CountingOutput string `toml:"counting_output"` // default: same as Output; "off" disables
|
||||
CountingFormat string `toml:"counting_format"` // "text" (Lua-like lines) or "json"
|
||||
}
|
||||
|
||||
// CountingEnabled reports whether the counting transaction log is written.
|
||||
func (l LoggingConfig) CountingEnabled() bool { return l.CountingOutput != "off" }
|
||||
|
||||
// RedisConfig describes the write (master) and read (replica) sides.
|
||||
// Sentinel mode is used when Sentinels is non-empty; otherwise the static
|
||||
// WriteAddr/ReadAddrs are used (development, tests).
|
||||
type RedisConfig struct {
|
||||
DB int `toml:"db"`
|
||||
Password string `toml:"password"`
|
||||
PoolSize int `toml:"pool_size"`
|
||||
TimeoutMS int `toml:"timeout_ms"`
|
||||
|
||||
MasterName string `toml:"master_name"`
|
||||
Sentinels []string `toml:"sentinels"`
|
||||
SentinelPassword string `toml:"sentinel_password"`
|
||||
|
||||
WriteAddr string `toml:"write_addr"`
|
||||
ReadAddrs []string `toml:"read_addrs"`
|
||||
}
|
||||
|
||||
// SentinelMode reports whether addresses are discovered through Sentinel.
|
||||
func (r RedisConfig) SentinelMode() bool { return len(r.Sentinels) > 0 }
|
||||
|
||||
// Timeout is the dial/read/write timeout for every Redis command.
|
||||
func (r RedisConfig) Timeout() time.Duration { return time.Duration(r.TimeoutMS) * time.Millisecond }
|
||||
|
||||
// NotifyConfig holds the per-level web app notification URLs. An empty URL
|
||||
// disables notifications for that level.
|
||||
type NotifyConfig struct {
|
||||
OrgURL string `toml:"org_url"`
|
||||
AddressURL string `toml:"address_url"`
|
||||
DomainURL string `toml:"domain_url"`
|
||||
User string `toml:"user"`
|
||||
Password string `toml:"password"`
|
||||
TimeoutSeconds int `toml:"timeout_seconds"`
|
||||
}
|
||||
|
||||
// Timeout is the per-attempt HTTP timeout for a notification.
|
||||
func (n NotifyConfig) Timeout() time.Duration { return time.Duration(n.TimeoutSeconds) * time.Second }
|
||||
|
||||
// Default returns the configuration with every default applied.
|
||||
func Default() Config {
|
||||
return Config{
|
||||
Addr: ":5555",
|
||||
AdminAddr: ":9095",
|
||||
Exempt: []string{"support@migadu.com", "contact@migadu.com"},
|
||||
QueueSize: 10000,
|
||||
Workers: 8,
|
||||
MaxCachedDomains: 20000,
|
||||
ReadHeaderTimeout: 5,
|
||||
WriteTimeout: 30,
|
||||
IdleTimeout: 60,
|
||||
HandlerTimeout: 4,
|
||||
Logging: LoggingConfig{Output: "stderr", Format: "console", Level: "info", CountingFormat: "text"},
|
||||
Redis: RedisConfig{PoolSize: 32, TimeoutMS: 500},
|
||||
Notify: NotifyConfig{TimeoutSeconds: 5},
|
||||
}
|
||||
}
|
||||
|
||||
// Discover returns flagPath when given, otherwise the first existing entry of
|
||||
// DefaultPaths, or an error naming every path that was tried.
|
||||
func Discover(flagPath string) (string, error) {
|
||||
if flagPath != "" {
|
||||
return flagPath, nil
|
||||
}
|
||||
for _, p := range DefaultPaths {
|
||||
if _, err := os.Stat(p); err == nil {
|
||||
return p, nil
|
||||
}
|
||||
}
|
||||
return "", fmt.Errorf("no config found at %s; specify -config PATH", strings.Join(DefaultPaths, " or "))
|
||||
}
|
||||
|
||||
// Load reads the TOML file at path, applies defaults, environment overrides
|
||||
// and normalisation, and validates the result. Unknown keys are fatal: a typo
|
||||
// such as "auth-token" must not start the daemon with the default in place.
|
||||
func Load(path string) (*Config, error) {
|
||||
cfg := Default()
|
||||
md, err := toml.DecodeFile(path, &cfg)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read config %s: %w", path, err)
|
||||
}
|
||||
var unknown []error
|
||||
for _, k := range md.Undecoded() {
|
||||
unknown = append(unknown, fmt.Errorf("unknown config key %s", k.String()))
|
||||
}
|
||||
if len(unknown) > 0 {
|
||||
return nil, fmt.Errorf("config %s: %w", path, errors.Join(unknown...))
|
||||
}
|
||||
|
||||
if v := os.Getenv(EnvAuthToken); v != "" {
|
||||
cfg.AuthToken = v
|
||||
}
|
||||
if v := os.Getenv(EnvRedisPassword); v != "" {
|
||||
cfg.Redis.Password = v
|
||||
}
|
||||
if v := os.Getenv(EnvNotifyPassword); v != "" {
|
||||
cfg.Notify.Password = v
|
||||
}
|
||||
|
||||
// A relative CDB path is resolved against the config file's directory,
|
||||
// like rcptd's cdb_tables_dir.
|
||||
if cfg.DomainsCDB != "" && !filepath.IsAbs(cfg.DomainsCDB) {
|
||||
cfg.DomainsCDB = filepath.Join(filepath.Dir(path), cfg.DomainsCDB)
|
||||
}
|
||||
|
||||
cfg.normalize()
|
||||
if err := cfg.Validate(); err != nil {
|
||||
return nil, fmt.Errorf("config %s: %w", path, err)
|
||||
}
|
||||
return &cfg, nil
|
||||
}
|
||||
|
||||
func (c *Config) normalize() {
|
||||
c.exempt = make(map[string]bool, len(c.Exempt))
|
||||
for _, a := range c.Exempt {
|
||||
if a = strings.ToLower(strings.TrimSpace(a)); a != "" {
|
||||
c.exempt[a] = true
|
||||
}
|
||||
}
|
||||
|
||||
c.Logging.Format = strings.ToLower(strings.TrimSpace(c.Logging.Format))
|
||||
c.Logging.Level = strings.ToLower(strings.TrimSpace(c.Logging.Level))
|
||||
c.Logging.CountingFormat = strings.ToLower(strings.TrimSpace(c.Logging.CountingFormat))
|
||||
c.Logging.CountingOutput = strings.TrimSpace(c.Logging.CountingOutput)
|
||||
switch {
|
||||
case c.Logging.CountingOutput == "":
|
||||
c.Logging.CountingOutput = c.Logging.Output
|
||||
case strings.EqualFold(c.Logging.CountingOutput, "off"):
|
||||
c.Logging.CountingOutput = "off"
|
||||
}
|
||||
|
||||
if !c.Redis.SentinelMode() && len(c.Redis.ReadAddrs) == 0 && c.Redis.WriteAddr != "" {
|
||||
c.Redis.ReadAddrs = []string{c.Redis.WriteAddr}
|
||||
}
|
||||
}
|
||||
|
||||
// Validate checks the configuration and returns every problem found, joined.
|
||||
func (c *Config) Validate() error {
|
||||
var errs []error
|
||||
fail := func(format string, args ...any) { errs = append(errs, fmt.Errorf(format, args...)) }
|
||||
|
||||
if c.DomainsCDB == "" {
|
||||
fail("domains_cdb is required")
|
||||
}
|
||||
if strings.TrimSpace(c.AuthToken) == "" {
|
||||
fail("auth_token is required; set it in the config or %s", EnvAuthToken)
|
||||
}
|
||||
if c.QueueSize <= 0 {
|
||||
fail("queue_size must be > 0")
|
||||
}
|
||||
if c.Workers <= 0 {
|
||||
fail("workers must be > 0")
|
||||
}
|
||||
if c.MaxCachedDomains < 0 {
|
||||
fail("max_cached_domains must be >= 0")
|
||||
}
|
||||
for _, a := range []struct{ name, addr string }{{"addr", c.Addr}, {"admin_addr", c.AdminAddr}} {
|
||||
if _, _, err := net.SplitHostPort(a.addr); err != nil {
|
||||
fail("%s: invalid listen address %q: %v", a.name, a.addr, err)
|
||||
}
|
||||
}
|
||||
for _, d := range []struct {
|
||||
name string
|
||||
v int
|
||||
}{{"read_header_timeout", c.ReadHeaderTimeout}, {"write_timeout", c.WriteTimeout}, {"idle_timeout", c.IdleTimeout}, {"handler_timeout", c.HandlerTimeout}} {
|
||||
if d.v <= 0 {
|
||||
fail("%s must be > 0 (seconds)", d.name)
|
||||
}
|
||||
}
|
||||
|
||||
c.prefixes = c.prefixes[:0]
|
||||
for _, s := range c.AllowedCIDRs {
|
||||
p, err := parsePrefix(s)
|
||||
if err != nil {
|
||||
fail("allowed_cidrs: invalid entry %q: %v", s, err)
|
||||
continue
|
||||
}
|
||||
c.prefixes = append(c.prefixes, p)
|
||||
}
|
||||
|
||||
switch c.Logging.Format {
|
||||
case "console", "json":
|
||||
default:
|
||||
fail("logging.format: %q (must be console or json)", c.Logging.Format)
|
||||
}
|
||||
if _, err := parseLogLevel(c.Logging.Level); err != nil {
|
||||
fail("logging.level: %v", err)
|
||||
}
|
||||
switch c.Logging.CountingFormat {
|
||||
case "text", "json":
|
||||
default:
|
||||
fail("logging.counting_format: %q (must be text or json)", c.Logging.CountingFormat)
|
||||
}
|
||||
|
||||
if c.Redis.DB != 0 {
|
||||
fail("redis.db must be 0: migadu_web reads the message counters from DB 0 (got %d)", c.Redis.DB)
|
||||
}
|
||||
if c.Redis.PoolSize <= 0 {
|
||||
fail("redis.pool_size must be > 0")
|
||||
}
|
||||
if c.Redis.TimeoutMS <= 0 {
|
||||
fail("redis.timeout_ms must be > 0")
|
||||
}
|
||||
if c.Redis.SentinelMode() {
|
||||
if c.Redis.MasterName == "" {
|
||||
fail("redis.master_name is required when redis.sentinels is set")
|
||||
}
|
||||
for _, s := range c.Redis.Sentinels {
|
||||
if _, _, err := net.SplitHostPort(s); err != nil {
|
||||
fail("redis.sentinels: invalid address %q: %v", s, err)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
if c.Redis.WriteAddr == "" {
|
||||
fail("redis.write_addr is required (or configure redis.sentinels)")
|
||||
} else if _, _, err := net.SplitHostPort(c.Redis.WriteAddr); err != nil {
|
||||
fail("redis.write_addr: invalid address %q: %v", c.Redis.WriteAddr, err)
|
||||
}
|
||||
for _, s := range c.Redis.ReadAddrs {
|
||||
if _, _, err := net.SplitHostPort(s); err != nil {
|
||||
fail("redis.read_addrs: invalid address %q: %v", s, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for _, u := range c.notifyURLs() {
|
||||
if u.url == "" {
|
||||
continue
|
||||
}
|
||||
parsed, err := url.Parse(u.url)
|
||||
if err != nil || (parsed.Scheme != "http" && parsed.Scheme != "https") || parsed.Host == "" {
|
||||
fail("notify.%s: must be an absolute http(s) URL (got %q)", u.name, u.url)
|
||||
}
|
||||
}
|
||||
if c.Notify.TimeoutSeconds <= 0 {
|
||||
fail("notify.timeout_seconds must be > 0")
|
||||
}
|
||||
|
||||
return errors.Join(errs...)
|
||||
}
|
||||
|
||||
// Warnings returns non-fatal findings worth logging at startup.
|
||||
func (c *Config) Warnings() []string {
|
||||
var w []string
|
||||
for _, u := range c.notifyURLs() {
|
||||
if u.url == "" {
|
||||
continue
|
||||
}
|
||||
for _, ph := range NotifyPlaceholders {
|
||||
if !strings.Contains(u.url, ph) {
|
||||
w = append(w, fmt.Sprintf("notify.%s: missing %s placeholder (the web app requires all six query parameters)", u.name, ph))
|
||||
}
|
||||
}
|
||||
}
|
||||
return w
|
||||
}
|
||||
|
||||
func (c *Config) notifyURLs() []struct{ name, url string } {
|
||||
return []struct{ name, url string }{
|
||||
{"org_url", c.Notify.OrgURL},
|
||||
{"address_url", c.Notify.AddressURL},
|
||||
{"domain_url", c.Notify.DomainURL},
|
||||
}
|
||||
}
|
||||
|
||||
// ExemptSet returns the normalised exempt addresses.
|
||||
func (c *Config) ExemptSet() map[string]bool { return c.exempt }
|
||||
|
||||
// AllowedPrefixes returns the parsed allowed_cidrs; empty means any source.
|
||||
func (c *Config) AllowedPrefixes() []netip.Prefix { return c.prefixes }
|
||||
|
||||
// HandlerTimeoutDuration is the http.TimeoutHandler budget.
|
||||
func (c *Config) HandlerTimeoutDuration() time.Duration {
|
||||
return time.Duration(c.HandlerTimeout) * time.Second
|
||||
}
|
||||
|
||||
// parsePrefix accepts "10.0.0.0/8" as well as a bare address ("192.168.1.5",
|
||||
// "::1"), which becomes a single-host prefix.
|
||||
func parsePrefix(s string) (netip.Prefix, error) {
|
||||
s = strings.TrimSpace(s)
|
||||
if p, err := netip.ParsePrefix(s); err == nil {
|
||||
return p.Masked(), nil
|
||||
}
|
||||
a, err := netip.ParseAddr(s)
|
||||
if err != nil {
|
||||
return netip.Prefix{}, errors.New("not a CIDR prefix or IP address")
|
||||
}
|
||||
return netip.PrefixFrom(a, a.BitLen()), nil
|
||||
}
|
||||
@@ -0,0 +1,288 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// writeConfig writes body to <tmpdir>/config.toml and returns its path.
|
||||
func writeConfig(t *testing.T, body string) string {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "config.toml")
|
||||
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
// TOML: top-level keys must precede any [table], and a table may be defined
|
||||
// only once — fixtures are therefore assembled as top + ... + redisStatic.
|
||||
const (
|
||||
top = "domains_cdb = \"/tmp/rspamd_domains_db\"\nauth_token = \"t\"\n"
|
||||
redisStatic = "[redis]\nwrite_addr = \"127.0.0.1:6379\"\n"
|
||||
minimal = top + redisStatic
|
||||
)
|
||||
|
||||
func mustLoad(t *testing.T, body string) *Config {
|
||||
t.Helper()
|
||||
cfg, err := Load(writeConfig(t, body))
|
||||
if err != nil {
|
||||
t.Fatalf("Load: %v\n--- config ---\n%s", err, body)
|
||||
}
|
||||
return cfg
|
||||
}
|
||||
|
||||
func TestLoad_Defaults(t *testing.T) {
|
||||
cfg := mustLoad(t, minimal)
|
||||
checks := []struct {
|
||||
name string
|
||||
got any
|
||||
want any
|
||||
}{
|
||||
{"addr", cfg.Addr, ":5555"},
|
||||
{"admin_addr", cfg.AdminAddr, ":9095"},
|
||||
{"auth_token", cfg.AuthToken, "t"},
|
||||
{"queue_size", cfg.QueueSize, 10000},
|
||||
{"workers", cfg.Workers, 8},
|
||||
{"max_cached_domains", cfg.MaxCachedDomains, 20000},
|
||||
{"count_junk", cfg.CountJunk, false},
|
||||
{"read_header_timeout", cfg.ReadHeaderTimeout, 5},
|
||||
{"write_timeout", cfg.WriteTimeout, 30},
|
||||
{"idle_timeout", cfg.IdleTimeout, 60},
|
||||
{"handler_timeout", cfg.HandlerTimeout, 4},
|
||||
{"logging.output", cfg.Logging.Output, "stderr"},
|
||||
{"logging.format", cfg.Logging.Format, "console"},
|
||||
{"logging.level", cfg.Logging.Level, "info"},
|
||||
{"logging.counting_output", cfg.Logging.CountingOutput, "stderr"},
|
||||
{"logging.counting_format", cfg.Logging.CountingFormat, "text"},
|
||||
{"redis.db", cfg.Redis.DB, 0},
|
||||
{"redis.pool_size", cfg.Redis.PoolSize, 32},
|
||||
{"redis.timeout_ms", cfg.Redis.TimeoutMS, 500},
|
||||
{"redis.sentinel_mode", cfg.Redis.SentinelMode(), false},
|
||||
{"notify.timeout_seconds", cfg.Notify.TimeoutSeconds, 5},
|
||||
{"redis timeout", cfg.Redis.Timeout(), 500 * time.Millisecond},
|
||||
{"notify timeout", cfg.Notify.Timeout(), 5 * time.Second},
|
||||
{"handler timeout", cfg.HandlerTimeoutDuration(), 4 * time.Second},
|
||||
}
|
||||
for _, c := range checks {
|
||||
if c.got != c.want {
|
||||
t.Errorf("%s = %v, want %v", c.name, c.got, c.want)
|
||||
}
|
||||
}
|
||||
if got := cfg.Redis.ReadAddrs; len(got) != 1 || got[0] != "127.0.0.1:6379" {
|
||||
t.Errorf("read_addrs = %v, want [write_addr]", got)
|
||||
}
|
||||
if got := cfg.ExemptSet(); len(got) != 2 || !got["support@migadu.com"] || !got["contact@migadu.com"] {
|
||||
t.Errorf("ExemptSet() = %v, want support@ and contact@", got)
|
||||
}
|
||||
if len(cfg.AllowedPrefixes()) != 0 {
|
||||
t.Errorf("AllowedPrefixes() = %v, want none", cfg.AllowedPrefixes())
|
||||
}
|
||||
if !cfg.Logging.CountingEnabled() {
|
||||
t.Error("counting log should be enabled by default")
|
||||
}
|
||||
}
|
||||
|
||||
// The shipped example must always load, validate and produce no warnings.
|
||||
func TestLoad_Example(t *testing.T) {
|
||||
cfg, err := Load("../../config.toml.example")
|
||||
if err != nil {
|
||||
t.Fatalf("Load(config.toml.example): %v", err)
|
||||
}
|
||||
if !cfg.Redis.SentinelMode() || cfg.Redis.MasterName != "mymaster" || len(cfg.Redis.Sentinels) != 3 {
|
||||
t.Errorf("example should be in sentinel mode with 3 sentinels: %+v", cfg.Redis)
|
||||
}
|
||||
if cfg.Redis.DB != 0 {
|
||||
t.Errorf("example redis.db = %d, must be 0", cfg.Redis.DB)
|
||||
}
|
||||
if cfg.Notify.OrgURL == "" || cfg.Notify.AddressURL == "" {
|
||||
t.Error("example should set org_url and address_url")
|
||||
}
|
||||
if cfg.Notify.DomainURL != "" {
|
||||
t.Error("example should leave domain_url empty (web app D branch is a TODO)")
|
||||
}
|
||||
for _, w := range cfg.Warnings() {
|
||||
t.Errorf("example produced warning: %s", w)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoad_MissingFile(t *testing.T) {
|
||||
if _, err := Load(filepath.Join(t.TempDir(), "nope.toml")); err == nil {
|
||||
t.Fatal("expected error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoad_BadTOML(t *testing.T) {
|
||||
if _, err := Load(writeConfig(t, "addr = [unterminated")); err == nil {
|
||||
t.Fatal("expected error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoad_RelativeDomainsCDBAnchoredToConfigDir(t *testing.T) {
|
||||
path := writeConfig(t, "domains_cdb = \"db/rspamd_domains_db\"\nauth_token = \"t\"\n"+redisStatic)
|
||||
cfg, err := Load(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := filepath.Join(filepath.Dir(path), "db/rspamd_domains_db")
|
||||
if cfg.DomainsCDB != want {
|
||||
t.Errorf("domains_cdb = %q, want %q", cfg.DomainsCDB, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoad_EnvOverrides(t *testing.T) {
|
||||
t.Setenv("MIGADU_COUNTERS_AUTH_TOKEN", "env-token")
|
||||
t.Setenv("MIGADU_COUNTERS_REDIS_PASSWORD", "env-redis")
|
||||
t.Setenv("MIGADU_COUNTERS_NOTIFY_PASSWORD", "env-notify")
|
||||
cfg := mustLoad(t, "domains_cdb = \"/tmp/x\"\nauth_token = \"file-token\"\n"+redisStatic+"password = \"file-redis\"\n[notify]\npassword = \"file-notify\"\n")
|
||||
if cfg.AuthToken != "env-token" || cfg.Redis.Password != "env-redis" || cfg.Notify.Password != "env-notify" {
|
||||
t.Errorf("env overrides not applied: %q %q %q", cfg.AuthToken, cfg.Redis.Password, cfg.Notify.Password)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoad_ExemptNormalized(t *testing.T) {
|
||||
cfg := mustLoad(t, top+"exempt = [\" Support@Migadu.com \", \"\", \"x@y.z\"]\n"+redisStatic)
|
||||
got := cfg.ExemptSet()
|
||||
if len(got) != 2 || !got["support@migadu.com"] || !got["x@y.z"] {
|
||||
t.Errorf("ExemptSet() = %v, want only the configured entries, normalised", got)
|
||||
}
|
||||
// An explicitly empty list disables the exemption entirely.
|
||||
if got := mustLoad(t, top+"exempt = []\n"+redisStatic).ExemptSet(); len(got) != 0 {
|
||||
t.Errorf("exempt = [] gave %v, want none", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoad_AllowedCIDRs(t *testing.T) {
|
||||
cfg := mustLoad(t, top+"allowed_cidrs = [\"10.0.0.0/8\", \"192.168.1.5\", \"::1\"]\n"+redisStatic)
|
||||
p := cfg.AllowedPrefixes()
|
||||
if len(p) != 3 || p[0].Bits() != 8 || p[1].Bits() != 32 || p[2].Bits() != 128 {
|
||||
t.Errorf("AllowedPrefixes() = %v", p)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoad_SentinelMode(t *testing.T) {
|
||||
cfg := mustLoad(t, top+"[redis]\nmaster_name = \"mymaster\"\nsentinels = [\"10.0.0.1:26379\", \"10.0.0.2:26379\"]\n")
|
||||
if !cfg.Redis.SentinelMode() || len(cfg.Redis.ReadAddrs) != 0 || cfg.Redis.WriteAddr != "" {
|
||||
t.Errorf("sentinel mode config = %+v", cfg.Redis)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidate_Errors(t *testing.T) {
|
||||
tests := []struct {
|
||||
name, toml, wantErr string
|
||||
}{
|
||||
{"redis db must be 0", top + redisStatic + "db = 1\n", "redis.db"},
|
||||
{"domains_cdb required", "auth_token = \"t\"\n" + redisStatic, "domains_cdb"},
|
||||
{"auth_token required", "domains_cdb = \"/tmp/x\"\n" + redisStatic, "auth_token is required"},
|
||||
{"auth_token blank", "domains_cdb = \"/tmp/x\"\nauth_token = \" \"\n" + redisStatic, "auth_token is required"},
|
||||
{"typo auth-token", "domains_cdb = \"/tmp/x\"\nauth-token = \"s3cret\"\n" + redisStatic, "unknown config key auth-token"},
|
||||
{"typo allowed_cidr", top + "allowed_cidr = [\"10.0.0.0/8\"]\n" + redisStatic, "unknown config key allowed_cidr"},
|
||||
{"typo in table", top + redisStatic + "pool = 3\n", "unknown config key redis.pool"},
|
||||
{"queue_size", top + "queue_size = 0\n" + redisStatic, "queue_size"},
|
||||
{"workers", top + "workers = 0\n" + redisStatic, "workers"},
|
||||
{"max_cached_domains", top + "max_cached_domains = -1\n" + redisStatic, "max_cached_domains"},
|
||||
{"sentinel needs master_name", top + "[redis]\nsentinels = [\"10.0.0.1:26379\"]\n", "master_name"},
|
||||
{"sentinel bad address", top + "[redis]\nmaster_name = \"m\"\nsentinels = [\"10.0.0.1\"]\n", "sentinels"},
|
||||
{"static needs write_addr", top, "write_addr"},
|
||||
{"static bad read_addr", top + redisStatic + "read_addrs = [\"nohost\"]\n", "read_addrs"},
|
||||
{"bad cidr mask", top + "allowed_cidrs = [\"10.0.0.0/33\"]\n" + redisStatic, "allowed_cidrs"},
|
||||
{"bad cidr text", top + "allowed_cidrs = [\"nonsense\"]\n" + redisStatic, "allowed_cidrs"},
|
||||
{"notify scheme", minimal + "[notify]\norg_url = \"ftp://x/limits\"\n", "org_url"},
|
||||
{"notify no scheme", minimal + "[notify]\naddress_url = \"not a url\"\n", "address_url"},
|
||||
{"notify no host", minimal + "[notify]\ndomain_url = \"https:///path\"\n", "domain_url"},
|
||||
{"logging format", minimal + "[logging]\nformat = \"xml\"\n", "logging.format"},
|
||||
{"logging level", minimal + "[logging]\nlevel = \"loud\"\n", "logging.level"},
|
||||
{"counting format", minimal + "[logging]\ncounting_format = \"csv\"\n", "counting_format"},
|
||||
{"pool size", top + redisStatic + "pool_size = -1\n", "pool_size"},
|
||||
{"timeout_ms", top + redisStatic + "timeout_ms = 0\n", "timeout_ms"},
|
||||
{"handler_timeout", top + "handler_timeout = 0\n" + redisStatic, "handler_timeout"},
|
||||
{"addr", top + "addr = \"no-port\"\n" + redisStatic, "addr"},
|
||||
{"admin_addr", top + "admin_addr = \"no-port\"\n" + redisStatic, "admin_addr"},
|
||||
{"notify timeout", minimal + "[notify]\ntimeout_seconds = 0\n", "timeout_seconds"},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
_, err := Load(writeConfig(t, tt.toml))
|
||||
if err == nil {
|
||||
t.Fatalf("expected error mentioning %q", tt.wantErr)
|
||||
}
|
||||
if !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Errorf("error %q does not mention %q", err, tt.wantErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestLogging_CountingOutput(t *testing.T) {
|
||||
tests := []struct {
|
||||
toml string
|
||||
wantOut string
|
||||
wantEnabled bool
|
||||
}{
|
||||
{"", "stderr", true},
|
||||
{"[logging]\noutput = \"/var/log/a.log\"\n", "/var/log/a.log", true}, // follows output
|
||||
{"[logging]\noutput = \"/var/log/a.log\"\ncounting_output = \"/var/log/c.log\"\n", "/var/log/c.log", true},
|
||||
{"[logging]\ncounting_output = \"off\"\n", "off", false},
|
||||
{"[logging]\ncounting_output = \"OFF\"\n", "off", false},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
cfg := mustLoad(t, minimal+tt.toml)
|
||||
if cfg.Logging.CountingOutput != tt.wantOut || cfg.Logging.CountingEnabled() != tt.wantEnabled {
|
||||
t.Errorf("%q: counting_output=%q enabled=%v, want %q %v", tt.toml,
|
||||
cfg.Logging.CountingOutput, cfg.Logging.CountingEnabled(), tt.wantOut, tt.wantEnabled)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWarnings_MissingPlaceholders(t *testing.T) {
|
||||
cfg := mustLoad(t, top+redisStatic+`
|
||||
[notify]
|
||||
org_url = "https://admin.example/api/limits?level=$LEVEL&id=$ID&flow=$FLOW&period=$PERIOD&by=$ADDRESS"
|
||||
address_url = "https://admin.example/api/limits?level=$LEVEL&id=$ID&flow=$FLOW&period=$PERIOD&by=$ADDRESS&count=$COUNT"
|
||||
`)
|
||||
w := cfg.Warnings()
|
||||
if len(w) != 1 || !strings.Contains(w[0], "org_url") || !strings.Contains(w[0], "$COUNT") {
|
||||
t.Errorf("Warnings() = %v, want exactly one about org_url missing $COUNT", w)
|
||||
}
|
||||
}
|
||||
|
||||
// The env var satisfies the mandatory token like the file does; a typo in the
|
||||
// key never starts the daemon unauthenticated.
|
||||
func TestLoad_TokenFromEnvOnly(t *testing.T) {
|
||||
t.Setenv("MIGADU_COUNTERS_AUTH_TOKEN", "env-token")
|
||||
cfg := mustLoad(t, "domains_cdb = \"/tmp/x\"\n"+redisStatic)
|
||||
if cfg.AuthToken != "env-token" {
|
||||
t.Errorf("auth_token = %q", cfg.AuthToken)
|
||||
}
|
||||
if _, err := Load(writeConfig(t, "domains_cdb = \"/tmp/x\"\nauth-token = \"x\"\n"+redisStatic)); err == nil {
|
||||
t.Error("typo key must be fatal even when the env var supplies a token")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoad_QueueAndCache(t *testing.T) {
|
||||
cfg := mustLoad(t, top+"queue_size = 50\nworkers = 2\nmax_cached_domains = 0\n"+redisStatic)
|
||||
if cfg.QueueSize != 50 || cfg.Workers != 2 || cfg.MaxCachedDomains != 0 {
|
||||
t.Errorf("queue/cache = %d %d %d", cfg.QueueSize, cfg.Workers, cfg.MaxCachedDomains)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiscover(t *testing.T) {
|
||||
if got, err := Discover("/explicit/path.toml"); err != nil || got != "/explicit/path.toml" {
|
||||
t.Errorf("Discover(explicit) = %q, %v", got, err)
|
||||
}
|
||||
// With no default file present on the test machine, Discover must explain where it looked.
|
||||
paths := DefaultPaths
|
||||
DefaultPaths = []string{filepath.Join(t.TempDir(), "missing.toml")}
|
||||
defer func() { DefaultPaths = paths }()
|
||||
if _, err := Discover(""); err == nil || !strings.Contains(err.Error(), "missing.toml") {
|
||||
t.Errorf("Discover(\"\") err = %v, want one listing the searched paths", err)
|
||||
}
|
||||
found := writeConfig(t, minimal)
|
||||
DefaultPaths = []string{DefaultPaths[0], found}
|
||||
if got, err := Discover(""); err != nil || got != found {
|
||||
t.Errorf("Discover(\"\") = %q, %v; want %q", got, err, found)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,116 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// LogWriter wraps a log destination and supports reopening for log rotation.
|
||||
// When the output is a file, Reopen closes the old handle and opens a new one
|
||||
// so that newsyslog rotation works after SIGHUP. Shared by the application log
|
||||
// and the counting transaction log. (Same implementation as rcptd.)
|
||||
type LogWriter struct {
|
||||
mu sync.Mutex
|
||||
path string // empty for stdout/stderr
|
||||
file *os.File // current output file (or os.Stdout/os.Stderr)
|
||||
}
|
||||
|
||||
func (w *LogWriter) Write(p []byte) (n int, err error) {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
return w.file.Write(p)
|
||||
}
|
||||
|
||||
// Reopen closes and reopens the log file. No-op for stdout/stderr.
|
||||
func (w *LogWriter) Reopen() error {
|
||||
if w.path == "" {
|
||||
return nil
|
||||
}
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
newFile, err := os.OpenFile(w.path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o644)
|
||||
if err != nil {
|
||||
return fmt.Errorf("reopen log file %s: %w", w.path, err)
|
||||
}
|
||||
w.file.Close()
|
||||
w.file = newFile
|
||||
return nil
|
||||
}
|
||||
|
||||
// Close closes the log file. No-op for stdout/stderr.
|
||||
func (w *LogWriter) Close() error {
|
||||
if w.path == "" {
|
||||
return nil
|
||||
}
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
return w.file.Close()
|
||||
}
|
||||
|
||||
// NewWriter opens a log destination: "stderr", "stdout" or a file path.
|
||||
func NewWriter(output string) (*LogWriter, error) { return newLogWriter(output) }
|
||||
|
||||
// NewLogger creates the application slog.Logger from cfg and installs it as
|
||||
// the slog default. The returned LogWriter must be closed by the caller.
|
||||
func NewLogger(cfg LoggingConfig) (*slog.Logger, *LogWriter, error) {
|
||||
level, err := parseLogLevel(cfg.Level)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("invalid log level: %w", err)
|
||||
}
|
||||
|
||||
writer, err := newLogWriter(cfg.Output)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("failed to create log writer: %w", err)
|
||||
}
|
||||
|
||||
var handler slog.Handler
|
||||
opts := &slog.HandlerOptions{Level: level}
|
||||
|
||||
switch strings.ToLower(cfg.Format) {
|
||||
case "json":
|
||||
handler = slog.NewJSONHandler(writer, opts)
|
||||
case "console", "":
|
||||
handler = slog.NewTextHandler(writer, opts)
|
||||
default:
|
||||
writer.Close()
|
||||
return nil, nil, fmt.Errorf("invalid log format: %s (must be 'console' or 'json')", cfg.Format)
|
||||
}
|
||||
|
||||
logger := slog.New(handler)
|
||||
slog.SetDefault(logger)
|
||||
|
||||
return logger, writer, nil
|
||||
}
|
||||
|
||||
func parseLogLevel(level string) (slog.Level, error) {
|
||||
switch strings.ToLower(level) {
|
||||
case "debug":
|
||||
return slog.LevelDebug, nil
|
||||
case "info", "":
|
||||
return slog.LevelInfo, nil
|
||||
case "warn", "warning":
|
||||
return slog.LevelWarn, nil
|
||||
case "error":
|
||||
return slog.LevelError, nil
|
||||
default:
|
||||
return slog.LevelInfo, fmt.Errorf("unknown level: %s (must be debug, info, warn, or error)", level)
|
||||
}
|
||||
}
|
||||
|
||||
func newLogWriter(output string) (*LogWriter, error) {
|
||||
switch strings.ToLower(output) {
|
||||
case "stderr", "":
|
||||
return &LogWriter{file: os.Stderr}, nil
|
||||
case "stdout":
|
||||
return &LogWriter{file: os.Stdout}, nil
|
||||
default:
|
||||
file, err := os.OpenFile(output, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o644)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to open log file %s: %w", output, err)
|
||||
}
|
||||
return &LogWriter{path: output, file: file}, nil
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestNewLogger_Invalid(t *testing.T) {
|
||||
if _, _, err := NewLogger(LoggingConfig{Format: "xml"}); err == nil {
|
||||
t.Error("expected error for format xml")
|
||||
}
|
||||
if _, _, err := NewLogger(LoggingConfig{Level: "loud"}); err == nil {
|
||||
t.Error("expected error for level loud")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewLogger_Levels(t *testing.T) {
|
||||
for _, tt := range []struct {
|
||||
in string
|
||||
want slog.Level
|
||||
}{{"", slog.LevelInfo}, {"debug", slog.LevelDebug}, {"INFO", slog.LevelInfo}, {"warn", slog.LevelWarn}, {"warning", slog.LevelWarn}, {"error", slog.LevelError}} {
|
||||
got, err := parseLogLevel(tt.in)
|
||||
if err != nil || got != tt.want {
|
||||
t.Errorf("parseLogLevel(%q) = %v, %v; want %v", tt.in, got, err, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Rotation: after the file is renamed away, Reopen must create a fresh file at
|
||||
// the configured path and subsequent writes must land there.
|
||||
func TestLogWriter_Reopen(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "app.log")
|
||||
w, err := newLogWriter(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer w.Close()
|
||||
if _, err := w.Write([]byte("one\n")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rotated := filepath.Join(dir, "app.log.0")
|
||||
if err := os.Rename(path, rotated); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w.Reopen(); err != nil {
|
||||
t.Fatalf("Reopen: %v", err)
|
||||
}
|
||||
if _, err := w.Write([]byte("two\n")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
old, _ := os.ReadFile(rotated)
|
||||
fresh, _ := os.ReadFile(path)
|
||||
if strings.TrimSpace(string(old)) != "one" || strings.TrimSpace(string(fresh)) != "two" {
|
||||
t.Errorf("rotated=%q fresh=%q", old, fresh)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLogWriter_StderrReopenIsNoop(t *testing.T) {
|
||||
w, err := newLogWriter("stderr")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w.Reopen(); err != nil {
|
||||
t.Errorf("Reopen on stderr: %v", err)
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
t.Errorf("Close on stderr: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,345 @@
|
||||
// Package counting turns one ingested recipient into the counter increments
|
||||
// it causes, and executes them against the store.
|
||||
//
|
||||
// Rules (see plans/2026-10-13-migadu-counters.md):
|
||||
//
|
||||
// - /out (authenticated submission): the sender's A/D/O outgoing counters
|
||||
// +1 when the sender's domain is hosted here, and the recipient's A/D/O
|
||||
// incoming counters +1 when the recipient's domain is hosted here.
|
||||
// - /in (inbound from outside): the recipient's A/D/O incoming counters +1
|
||||
// when hosted here; the sender is never counted; junk is skipped unless
|
||||
// configured otherwise.
|
||||
// - Mail to or from an exempt address (support@, contact@) never counts.
|
||||
// - A domain that is not in the domains CDB is not counted.
|
||||
//
|
||||
// The Planner is pure (no I/O besides the domain lookup) so the rules are
|
||||
// testable without Redis; Counter runs a Plan's increments in one pipeline.
|
||||
package counting
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"migadu_counters/internal/domains"
|
||||
"migadu_counters/internal/store"
|
||||
)
|
||||
|
||||
// Routes and skip reasons as reported in responses and the transaction log.
|
||||
const (
|
||||
RouteIn = "in"
|
||||
RouteOut = "out"
|
||||
|
||||
SkipExempt = "exempt"
|
||||
SkipJunk = "junk"
|
||||
SkipUnknownDomain = "unknown_domain"
|
||||
)
|
||||
|
||||
// Lookuper resolves a domain to its CDB record; *domains.DB implements it.
|
||||
type Lookuper interface {
|
||||
Lookup(domain string) (*domains.Domain, error)
|
||||
}
|
||||
|
||||
// LookupFunc adapts a function (e.g. one reading an atomic.Pointer to the
|
||||
// current DB) to Lookuper.
|
||||
type LookupFunc func(domain string) (*domains.Domain, error)
|
||||
|
||||
// Lookup implements Lookuper.
|
||||
func (f LookupFunc) Lookup(domain string) (*domains.Domain, error) { return f(domain) }
|
||||
|
||||
// Planner applies the counting rules.
|
||||
type Planner struct {
|
||||
Lookup Lookuper
|
||||
Exempt map[string]bool // lowercased addresses that never count
|
||||
CountJunk bool // count /in requests flagged junk
|
||||
Now func() time.Time // nil = time.Now; the UTC day of Now() is the counter day
|
||||
}
|
||||
|
||||
// Job is one validated request handed to the queue: the fields the counting
|
||||
// rules need and nothing else. A side that failed validation is empty, with
|
||||
// the reason in SenderUnusable / RecipientUnusable for the log.
|
||||
type Job struct {
|
||||
Route string // RouteIn or RouteOut
|
||||
TraceID string
|
||||
Received time.Time // when the request arrived (queue wait metric)
|
||||
|
||||
AuthUser string // /out: normalised SMTP AUTH user, "" when not usable
|
||||
LoginSuffix string // /out: "SUPPORTTOKEN" of a user@dom@SUPPORTTOKEN login
|
||||
MailFrom string // normalised, or the sanitised input when not an address
|
||||
// MailFromUsable tells whether MailFrom is a normalised address (and may
|
||||
// match the exempt list); an unusable value is never exempt.
|
||||
MailFromUsable bool
|
||||
Recipient string // normalised, "" when not usable
|
||||
Junk bool // /in only
|
||||
|
||||
SenderUnusable Unusable // /out: auth_user not usable (Value == "" when usable)
|
||||
RecipientUnusable Unusable // recipient not usable
|
||||
}
|
||||
|
||||
// Unusable records an address side that could not be counted.
|
||||
type Unusable struct {
|
||||
Role string // "sender" or "recipient"
|
||||
Value string // sanitised input, "" when the field was missing
|
||||
Reason string // "missing" or "invalid e-mail address"
|
||||
}
|
||||
|
||||
// Set reports whether a side was recorded as unusable.
|
||||
func (u Unusable) Set() bool { return u.Role != "" }
|
||||
|
||||
// Lookup records one domain resolution for the transaction log.
|
||||
type Lookup struct {
|
||||
Role string // "sender" or "recipient"
|
||||
Address string
|
||||
DomainName string
|
||||
Domain *domains.Domain // nil when not found
|
||||
Err error // domains.ErrNotFound when not hosted here
|
||||
}
|
||||
|
||||
// Increment is one counter the request bumps.
|
||||
type Increment struct {
|
||||
Level string // A, D or O
|
||||
ID string // address, domain name or organisation id
|
||||
Flow string // i or o
|
||||
Key string // Redis key for the day
|
||||
By string // address that caused it (sender for o, recipient for i); $ADDRESS in notifications
|
||||
Limit int64 // configured limit, 0 when unlimited
|
||||
Limited bool // false when no limit is configured for this level/flow
|
||||
Domain *domains.Domain
|
||||
}
|
||||
|
||||
// Plan is the result of applying the rules to one request.
|
||||
type Plan struct {
|
||||
Route string
|
||||
TraceID string
|
||||
Date string // UTC day used in the keys
|
||||
Skipped string // "" or one of the Skip* reasons
|
||||
AuthUser string
|
||||
LoginSuffix string
|
||||
MailFrom string
|
||||
Recipient string
|
||||
Junk bool // /in only: the message was flagged junk by mizu
|
||||
|
||||
Unusable []Unusable // sides that were not counted because they failed validation
|
||||
Lookups []Lookup
|
||||
Increments []Increment
|
||||
}
|
||||
|
||||
func (p *Planner) now() time.Time {
|
||||
if p.Now != nil {
|
||||
return p.Now()
|
||||
}
|
||||
return time.Now()
|
||||
}
|
||||
|
||||
func (p *Planner) isExempt(addr string) bool { return addr != "" && p.Exempt[addr] }
|
||||
|
||||
func normalize(addr string) string { return strings.ToLower(strings.TrimSpace(addr)) }
|
||||
|
||||
// splitAddress returns local part and domain; ok is false without an "@".
|
||||
func splitAddress(addr string) (local, domain string, ok bool) {
|
||||
i := strings.LastIndex(addr, "@")
|
||||
if i <= 0 || i == len(addr)-1 {
|
||||
return "", "", false
|
||||
}
|
||||
return addr[:i], addr[i+1:], true
|
||||
}
|
||||
|
||||
// PlanOut applies the /out rules for one authenticated message recipient
|
||||
// whose fields all validated.
|
||||
func (p *Planner) PlanOut(traceID, authUser, mailFrom, recipient string) (*Plan, error) {
|
||||
return p.Plan(Job{Route: RouteOut, TraceID: traceID, AuthUser: authUser, MailFrom: mailFrom, MailFromUsable: mailFrom != "", Recipient: recipient})
|
||||
}
|
||||
|
||||
// PlanIn applies the /in rules for one inbound recipient. mailFrom may be
|
||||
// empty (bounce sender <>).
|
||||
func (p *Planner) PlanIn(traceID, mailFrom, recipient string, junk bool) (*Plan, error) {
|
||||
return p.Plan(Job{Route: RouteIn, TraceID: traceID, MailFrom: mailFrom, MailFromUsable: mailFrom != "", Recipient: recipient, Junk: junk})
|
||||
}
|
||||
|
||||
// Plan applies the counting rules to a validated Job. Unusable sides are
|
||||
// recorded for the log and skipped; the usable side is counted.
|
||||
func (p *Planner) Plan(job Job) (*Plan, error) {
|
||||
authUser, mailFrom, recipient := normalize(job.AuthUser), job.MailFrom, normalize(job.Recipient)
|
||||
if job.MailFromUsable {
|
||||
mailFrom = normalize(mailFrom)
|
||||
}
|
||||
plan := &Plan{Route: job.Route, TraceID: job.TraceID, Date: store.DayKey(p.now()),
|
||||
AuthUser: authUser, LoginSuffix: job.LoginSuffix, MailFrom: mailFrom, Recipient: recipient, Junk: job.Junk}
|
||||
if job.SenderUnusable.Set() {
|
||||
plan.Unusable = append(plan.Unusable, job.SenderUnusable)
|
||||
}
|
||||
if job.RecipientUnusable.Set() {
|
||||
plan.Unusable = append(plan.Unusable, job.RecipientUnusable)
|
||||
}
|
||||
if p.isExempt(domains.Mailbox(authUser)) || (job.MailFromUsable && p.isExempt(domains.Mailbox(mailFrom))) || p.isExempt(domains.Mailbox(recipient)) {
|
||||
plan.Skipped = SkipExempt
|
||||
return plan, nil
|
||||
}
|
||||
if job.Route == RouteIn && job.Junk && !p.CountJunk {
|
||||
plan.Skipped = SkipJunk
|
||||
return plan, nil
|
||||
}
|
||||
if job.Route == RouteOut && authUser != "" {
|
||||
if err := p.add(plan, "sender", authUser, domains.FlowOut); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
if recipient != "" {
|
||||
if err := p.add(plan, "recipient", recipient, domains.FlowIn); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
if len(plan.Increments) == 0 {
|
||||
plan.Skipped = SkipUnknownDomain
|
||||
}
|
||||
return plan, nil
|
||||
}
|
||||
|
||||
// add looks up the address's domain and, when hosted here, appends its
|
||||
// A, D and O increments for flow. The address is counted as its mailbox
|
||||
// (plus-tag stripped). A not-found or malformed domain row is recorded and
|
||||
// skipped; any other lookup error (unreadable CDB) aborts the plan.
|
||||
func (p *Planner) add(plan *Plan, role, address, flow string) error {
|
||||
address = domains.Mailbox(address)
|
||||
local, domain, ok := splitAddress(address)
|
||||
lk := Lookup{Role: role, Address: address, DomainName: domain}
|
||||
if !ok {
|
||||
lk.Err = domains.ErrNotFound
|
||||
plan.Lookups = append(plan.Lookups, lk)
|
||||
return nil
|
||||
}
|
||||
d, err := p.Lookup.Lookup(domain)
|
||||
if err != nil {
|
||||
lk.Err = err
|
||||
plan.Lookups = append(plan.Lookups, lk)
|
||||
if errors.Is(err, domains.ErrNotFound) || errors.Is(err, domains.ErrMalformed) {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("lookup %s domain %s: %w", role, domain, err)
|
||||
}
|
||||
lk.Domain = d
|
||||
plan.Lookups = append(plan.Lookups, lk)
|
||||
|
||||
levels := []struct{ level, id string }{
|
||||
{domains.AddressLevel, address},
|
||||
{domains.DomainLevel, d.Name},
|
||||
{domains.OrganizationLevel, strconv.FormatInt(d.OrgID, 10)},
|
||||
}
|
||||
for _, l := range levels {
|
||||
limit, limited := d.Limits.Limit(l.level, flow, local)
|
||||
plan.Increments = append(plan.Increments, Increment{
|
||||
Level: l.level, ID: l.id, Flow: flow,
|
||||
Key: store.Key(l.level, l.id, plan.Date, flow),
|
||||
By: address, Limit: limit, Limited: limited, Domain: d,
|
||||
})
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Crossed reports whether an increment of amount that produced newValue is
|
||||
// the one that reached limit: old < limit <= new. Because INCRBY is atomic
|
||||
// exactly one increment can cross, so this is the once-per-crossing trigger
|
||||
// for notifications; a limit of 0 (or unlimited) never crosses.
|
||||
func Crossed(newValue, amount, limit int64) bool {
|
||||
return limit > 0 && newValue-amount < limit && limit <= newValue
|
||||
}
|
||||
|
||||
// Result is an executed Increment.
|
||||
type Result struct {
|
||||
Increment
|
||||
Count int64 // value after the increment (from EXEC, or re-read after a lost reply)
|
||||
Reached bool // this increment crossed the limit
|
||||
Err error // this key's own INCR failed (e.g. it holds a non-number); Count is 0
|
||||
}
|
||||
|
||||
// Outcome is an executed Plan.
|
||||
type Outcome struct {
|
||||
*Plan
|
||||
Results []Result
|
||||
// Unknown is set when the EXEC reply was lost after the write; Counts
|
||||
// were then re-read from the master and Reached means count >= limit.
|
||||
Unknown error
|
||||
}
|
||||
|
||||
// Counter plans and executes.
|
||||
type Counter struct {
|
||||
Planner *Planner
|
||||
Store *store.Store
|
||||
ReReadTimeout time.Duration // budget for the master re-read after a lost reply (default 2s)
|
||||
}
|
||||
|
||||
// CountOut plans and executes an /out request.
|
||||
func (c *Counter) CountOut(ctx context.Context, traceID, authUser, mailFrom, recipient string) (*Outcome, error) {
|
||||
plan, err := c.Planner.PlanOut(traceID, authUser, mailFrom, recipient)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return c.Execute(ctx, plan)
|
||||
}
|
||||
|
||||
// CountIn plans and executes an /in request.
|
||||
func (c *Counter) CountIn(ctx context.Context, traceID, mailFrom, recipient string, junk bool) (*Outcome, error) {
|
||||
plan, err := c.Planner.PlanIn(traceID, mailFrom, recipient, junk)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return c.Execute(ctx, plan)
|
||||
}
|
||||
|
||||
// Execute runs all increments of plan in one pipeline. A skipped plan does
|
||||
// not touch Redis.
|
||||
//
|
||||
// Failure handling (review #4): a key whose own INCR failed is reported in
|
||||
// its Result.Err while the other keys are counted normally. When the whole
|
||||
// reply was lost (timeout, reset) after the write may have been applied, the
|
||||
// keys are re-read once from the master and the outcome is flagged Unknown;
|
||||
// crossing detection then uses count >= limit so a crossing that happened in
|
||||
// the lost transaction is still notified. A definite failure (connection
|
||||
// refused, cancelled) is returned as an error: nothing was counted.
|
||||
func (c *Counter) Execute(ctx context.Context, plan *Plan) (*Outcome, error) {
|
||||
out := &Outcome{Plan: plan}
|
||||
if len(plan.Increments) == 0 {
|
||||
return out, nil
|
||||
}
|
||||
keys := make([]string, len(plan.Increments))
|
||||
for i, inc := range plan.Increments {
|
||||
keys[i] = inc.Key
|
||||
}
|
||||
out.Results = make([]Result, len(plan.Increments))
|
||||
results, err := c.Store.Incr(ctx, keys)
|
||||
if err == nil {
|
||||
for i, inc := range plan.Increments {
|
||||
r := Result{Increment: inc, Err: results[i].Err}
|
||||
if r.Err == nil {
|
||||
r.Count = results[i].Count
|
||||
r.Reached = inc.Limited && Crossed(r.Count, 1, inc.Limit)
|
||||
}
|
||||
out.Results[i] = r
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
if !store.OutcomeUnknown(err) {
|
||||
return nil, err
|
||||
}
|
||||
rctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), c.reReadTimeout())
|
||||
defer cancel()
|
||||
counts, rerr := c.Store.GetMaster(rctx, keys)
|
||||
if rerr != nil {
|
||||
return nil, fmt.Errorf("%w (outcome unknown, re-read failed: %v)", err, rerr)
|
||||
}
|
||||
out.Unknown = err
|
||||
for i, inc := range plan.Increments {
|
||||
out.Results[i] = Result{Increment: inc, Count: counts[i], Reached: inc.Limited && counts[i] >= inc.Limit}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (c *Counter) reReadTimeout() time.Duration {
|
||||
if c.ReReadTimeout > 0 {
|
||||
return c.ReReadTimeout
|
||||
}
|
||||
return 2 * time.Second
|
||||
}
|
||||
@@ -0,0 +1,677 @@
|
||||
package counting
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/alicebob/miniredis/v2"
|
||||
"github.com/redis/go-redis/v9"
|
||||
|
||||
"migadu_counters/internal/domains"
|
||||
"migadu_counters/internal/store"
|
||||
)
|
||||
|
||||
// table is a fake domains lookup: domain -> record. Unknown domains return
|
||||
// domains.ErrNotFound; the special domain "broken.tld" returns a read error.
|
||||
type table map[string]*domains.Domain
|
||||
|
||||
func (t table) Lookup(domain string) (*domains.Domain, error) {
|
||||
if domain == "broken.tld" {
|
||||
return nil, errors.New("cdb read error")
|
||||
}
|
||||
if d, ok := t[domain]; ok {
|
||||
return d, nil
|
||||
}
|
||||
return nil, domains.ErrNotFound
|
||||
}
|
||||
|
||||
var (
|
||||
fixedNow = time.Date(2026, 10, 13, 12, 0, 0, 0, time.UTC)
|
||||
day = "2026-10-13"
|
||||
exempt = map[string]bool{"support@migadu.com": true, "contact@migadu.com": true}
|
||||
)
|
||||
|
||||
func newTable() table {
|
||||
return table{
|
||||
"example.com": {Name: "example.com", OrgID: 7,
|
||||
Limits: domains.ParseLimits("O/i=10000,O/o=2200,D/i=10000,D/o=2200,A/i=500,A/o=100,noreply@/o=5")},
|
||||
"other.org": {Name: "other.org", OrgID: 9,
|
||||
Limits: domains.ParseLimits("O/i=200,O/o=20,D/i=200,D/o=20,A/i=200,A/o=20")},
|
||||
"nolimits.net": {Name: "nolimits.net", OrgID: 11, Limits: domains.Limits{}},
|
||||
}
|
||||
}
|
||||
|
||||
func newPlanner(countJunk bool) *Planner {
|
||||
return &Planner{Lookup: newTable(), Exempt: exempt, CountJunk: countJunk, Now: func() time.Time { return fixedNow }}
|
||||
}
|
||||
|
||||
type want struct {
|
||||
level, id, flow, by string
|
||||
limit int64
|
||||
limited bool
|
||||
}
|
||||
|
||||
func checkIncrements(t *testing.T, got []Increment, wants []want) {
|
||||
t.Helper()
|
||||
if len(got) != len(wants) {
|
||||
t.Fatalf("got %d increments, want %d:\n%s", len(got), len(wants), dump(got))
|
||||
}
|
||||
for i, w := range wants {
|
||||
g := got[i]
|
||||
wantKey := store.Key(w.level, w.id, day, w.flow)
|
||||
if g.Level != w.level || g.ID != w.id || g.Flow != w.flow || g.By != w.by || g.Limit != w.limit || g.Limited != w.limited || g.Key != wantKey {
|
||||
t.Errorf("increment %d = %+v,\n want level=%s id=%s flow=%s by=%s limit=%d limited=%v key=%s",
|
||||
i, g, w.level, w.id, w.flow, w.by, w.limit, w.limited, wantKey)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func dump(incs []Increment) string {
|
||||
s := ""
|
||||
for i, in := range incs {
|
||||
s += fmt.Sprintf(" %d: %+v\n", i, in)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// x@example.com -> y@example.com: sender outgoing on A/D/O, recipient incoming on A/D/O.
|
||||
func TestPlanOut_LocalRecipient(t *testing.T) {
|
||||
p := newPlanner(false)
|
||||
plan, err := p.PlanOut("tr1", "x@example.com", "x@example.com", "y@example.com")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if plan.Skipped != "" || plan.Route != "out" || plan.TraceID != "tr1" || plan.Date != day {
|
||||
t.Errorf("plan header = %+v", plan)
|
||||
}
|
||||
checkIncrements(t, plan.Increments, []want{
|
||||
{"A", "x@example.com", "o", "x@example.com", 100, true},
|
||||
{"D", "example.com", "o", "x@example.com", 2200, true},
|
||||
{"O", "7", "o", "x@example.com", 2200, true},
|
||||
{"A", "y@example.com", "i", "y@example.com", 500, true},
|
||||
{"D", "example.com", "i", "y@example.com", 10000, true},
|
||||
{"O", "7", "i", "y@example.com", 10000, true},
|
||||
})
|
||||
if len(plan.Lookups) != 2 || plan.Lookups[0].Role != "sender" || plan.Lookups[0].Domain == nil ||
|
||||
plan.Lookups[1].Role != "recipient" || plan.Lookups[1].Domain == nil {
|
||||
t.Errorf("lookups = %+v", plan.Lookups)
|
||||
}
|
||||
}
|
||||
|
||||
// x@example.com -> s@gmail.com: only the sender side; the recipient lookup is recorded as not found.
|
||||
func TestPlanOut_ExternalRecipient(t *testing.T) {
|
||||
plan, err := newPlanner(false).PlanOut("tr", "x@example.com", "x@example.com", "s@gmail.com")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
checkIncrements(t, plan.Increments, []want{
|
||||
{"A", "x@example.com", "o", "x@example.com", 100, true},
|
||||
{"D", "example.com", "o", "x@example.com", 2200, true},
|
||||
{"O", "7", "o", "x@example.com", 2200, true},
|
||||
})
|
||||
if len(plan.Lookups) != 2 || plan.Lookups[1].Domain != nil || !errors.Is(plan.Lookups[1].Err, domains.ErrNotFound) {
|
||||
t.Errorf("recipient lookup should be recorded as not found: %+v", plan.Lookups)
|
||||
}
|
||||
}
|
||||
|
||||
// Recipient in another hosted org: its own domain row decides D/O ids and limits.
|
||||
func TestPlanOut_RecipientInOtherOrg(t *testing.T) {
|
||||
plan, err := newPlanner(false).PlanOut("tr", "x@example.com", "", "z@other.org")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
checkIncrements(t, plan.Increments[3:], []want{
|
||||
{"A", "z@other.org", "i", "z@other.org", 200, true},
|
||||
{"D", "other.org", "i", "z@other.org", 200, true},
|
||||
{"O", "9", "i", "z@other.org", 200, true},
|
||||
})
|
||||
}
|
||||
|
||||
func TestPlanOut_SenderDomainUnknown(t *testing.T) {
|
||||
plan, err := newPlanner(false).PlanOut("tr", "staff@unknown.tld", "", "y@example.com")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if plan.Skipped != "" {
|
||||
t.Errorf("skipped = %q, want counting of the local recipient", plan.Skipped)
|
||||
}
|
||||
checkIncrements(t, plan.Increments, []want{
|
||||
{"A", "y@example.com", "i", "y@example.com", 500, true},
|
||||
{"D", "example.com", "i", "y@example.com", 10000, true},
|
||||
{"O", "7", "i", "y@example.com", 10000, true},
|
||||
})
|
||||
}
|
||||
|
||||
func TestPlanOut_BothUnknown(t *testing.T) {
|
||||
plan, err := newPlanner(false).PlanOut("tr", "a@unknown.tld", "", "b@gmail.com")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if plan.Skipped != SkipUnknownDomain || len(plan.Increments) != 0 {
|
||||
t.Errorf("plan = %+v", plan)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPlanOut_Exempt(t *testing.T) {
|
||||
cases := []struct{ authUser, mailFrom, recipient string }{
|
||||
{"support@migadu.com", "support@migadu.com", "y@example.com"},
|
||||
{"x@example.com", "x@example.com", "support@migadu.com"},
|
||||
{"x@example.com", "contact@migadu.com", "y@example.com"},
|
||||
{"x@example.com", "x@example.com", "Contact@Migadu.com"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
plan, err := newPlanner(false).PlanOut("tr", c.authUser, c.mailFrom, c.recipient)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if plan.Skipped != SkipExempt || len(plan.Increments) != 0 || len(plan.Lookups) != 0 {
|
||||
t.Errorf("%+v: plan = %+v, want exempt with no lookups", c, plan)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPlanOut_AddressOverrideAndUnlimited(t *testing.T) {
|
||||
plan, err := newPlanner(false).PlanOut("tr", "noreply@example.com", "", "q@nolimits.net")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
checkIncrements(t, plan.Increments, []want{
|
||||
{"A", "noreply@example.com", "o", "noreply@example.com", 5, true}, // noreply@/o override
|
||||
{"D", "example.com", "o", "noreply@example.com", 2200, true},
|
||||
{"O", "7", "o", "noreply@example.com", 2200, true},
|
||||
{"A", "q@nolimits.net", "i", "q@nolimits.net", 0, false},
|
||||
{"D", "nolimits.net", "i", "q@nolimits.net", 0, false},
|
||||
{"O", "11", "i", "q@nolimits.net", 0, false},
|
||||
})
|
||||
}
|
||||
|
||||
func TestPlanOut_NormalizesAddresses(t *testing.T) {
|
||||
plan, err := newPlanner(false).PlanOut("tr", " X@Example.COM ", "", "Y@EXAMPLE.com")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if plan.Increments[0].ID != "x@example.com" || plan.Increments[3].ID != "y@example.com" {
|
||||
t.Errorf("ids not normalised: %s", dump(plan.Increments))
|
||||
}
|
||||
}
|
||||
|
||||
func TestPlanIn_LocalRecipient(t *testing.T) {
|
||||
plan, err := newPlanner(false).PlanIn("tr9", "s@gmail.com", "y@example.com", false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if plan.Route != "in" || plan.TraceID != "tr9" || plan.Skipped != "" {
|
||||
t.Errorf("plan header = %+v", plan)
|
||||
}
|
||||
checkIncrements(t, plan.Increments, []want{
|
||||
{"A", "y@example.com", "i", "y@example.com", 500, true},
|
||||
{"D", "example.com", "i", "y@example.com", 10000, true},
|
||||
{"O", "7", "i", "y@example.com", 10000, true},
|
||||
})
|
||||
if len(plan.Lookups) != 1 || plan.Lookups[0].Role != "recipient" {
|
||||
t.Errorf("lookups = %+v", plan.Lookups)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPlanIn_SenderNeverCounted(t *testing.T) {
|
||||
// A hosted sender on /in (e.g. mail relayed back in) still only counts the recipient.
|
||||
plan, err := newPlanner(false).PlanIn("tr", "x@example.com", "z@other.org", false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, in := range plan.Increments {
|
||||
if in.Flow != "i" || in.By != "z@other.org" {
|
||||
t.Errorf("unexpected increment %+v", in)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPlanIn_UnknownDomain(t *testing.T) {
|
||||
plan, err := newPlanner(false).PlanIn("tr", "s@gmail.com", "nobody@gmail.com", false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if plan.Skipped != SkipUnknownDomain || len(plan.Increments) != 0 {
|
||||
t.Errorf("plan = %+v", plan)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPlanIn_Junk(t *testing.T) {
|
||||
plan, err := newPlanner(false).PlanIn("tr", "s@gmail.com", "y@example.com", true)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if plan.Skipped != SkipJunk || len(plan.Increments) != 0 || len(plan.Lookups) != 0 {
|
||||
t.Errorf("junk not skipped: %+v", plan)
|
||||
}
|
||||
plan, err = newPlanner(true).PlanIn("tr", "s@gmail.com", "y@example.com", true)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if plan.Skipped != "" || len(plan.Increments) != 3 {
|
||||
t.Errorf("count_junk=true should count: %+v", plan)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPlanIn_Exempt(t *testing.T) {
|
||||
for _, c := range []struct{ from, rcpt string }{{"support@migadu.com", "y@example.com"}, {"s@gmail.com", "contact@migadu.com"}} {
|
||||
plan, err := newPlanner(false).PlanIn("tr", c.from, c.rcpt, false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if plan.Skipped != SkipExempt || len(plan.Increments) != 0 {
|
||||
t.Errorf("%+v: plan = %+v", c, plan)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPlanIn_EmptyMailFromIsBounce(t *testing.T) {
|
||||
plan, err := newPlanner(false).PlanIn("tr", "", "y@example.com", false)
|
||||
if err != nil || plan.Skipped != "" || len(plan.Increments) != 3 {
|
||||
t.Errorf("bounce (<>) must still count the recipient: %+v, %v", plan, err)
|
||||
}
|
||||
}
|
||||
|
||||
// A CDB read error (not "not found") must surface so the server answers 503.
|
||||
func TestPlan_LookupError(t *testing.T) {
|
||||
if _, err := newPlanner(false).PlanIn("tr", "s@gmail.com", "y@broken.tld", false); err == nil {
|
||||
t.Error("PlanIn: expected error")
|
||||
}
|
||||
if _, err := newPlanner(false).PlanOut("tr", "x@broken.tld", "", "y@example.com"); err == nil {
|
||||
t.Error("PlanOut: expected error")
|
||||
}
|
||||
}
|
||||
|
||||
// Against the real CDB: yagolivas.com is org 1049 with A/o=20, D/o=20, O/o=20.
|
||||
func TestPlanOut_RealCDB(t *testing.T) {
|
||||
db, err := domains.Open("../../test_files/rspamd_domains_db.cdb")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
p := &Planner{Lookup: db, Exempt: exempt, Now: func() time.Time { return fixedNow }}
|
||||
plan, err := p.PlanOut("tr", "someone@yagolivas.com", "", "ext@gmail.com")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
checkIncrements(t, plan.Increments, []want{
|
||||
{"A", "someone@yagolivas.com", "o", "someone@yagolivas.com", 20, true},
|
||||
{"D", "yagolivas.com", "o", "someone@yagolivas.com", 20, true},
|
||||
{"O", "1049", "o", "someone@yagolivas.com", 20, true},
|
||||
})
|
||||
}
|
||||
|
||||
func TestCrossed(t *testing.T) {
|
||||
tests := []struct {
|
||||
newValue, amount, limit int64
|
||||
want bool
|
||||
}{
|
||||
{1, 1, 1, true},
|
||||
{5, 1, 5, true},
|
||||
{4, 1, 5, false},
|
||||
{6, 1, 5, false}, // already over: no second notification
|
||||
{5, 3, 5, true}, // 2 -> 5 crosses
|
||||
{7, 3, 5, true}, // 4 -> 7 crosses
|
||||
{8, 3, 5, false}, // 5 -> 8 was already at the limit
|
||||
{1, 1, 0, false}, // 0 / unlimited never crosses
|
||||
{1, 1, -1, false},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
if got := Crossed(tt.newValue, tt.amount, tt.limit); got != tt.want {
|
||||
t.Errorf("Crossed(%d, %d, %d) = %v, want %v", tt.newValue, tt.amount, tt.limit, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func newCounter(t *testing.T) (*Counter, *miniredis.Miniredis) {
|
||||
t.Helper()
|
||||
m := miniredis.RunT(t)
|
||||
st := store.NewWithClients(redis.NewClient(&redis.Options{Addr: m.Addr()}), nil)
|
||||
t.Cleanup(func() { st.Close() })
|
||||
return &Counter{Planner: newPlanner(false), Store: st}, m
|
||||
}
|
||||
|
||||
// Executes the plan: counts come back from Redis and the crossing increment
|
||||
// is flagged exactly once.
|
||||
func TestCounter_Out(t *testing.T) {
|
||||
c, m := newCounter(t)
|
||||
ctx := context.Background()
|
||||
// noreply@example.com has A/o=5: the fifth message reaches it.
|
||||
var last *Outcome
|
||||
for i := 1; i <= 6; i++ {
|
||||
out, err := c.CountOut(ctx, "tr", "noreply@example.com", "", "ext@gmail.com")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(out.Results) != 3 || out.Results[0].Count != int64(i) {
|
||||
t.Fatalf("message %d: results = %+v", i, out.Results)
|
||||
}
|
||||
if reached := out.Results[0].Reached; reached != (i == 5) {
|
||||
t.Errorf("message %d: A reached = %v", i, reached)
|
||||
}
|
||||
last = out
|
||||
}
|
||||
if v, _ := m.Get(store.Key("A", "noreply@example.com", day, "o")); v != "6" {
|
||||
t.Errorf("redis A/o = %q, want 6", v)
|
||||
}
|
||||
if v, _ := m.Get(store.Key("O", "7", day, "o")); v != "6" {
|
||||
t.Errorf("redis O/o = %q, want 6", v)
|
||||
}
|
||||
if last.Skipped != "" || last.Route != "out" {
|
||||
t.Errorf("outcome = %+v", last)
|
||||
}
|
||||
}
|
||||
|
||||
// After the web app resets a counter (DEL), the next crossing fires again.
|
||||
func TestCounter_ReachedAgainAfterReset(t *testing.T) {
|
||||
c, m := newCounter(t)
|
||||
ctx := context.Background()
|
||||
for i := 0; i < 5; i++ {
|
||||
if _, err := c.CountOut(ctx, "tr", "noreply@example.com", "", "ext@gmail.com"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
m.Del(store.Key("A", "noreply@example.com", day, "o"))
|
||||
var reachedAt int
|
||||
for i := 1; i <= 5; i++ {
|
||||
out, err := c.CountOut(ctx, "tr", "noreply@example.com", "", "ext@gmail.com")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if out.Results[0].Reached {
|
||||
reachedAt = i
|
||||
}
|
||||
}
|
||||
if reachedAt != 5 {
|
||||
t.Errorf("after reset the limit was reached at message %d, want 5", reachedAt)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCounter_In(t *testing.T) {
|
||||
c, m := newCounter(t)
|
||||
out, err := c.CountIn(context.Background(), "tr", "s@gmail.com", "y@example.com", false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(out.Results) != 3 || out.Results[2].Count != 1 || out.Results[2].Level != "O" {
|
||||
t.Errorf("results = %+v", out.Results)
|
||||
}
|
||||
if v, _ := m.Get(store.Key("D", "example.com", day, "i")); v != "1" {
|
||||
t.Errorf("redis D/i = %q, want 1", v)
|
||||
}
|
||||
}
|
||||
|
||||
// A skipped plan must not touch Redis at all.
|
||||
func TestCounter_SkippedDoesNotTouchRedis(t *testing.T) {
|
||||
c, m := newCounter(t)
|
||||
out, err := c.CountIn(context.Background(), "tr", "support@migadu.com", "y@example.com", false)
|
||||
if err != nil || out.Skipped != SkipExempt || len(out.Results) != 0 {
|
||||
t.Fatalf("outcome = %+v, %v", out, err)
|
||||
}
|
||||
if keys := m.Keys(); len(keys) != 0 {
|
||||
t.Errorf("redis keys = %v, want none", keys)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCounter_RedisDown(t *testing.T) {
|
||||
c, m := newCounter(t)
|
||||
m.Close()
|
||||
if _, err := c.CountIn(context.Background(), "tr", "s@gmail.com", "y@example.com", false); err == nil {
|
||||
t.Error("expected error when redis is down")
|
||||
}
|
||||
}
|
||||
|
||||
// --- Jobs with unusable sides (review #2, #3) --------------------------------
|
||||
|
||||
func TestPlan_OutSenderUnusableCountsRecipient(t *testing.T) {
|
||||
job := Job{Route: RouteOut, TraceID: "u1", Recipient: "y@example.com",
|
||||
SenderUnusable: Unusable{Role: "sender", Value: "nobody", Reason: "invalid e-mail address"}}
|
||||
plan, err := newPlanner(false).Plan(job)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
checkIncrements(t, plan.Increments, []want{
|
||||
{"A", "y@example.com", "i", "y@example.com", 500, true},
|
||||
{"D", "example.com", "i", "y@example.com", 10000, true},
|
||||
{"O", "7", "i", "y@example.com", 10000, true},
|
||||
})
|
||||
if len(plan.Unusable) != 1 || plan.Unusable[0].Role != "sender" || plan.Unusable[0].Value != "nobody" {
|
||||
t.Errorf("unusable = %+v", plan.Unusable)
|
||||
}
|
||||
if len(plan.Lookups) != 1 || plan.Lookups[0].Role != "recipient" {
|
||||
t.Errorf("only the recipient should be looked up: %+v", plan.Lookups)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPlan_OutRecipientUnusableCountsSender(t *testing.T) {
|
||||
job := Job{Route: RouteOut, TraceID: "u2", AuthUser: "x@example.com", LoginSuffix: "SUPPORTTOKEN", MailFrom: "x@example.com",
|
||||
RecipientUnusable: Unusable{Role: "recipient", Value: "u@[192.0.2.1]", Reason: "invalid e-mail address"}}
|
||||
plan, err := newPlanner(false).Plan(job)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
checkIncrements(t, plan.Increments, []want{
|
||||
{"A", "x@example.com", "o", "x@example.com", 100, true},
|
||||
{"D", "example.com", "o", "x@example.com", 2200, true},
|
||||
{"O", "7", "o", "x@example.com", 2200, true},
|
||||
})
|
||||
if plan.LoginSuffix != "SUPPORTTOKEN" || plan.AuthUser != "x@example.com" {
|
||||
t.Errorf("login suffix not carried: %+v", plan)
|
||||
}
|
||||
if len(plan.Unusable) != 1 || plan.Unusable[0].Role != "recipient" {
|
||||
t.Errorf("unusable = %+v", plan.Unusable)
|
||||
}
|
||||
}
|
||||
|
||||
// An unusable mail_from on /in is logged, treated as non-exempt, and the
|
||||
// recipient is counted.
|
||||
func TestPlan_InMailFromUnusable(t *testing.T) {
|
||||
job := Job{Route: RouteIn, TraceID: "u3", MailFrom: "john smith@x.com", Recipient: "y@example.com"}
|
||||
plan, err := newPlanner(false).Plan(job)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if plan.Skipped != "" || len(plan.Increments) != 3 || plan.MailFrom != "john smith@x.com" {
|
||||
t.Errorf("plan = %+v", plan)
|
||||
}
|
||||
}
|
||||
|
||||
// The exempt check still applies to usable sides of a Job.
|
||||
func TestPlan_ExemptViaJob(t *testing.T) {
|
||||
plan, err := newPlanner(false).Plan(Job{Route: RouteIn, MailFrom: "support@migadu.com", MailFromUsable: true, Recipient: "y@example.com"})
|
||||
if err != nil || plan.Skipped != SkipExempt {
|
||||
t.Errorf("plan = %+v, %v", plan, err)
|
||||
}
|
||||
}
|
||||
|
||||
// --- Execute: partial EXEC failure and lost replies (review #4) --------------
|
||||
|
||||
// One key holding garbage fails its own INCR; the others are counted and the
|
||||
// crossing among them is still detected.
|
||||
func TestCounter_PartialExecFailure(t *testing.T) {
|
||||
c, m := newCounter(t)
|
||||
m.Set(store.Key("O", "7", day, "o"), "garbage")
|
||||
m.Set(store.Key("A", "noreply@example.com", day, "o"), "4") // A/o=5: this increment crosses
|
||||
out, err := c.CountOut(context.Background(), "p1", "noreply@example.com", "", "ext@gmail.com")
|
||||
if err != nil {
|
||||
t.Fatalf("a per-key failure must not fail the request: %v", err)
|
||||
}
|
||||
if len(out.Results) != 3 {
|
||||
t.Fatalf("results = %+v", out.Results)
|
||||
}
|
||||
a, d, o := out.Results[0], out.Results[1], out.Results[2]
|
||||
if a.Err != nil || a.Count != 5 || !a.Reached {
|
||||
t.Errorf("A = %+v, want count 5 reached", a)
|
||||
}
|
||||
if d.Err != nil || d.Count != 1 {
|
||||
t.Errorf("D = %+v", d)
|
||||
}
|
||||
if o.Err == nil || o.Reached {
|
||||
t.Errorf("O = %+v, want its own error and no crossing", o)
|
||||
}
|
||||
if out.Unknown != nil {
|
||||
t.Errorf("outcome is known: %v", out.Unknown)
|
||||
}
|
||||
}
|
||||
|
||||
// delayProxy forwards TCP to backend and, once it has seen EXEC in the
|
||||
// request stream, holds back the next `delayed` replies by `delay`, so the
|
||||
// client's read timeout fires after Redis has applied the transaction (the
|
||||
// connection handshake replies pass undelayed).
|
||||
func delayProxy(t *testing.T, backend string, delayed int, delay time.Duration) string {
|
||||
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { ln.Close() })
|
||||
var remaining atomic.Int64
|
||||
remaining.Store(int64(delayed))
|
||||
go func() {
|
||||
for {
|
||||
c, err := ln.Accept()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
b, err := net.Dial("tcp", backend)
|
||||
if err != nil {
|
||||
c.Close()
|
||||
continue
|
||||
}
|
||||
var armed atomic.Bool
|
||||
go func() {
|
||||
defer b.Close()
|
||||
buf := make([]byte, 32*1024)
|
||||
for {
|
||||
n, err := c.Read(buf)
|
||||
if n > 0 {
|
||||
if bytes.Contains(bytes.ToLower(buf[:n]), []byte("exec")) {
|
||||
armed.Store(true)
|
||||
}
|
||||
if _, werr := b.Write(buf[:n]); werr != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
go func() {
|
||||
defer c.Close()
|
||||
buf := make([]byte, 32*1024)
|
||||
for {
|
||||
n, err := b.Read(buf)
|
||||
if n > 0 {
|
||||
if armed.Load() && remaining.Add(-1) >= 0 {
|
||||
time.Sleep(delay)
|
||||
}
|
||||
if _, werr := c.Write(buf[:n]); werr != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
}()
|
||||
return ln.Addr().String()
|
||||
}
|
||||
|
||||
// The EXEC reply is lost after Redis applied the increments: the counter is
|
||||
// re-read from the master and the crossing is still reported.
|
||||
func TestCounter_LostReplyReReadsMaster(t *testing.T) {
|
||||
m := miniredis.RunT(t)
|
||||
proxy := delayProxy(t, m.Addr(), 1, 800*time.Millisecond)
|
||||
st := store.NewWithClients(redis.NewClient(&redis.Options{Addr: proxy, ReadTimeout: 200 * time.Millisecond, PoolSize: 2}), nil)
|
||||
t.Cleanup(func() { st.Close() })
|
||||
c := &Counter{Planner: newPlanner(false), Store: st}
|
||||
m.Set(store.Key("A", "noreply@example.com", day, "o"), "4") // A/o=5
|
||||
|
||||
out, err := c.CountOut(context.Background(), "lost", "noreply@example.com", "", "ext@gmail.com")
|
||||
if err != nil {
|
||||
t.Fatalf("lost reply must be resolved by a re-read, got error: %v", err)
|
||||
}
|
||||
if out.Unknown == nil {
|
||||
t.Fatal("outcome should be flagged unknown (reply lost)")
|
||||
}
|
||||
if v, _ := m.Get(store.Key("A", "noreply@example.com", day, "o")); v != "5" {
|
||||
t.Fatalf("redis A/o = %q, the increment should have been applied", v)
|
||||
}
|
||||
if a := out.Results[0]; a.Count != 5 || !a.Reached {
|
||||
t.Errorf("re-read A = %+v, want count 5 reached", a)
|
||||
}
|
||||
if o := out.Results[2]; o.Count != 1 || o.Reached {
|
||||
t.Errorf("re-read O = %+v", o)
|
||||
}
|
||||
}
|
||||
|
||||
// A definite failure (connection refused) is still an error: nothing counted.
|
||||
func TestCounter_RefusedIsNotReRead(t *testing.T) {
|
||||
c, m := newCounter(t)
|
||||
m.Close()
|
||||
out, err := c.CountIn(context.Background(), "tr", "s@gmail.com", "y@example.com", false)
|
||||
if err == nil {
|
||||
t.Errorf("expected error, got outcome %+v", out)
|
||||
}
|
||||
}
|
||||
|
||||
// --- review #7: plus-addressing counts against the mailbox -------------------
|
||||
|
||||
func TestPlan_PlusTagCountsMailbox(t *testing.T) {
|
||||
p := newPlanner(false)
|
||||
plan, err := p.PlanOut("pt", "noreply+news@example.com", "", "y+1@example.com")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
checkIncrements(t, plan.Increments, []want{
|
||||
{"A", "noreply@example.com", "o", "noreply@example.com", 5, true}, // override noreply@/o still applies
|
||||
{"D", "example.com", "o", "noreply@example.com", 2200, true},
|
||||
{"O", "7", "o", "noreply@example.com", 2200, true},
|
||||
{"A", "y@example.com", "i", "y@example.com", 500, true},
|
||||
{"D", "example.com", "i", "y@example.com", 10000, true},
|
||||
{"O", "7", "i", "y@example.com", 10000, true},
|
||||
})
|
||||
if plan.Recipient != "y+1@example.com" || plan.AuthUser != "noreply+news@example.com" {
|
||||
t.Errorf("the request line keeps the addresses as sent: %+v", plan)
|
||||
}
|
||||
// Exempt applies to the mailbox too.
|
||||
plan, _ = p.PlanIn("pt", "s@gmail.com", "support+ticket@migadu.com", false)
|
||||
if plan.Skipped != SkipExempt {
|
||||
t.Errorf("support+ticket@ should be exempt: %+v", plan)
|
||||
}
|
||||
}
|
||||
|
||||
// --- review #8: a malformed CDB row is "not hosted", not a failure -----------
|
||||
|
||||
func TestPlan_MalformedRowIsNotHosted(t *testing.T) {
|
||||
db, err := domains.Open("../../review/_proof_tests/b/bad.cdb")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
p := &Planner{Lookup: db, Now: func() time.Time { return fixedNow }}
|
||||
plan, err := p.PlanOut("m", "a@good.com", "", "b@bad.com")
|
||||
if err != nil {
|
||||
t.Fatalf("malformed recipient row must not fail the request: %v", err)
|
||||
}
|
||||
checkIncrements(t, plan.Increments, []want{
|
||||
{"A", "a@good.com", "o", "a@good.com", 5, true},
|
||||
{"D", "good.com", "o", "a@good.com", 0, false},
|
||||
{"O", "7", "o", "a@good.com", 0, false},
|
||||
})
|
||||
if len(plan.Lookups) != 2 || plan.Lookups[1].Domain != nil || !errors.Is(plan.Lookups[1].Err, domains.ErrMalformed) {
|
||||
t.Errorf("recipient lookup should record the malformed row: %+v", plan.Lookups)
|
||||
}
|
||||
plan, err = p.PlanIn("m", "s@gmail.com", "b@bad.com", false)
|
||||
if err != nil || plan.Skipped != SkipUnknownDomain || len(plan.Increments) != 0 {
|
||||
t.Errorf("/in to a malformed row: plan=%+v err=%v, want skipped unknown_domain", plan, err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,138 @@
|
||||
package counting
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"runtime/debug"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
)
|
||||
|
||||
// QueueConfig sizes the counting queue.
|
||||
type QueueConfig struct {
|
||||
Size int // queued jobs before Enqueue refuses (default 10 000)
|
||||
Workers int // concurrent workers (default 8)
|
||||
Timeout time.Duration // budget per job: plan, Redis, log (default 4s)
|
||||
}
|
||||
|
||||
// Observer receives queue events for metrics and logging; nil funcs are ignored.
|
||||
type Observer struct {
|
||||
Depth func(n int) // queue depth after every enqueue/dequeue
|
||||
Wait func(d time.Duration) // time a job waited in the queue
|
||||
Panic func(job Job, p any, stack []byte) // a handler panicked (recovered)
|
||||
}
|
||||
|
||||
// Queue decouples counting from the HTTP request: /in and /out validate,
|
||||
// enqueue a Job and answer 202; workers run the handler with their own
|
||||
// bounded context, so a client that hangs up or times out can no longer
|
||||
// cancel a count (review #1).
|
||||
type Queue struct {
|
||||
cfg QueueConfig
|
||||
jobs chan Job
|
||||
handler func(ctx context.Context, job Job)
|
||||
obs Observer
|
||||
|
||||
pending atomic.Int64 // enqueued and not yet finished
|
||||
mu sync.RWMutex // Enqueue vs Drain (sending on a closed channel panics)
|
||||
closed bool
|
||||
wg sync.WaitGroup
|
||||
}
|
||||
|
||||
// NewQueue applies defaults and starts the workers.
|
||||
func NewQueue(cfg QueueConfig, handler func(ctx context.Context, job Job), obs Observer) *Queue {
|
||||
if cfg.Size <= 0 {
|
||||
cfg.Size = 10000
|
||||
}
|
||||
if cfg.Workers <= 0 {
|
||||
cfg.Workers = 8
|
||||
}
|
||||
if cfg.Timeout <= 0 {
|
||||
cfg.Timeout = 4 * time.Second
|
||||
}
|
||||
q := &Queue{cfg: cfg, jobs: make(chan Job, cfg.Size), handler: handler, obs: obs}
|
||||
for i := 0; i < cfg.Workers; i++ {
|
||||
q.wg.Add(1)
|
||||
go q.worker()
|
||||
}
|
||||
return q
|
||||
}
|
||||
|
||||
// Enqueue hands a job to the workers without blocking. It returns false when
|
||||
// the queue is full or draining; the caller answers 503 and logs the drop.
|
||||
func (q *Queue) Enqueue(job Job) bool {
|
||||
if job.Received.IsZero() {
|
||||
job.Received = time.Now()
|
||||
}
|
||||
q.mu.RLock()
|
||||
defer q.mu.RUnlock()
|
||||
if q.closed {
|
||||
return false
|
||||
}
|
||||
select {
|
||||
case q.jobs <- job:
|
||||
q.pending.Add(1)
|
||||
q.observeDepth()
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// Depth is the number of jobs waiting for a worker.
|
||||
func (q *Queue) Depth() int { return len(q.jobs) }
|
||||
|
||||
// Pending is the number of jobs enqueued and not yet finished (waiting or
|
||||
// in flight); tests and the shutdown log use it.
|
||||
func (q *Queue) Pending() int { return int(q.pending.Load()) }
|
||||
|
||||
// Drain stops accepting, lets the workers finish every queued job and waits
|
||||
// for them, bounded by ctx. On an expired ctx it reports how many jobs were
|
||||
// still pending.
|
||||
func (q *Queue) Drain(ctx context.Context) error {
|
||||
q.mu.Lock()
|
||||
if !q.closed {
|
||||
q.closed = true
|
||||
close(q.jobs)
|
||||
}
|
||||
q.mu.Unlock()
|
||||
done := make(chan struct{})
|
||||
go func() { q.wg.Wait(); close(done) }()
|
||||
select {
|
||||
case <-done:
|
||||
return nil
|
||||
case <-ctx.Done():
|
||||
return fmt.Errorf("counting queue: %d jobs still pending: %w", q.Pending(), ctx.Err())
|
||||
}
|
||||
}
|
||||
|
||||
func (q *Queue) worker() {
|
||||
defer q.wg.Done()
|
||||
for job := range q.jobs {
|
||||
if q.obs.Wait != nil {
|
||||
q.obs.Wait(time.Since(job.Received))
|
||||
}
|
||||
q.run(job)
|
||||
q.pending.Add(-1)
|
||||
q.observeDepth()
|
||||
}
|
||||
}
|
||||
|
||||
// run executes one job under the queue's own timeout, never a request
|
||||
// context, and keeps the worker alive when the handler panics.
|
||||
func (q *Queue) run(job Job) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), q.cfg.Timeout)
|
||||
defer cancel()
|
||||
defer func() {
|
||||
if p := recover(); p != nil && q.obs.Panic != nil {
|
||||
q.obs.Panic(job, p, debug.Stack())
|
||||
}
|
||||
}()
|
||||
q.handler(ctx, job)
|
||||
}
|
||||
|
||||
func (q *Queue) observeDepth() {
|
||||
if q.obs.Depth != nil {
|
||||
q.obs.Depth(len(q.jobs))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,179 @@
|
||||
package counting
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type recorder struct {
|
||||
mu sync.Mutex
|
||||
jobs []Job
|
||||
waits []time.Duration
|
||||
depths []int
|
||||
panics int
|
||||
}
|
||||
|
||||
func (r *recorder) handle(ctx context.Context, job Job) {
|
||||
r.mu.Lock()
|
||||
r.jobs = append(r.jobs, job)
|
||||
r.mu.Unlock()
|
||||
}
|
||||
|
||||
func (r *recorder) observer() Observer {
|
||||
return Observer{
|
||||
Depth: func(n int) { r.mu.Lock(); r.depths = append(r.depths, n); r.mu.Unlock() },
|
||||
Wait: func(d time.Duration) { r.mu.Lock(); r.waits = append(r.waits, d); r.mu.Unlock() },
|
||||
Panic: func(job Job, p any, stack []byte) { r.mu.Lock(); r.panics++; r.mu.Unlock() },
|
||||
}
|
||||
}
|
||||
|
||||
func (r *recorder) count() int {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return len(r.jobs)
|
||||
}
|
||||
|
||||
func settle(t *testing.T, q *Queue) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for q.Pending() > 0 {
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("queue did not settle: pending=%d", q.Pending())
|
||||
}
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func TestQueue_ProcessesJobs(t *testing.T) {
|
||||
r := &recorder{}
|
||||
q := NewQueue(QueueConfig{Size: 10, Workers: 2, Timeout: time.Second}, r.handle, r.observer())
|
||||
defer q.Drain(context.Background())
|
||||
for i := 0; i < 5; i++ {
|
||||
if !q.Enqueue(Job{Route: RouteIn, TraceID: "t", Recipient: "y@example.com", Received: time.Now()}) {
|
||||
t.Fatalf("enqueue %d refused", i)
|
||||
}
|
||||
}
|
||||
settle(t, q)
|
||||
if r.count() != 5 {
|
||||
t.Errorf("handled %d jobs, want 5", r.count())
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if len(r.waits) != 5 || len(r.depths) == 0 {
|
||||
t.Errorf("observer: waits=%d depths=%d", len(r.waits), len(r.depths))
|
||||
}
|
||||
for _, w := range r.waits {
|
||||
if w < 0 || w > time.Second {
|
||||
t.Errorf("queue wait %v out of range", w)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Workers run the handler with their own bounded context, never a request's.
|
||||
func TestQueue_HandlerContextHasTimeout(t *testing.T) {
|
||||
var got atomic.Value
|
||||
q := NewQueue(QueueConfig{Size: 1, Workers: 1, Timeout: 250 * time.Millisecond}, func(ctx context.Context, job Job) {
|
||||
dl, ok := ctx.Deadline()
|
||||
got.Store([2]any{ok, time.Until(dl)})
|
||||
}, Observer{})
|
||||
defer q.Drain(context.Background())
|
||||
q.Enqueue(Job{})
|
||||
settle(t, q)
|
||||
v := got.Load().([2]any)
|
||||
if !v[0].(bool) || v[1].(time.Duration) > 250*time.Millisecond || v[1].(time.Duration) < 100*time.Millisecond {
|
||||
t.Errorf("deadline = %v", v)
|
||||
}
|
||||
}
|
||||
|
||||
// A full queue refuses the job instead of blocking the HTTP handler.
|
||||
func TestQueue_FullRefuses(t *testing.T) {
|
||||
release := make(chan struct{})
|
||||
q := NewQueue(QueueConfig{Size: 2, Workers: 1, Timeout: time.Second}, func(ctx context.Context, job Job) { <-release }, Observer{})
|
||||
// One job occupies the worker ...
|
||||
if !q.Enqueue(Job{}) {
|
||||
t.Fatal("first enqueue refused")
|
||||
}
|
||||
deadline := time.Now().Add(time.Second)
|
||||
for q.Depth() != 0 && time.Now().Before(deadline) {
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
// ... two fill the channel.
|
||||
for i := 0; i < 2; i++ {
|
||||
if !q.Enqueue(Job{}) {
|
||||
t.Fatalf("enqueue %d should be accepted", i)
|
||||
}
|
||||
}
|
||||
if q.Enqueue(Job{}) {
|
||||
t.Error("enqueue on a full queue should be refused")
|
||||
}
|
||||
if q.Depth() != 2 || q.Pending() != 3 {
|
||||
t.Errorf("depth=%d pending=%d, want 2/3", q.Depth(), q.Pending())
|
||||
}
|
||||
close(release)
|
||||
if err := q.Drain(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if q.Enqueue(Job{}) {
|
||||
t.Error("enqueue after Drain must be refused, not panic")
|
||||
}
|
||||
}
|
||||
|
||||
// Drain waits for queued and in-flight jobs; a context that expires first
|
||||
// reports what was left behind.
|
||||
func TestQueue_Drain(t *testing.T) {
|
||||
r := &recorder{}
|
||||
slow := func(ctx context.Context, job Job) { time.Sleep(50 * time.Millisecond); r.handle(ctx, job) }
|
||||
q := NewQueue(QueueConfig{Size: 10, Workers: 1, Timeout: time.Second}, slow, Observer{})
|
||||
for i := 0; i < 4; i++ {
|
||||
q.Enqueue(Job{})
|
||||
}
|
||||
start := time.Now()
|
||||
if err := q.Drain(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if r.count() != 4 || time.Since(start) < 150*time.Millisecond {
|
||||
t.Errorf("Drain returned after %v with %d jobs handled, want all 4", time.Since(start), r.count())
|
||||
}
|
||||
|
||||
block := make(chan struct{})
|
||||
q2 := NewQueue(QueueConfig{Size: 10, Workers: 1, Timeout: time.Second}, func(ctx context.Context, job Job) { <-block }, Observer{})
|
||||
q2.Enqueue(Job{})
|
||||
q2.Enqueue(Job{})
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
|
||||
defer cancel()
|
||||
err := q2.Drain(ctx)
|
||||
if err == nil || !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Errorf("Drain with a stuck worker should report the deadline: %v", err)
|
||||
}
|
||||
close(block)
|
||||
}
|
||||
|
||||
// A panicking handler is recovered and reported; the worker keeps serving.
|
||||
func TestQueue_RecoversPanic(t *testing.T) {
|
||||
r := &recorder{}
|
||||
q := NewQueue(QueueConfig{Size: 10, Workers: 1, Timeout: time.Second}, func(ctx context.Context, job Job) {
|
||||
if job.TraceID == "boom" {
|
||||
panic("boom")
|
||||
}
|
||||
r.handle(ctx, job)
|
||||
}, r.observer())
|
||||
defer q.Drain(context.Background())
|
||||
q.Enqueue(Job{TraceID: "boom"})
|
||||
q.Enqueue(Job{TraceID: "ok"})
|
||||
settle(t, q)
|
||||
if r.count() != 1 || r.panics != 1 {
|
||||
t.Errorf("handled=%d panics=%d, want 1/1", r.count(), r.panics)
|
||||
}
|
||||
}
|
||||
|
||||
func TestQueue_Defaults(t *testing.T) {
|
||||
q := NewQueue(QueueConfig{}, func(context.Context, Job) {}, Observer{})
|
||||
defer q.Drain(context.Background())
|
||||
if q.cfg.Size != 10000 || q.cfg.Workers != 8 || q.cfg.Timeout != 4*time.Second {
|
||||
t.Errorf("defaults = %+v", q.cfg)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,250 @@
|
||||
// Package domains reads the rspamd domains CDB (rspamd_domains_db) that holds
|
||||
// the organisation id and the daily limits of every Migadu-hosted domain.
|
||||
//
|
||||
// The file is built by the agent's extract.sh with tinycdb (`cdb -m`) from
|
||||
// lines of the form
|
||||
//
|
||||
// example.com 1049|O/i=200,O/o=20,D/i=200,D/o=20,A/i=200,A/o=20,info@/o=5
|
||||
//
|
||||
// i.e. key = lowercase domain (no NUL terminator), value =
|
||||
// "<org_id>|<comma separated limits>". Limit keys are "<level>/<flow>" with
|
||||
// level O (organisation), D (domain) or A (address default) and flow i
|
||||
// (incoming) or o (outgoing); "<localpart>@/<flow>" overrides the A default for
|
||||
// one mailbox. A level/flow without a key is unlimited. A domain that is not
|
||||
// in the file is not hosted by Migadu and is never counted.
|
||||
package domains
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
|
||||
"github.com/colinmarc/cdb"
|
||||
)
|
||||
|
||||
// DefaultMaxCached is the number of parsed records a DB caches when opened
|
||||
// with Open; the daemon passes its max_cached_domains instead.
|
||||
const DefaultMaxCached = 20000
|
||||
|
||||
// Limit levels and flows, as used in the CDB values and the Redis keys.
|
||||
const (
|
||||
OrganizationLevel = "O"
|
||||
DomainLevel = "D"
|
||||
AddressLevel = "A"
|
||||
|
||||
FlowIn = "i"
|
||||
FlowOut = "o"
|
||||
)
|
||||
|
||||
// ErrNotFound is returned by Lookup when the domain is not in the CDB.
|
||||
var ErrNotFound = errors.New("domain not found in domains db")
|
||||
|
||||
// ErrMalformed wraps a Lookup error for a row that exists but cannot be
|
||||
// parsed (non-numeric org id). Callers treat it like "not hosted" and warn:
|
||||
// one bad generated row must never fail a request (review #8).
|
||||
var ErrMalformed = errors.New("malformed row in domains db")
|
||||
|
||||
// Mailbox strips a plus-address tag from the local part: "b+tag@d" → "b@d".
|
||||
// Counters are kept per mailbox, which is what the web app reads and what
|
||||
// rcptd resolves; a tag never names a different mailbox (review #7). An
|
||||
// address without "@", or whose local part would become empty, is returned
|
||||
// unchanged.
|
||||
func Mailbox(addr string) string {
|
||||
at := strings.LastIndex(addr, "@")
|
||||
if at <= 0 {
|
||||
return addr
|
||||
}
|
||||
plus := strings.Index(addr[:at], "+")
|
||||
if plus <= 0 {
|
||||
return addr
|
||||
}
|
||||
return addr[:plus] + addr[at:]
|
||||
}
|
||||
|
||||
// Limits maps limit keys ("O/i", "A/o", "alice@/o", ...) to their values.
|
||||
type Limits map[string]int64
|
||||
|
||||
// ParseLimits parses "O/i=5000, A/o=200, alice@/o=7" into a Limits map.
|
||||
// Keys are trimmed; pairs without "=", with an empty key or with a
|
||||
// non-numeric value are skipped rather than failing the whole record — the
|
||||
// file is generated, and one bad pair must not disable counting for a domain.
|
||||
func ParseLimits(s string) Limits {
|
||||
limits := Limits{}
|
||||
for _, pair := range strings.Split(s, ",") {
|
||||
k, v, ok := strings.Cut(pair, "=")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
k = strings.TrimSpace(k)
|
||||
if k == "" {
|
||||
continue
|
||||
}
|
||||
n, err := strconv.ParseInt(strings.TrimSpace(v), 10, 64)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
limits[k] = n
|
||||
}
|
||||
return limits
|
||||
}
|
||||
|
||||
// Limit resolves the configured limit for a level and flow, or (0, false)
|
||||
// when unlimited. At address level the per-mailbox override "<local>@/<flow>"
|
||||
// wins over the domain-wide default "A/<flow>" — same rule as the rspamd
|
||||
// plugin. localPart is ignored for the D and O levels.
|
||||
func (l Limits) Limit(level, flow, localPart string) (int64, bool) {
|
||||
switch level {
|
||||
case AddressLevel:
|
||||
if n, ok := l[localPart+"@/"+flow]; ok {
|
||||
return n, true
|
||||
}
|
||||
n, ok := l[AddressLevel+"/"+flow]
|
||||
return n, ok
|
||||
case DomainLevel, OrganizationLevel:
|
||||
n, ok := l[level+"/"+flow]
|
||||
return n, ok
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
// String renders the limits sorted by key, e.g. "{A/i=200, A/o=20, O/i=200}",
|
||||
// the limits_map notation used in the counting transaction log.
|
||||
func (l Limits) String() string {
|
||||
keys := make([]string, 0, len(l))
|
||||
for k := range l {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
var b strings.Builder
|
||||
b.WriteByte('{')
|
||||
for i, k := range keys {
|
||||
if i > 0 {
|
||||
b.WriteString(", ")
|
||||
}
|
||||
fmt.Fprintf(&b, "%s=%d", k, l[k])
|
||||
}
|
||||
b.WriteByte('}')
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// Domain is one parsed CDB record.
|
||||
type Domain struct {
|
||||
Name string
|
||||
OrgID int64
|
||||
Limits Limits
|
||||
}
|
||||
|
||||
// ParseRecord parses a CDB value "<org_id>|<limits>" for domain. A missing
|
||||
// "|" yields a domain without limits; a missing or non-numeric org id is an
|
||||
// error because the O counters cannot be keyed without it.
|
||||
func ParseRecord(domain, value string) (*Domain, error) {
|
||||
orgPart, limitsPart, _ := strings.Cut(value, "|")
|
||||
orgPart = strings.TrimSpace(orgPart)
|
||||
orgID, err := strconv.ParseInt(orgPart, 10, 64)
|
||||
if err != nil || orgID < 0 {
|
||||
return nil, fmt.Errorf("%w: domain %s: invalid org id %q", ErrMalformed, domain, orgPart)
|
||||
}
|
||||
return &Domain{Name: domain, OrgID: orgID, Limits: ParseLimits(limitsPart)}, nil
|
||||
}
|
||||
|
||||
// DB is an open domains CDB. It is safe for concurrent use.
|
||||
//
|
||||
// Parsed records are cached per DB (domain → *Domain) up to maxCached
|
||||
// entries, because some values are huge (boxmailhere.com: 193 KB, 9 798
|
||||
// overrides; ~1 ms and 1.4 MB to parse). The cache is never invalidated: a
|
||||
// reload replaces the whole DB object, and the cache goes with it. The cached
|
||||
// *Domain is shared read-only; callers must not mutate it.
|
||||
type DB struct {
|
||||
path string
|
||||
db *cdb.CDB
|
||||
count int
|
||||
maxCached int
|
||||
cache sync.Map // domain -> *Domain
|
||||
cached atomic.Int64
|
||||
}
|
||||
|
||||
// Open opens the CDB at path with the default cache size.
|
||||
func Open(path string) (*DB, error) { return OpenWithCache(path, DefaultMaxCached) }
|
||||
|
||||
// OpenWithCache opens the CDB at path and counts its records (a few
|
||||
// milliseconds for ~100k records; the count feeds the cdb_domains gauge and
|
||||
// the reload log). maxCached bounds the parsed-record cache; 0 disables it.
|
||||
func OpenWithCache(path string, maxCached int) (*DB, error) {
|
||||
c, err := cdb.Open(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open domains db %s: %w", path, err)
|
||||
}
|
||||
n := 0
|
||||
it := c.Iter()
|
||||
for it.Next() {
|
||||
n++
|
||||
}
|
||||
if err := it.Err(); err != nil {
|
||||
c.Close()
|
||||
return nil, fmt.Errorf("read domains db %s: %w", path, err)
|
||||
}
|
||||
if maxCached < 0 {
|
||||
maxCached = 0
|
||||
}
|
||||
return &DB{path: path, db: c, count: n, maxCached: maxCached}, nil
|
||||
}
|
||||
|
||||
// Lookup returns the record for domain (trimmed and lowercased first), or
|
||||
// ErrNotFound when the domain is not hosted here. The returned *Domain may
|
||||
// be shared with other callers and must be treated as read-only.
|
||||
func (d *DB) Lookup(domain string) (*Domain, error) {
|
||||
domain = strings.ToLower(strings.TrimSpace(domain))
|
||||
if domain == "" {
|
||||
return nil, ErrNotFound
|
||||
}
|
||||
if hit, ok := d.cache.Load(domain); ok {
|
||||
return hit.(*Domain), nil
|
||||
}
|
||||
v, err := d.db.Get([]byte(domain))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read domains db %s: %w", d.path, err)
|
||||
}
|
||||
if v == nil {
|
||||
return nil, ErrNotFound
|
||||
}
|
||||
rec, err := ParseRecord(domain, string(v))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if d.maxCached > 0 && d.cached.Load() < int64(d.maxCached) {
|
||||
if prev, loaded := d.cache.LoadOrStore(domain, rec); loaded {
|
||||
return prev.(*Domain), nil
|
||||
}
|
||||
d.cached.Add(1)
|
||||
}
|
||||
return rec, nil
|
||||
}
|
||||
|
||||
// Cached returns the number of parsed records currently cached.
|
||||
func (d *DB) Cached() int { return int(d.cached.Load()) }
|
||||
|
||||
// Ping reads the first record and reports an error when the file can no
|
||||
// longer be read or holds no domains at all (nothing would ever be counted).
|
||||
func (d *DB) Ping() error {
|
||||
it := d.db.Iter()
|
||||
if it.Next() {
|
||||
return nil
|
||||
}
|
||||
if err := it.Err(); err != nil {
|
||||
return fmt.Errorf("read domains db %s: %w", d.path, err)
|
||||
}
|
||||
return fmt.Errorf("domains db %s is empty", d.path)
|
||||
}
|
||||
|
||||
// Path returns the file the DB was opened from.
|
||||
func (d *DB) Path() string { return d.path }
|
||||
|
||||
// Count returns the number of records counted at Open.
|
||||
func (d *DB) Count() int { return d.count }
|
||||
|
||||
// Close closes the underlying file; in-flight Lookups then fail with an error.
|
||||
func (d *DB) Close() error { return d.db.Close() }
|
||||
@@ -0,0 +1,389 @@
|
||||
package domains
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/colinmarc/cdb"
|
||||
)
|
||||
|
||||
// All expectations below come from the real production-shaped CDB shipped in
|
||||
// test_files/, dumped with `cdb -d -m` (tinycdb). 93 549 records, every value
|
||||
// "<org_id>|O/i,O/o,D/i,D/o,A/i,A/o[,<local>@/<flow>...]".
|
||||
const testCDB = "../../test_files/rspamd_domains_db.cdb"
|
||||
|
||||
func openTestDB(t *testing.T) *DB {
|
||||
t.Helper()
|
||||
db, err := Open(testCDB)
|
||||
if err != nil {
|
||||
t.Fatalf("Open(%s): %v", testCDB, err)
|
||||
}
|
||||
t.Cleanup(func() { db.Close() })
|
||||
return db
|
||||
}
|
||||
|
||||
func mustLookup(t *testing.T, db *DB, domain string) *Domain {
|
||||
t.Helper()
|
||||
d, err := db.Lookup(domain)
|
||||
if err != nil {
|
||||
t.Fatalf("Lookup(%q): %v", domain, err)
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
func TestOpen_MissingFile(t *testing.T) {
|
||||
if _, err := Open("/nonexistent/rspamd_domains_db"); err == nil {
|
||||
t.Fatal("expected error for missing file")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDB_PathAndCount(t *testing.T) {
|
||||
db := openTestDB(t)
|
||||
if db.Path() != testCDB {
|
||||
t.Errorf("Path() = %q", db.Path())
|
||||
}
|
||||
if got := db.Count(); got != 93549 {
|
||||
t.Errorf("Count() = %d, want 93549", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLookup_RealRecord(t *testing.T) {
|
||||
db := openTestDB(t)
|
||||
d := mustLookup(t, db, "yagolivas.com")
|
||||
if d.Name != "yagolivas.com" || d.OrgID != 1049 {
|
||||
t.Fatalf("got %+v", d)
|
||||
}
|
||||
want := Limits{"O/i": 200, "O/o": 20, "D/i": 200, "D/o": 20, "A/i": 200, "A/o": 20}
|
||||
if len(d.Limits) != len(want) {
|
||||
t.Fatalf("limits = %v, want %v", d.Limits, want)
|
||||
}
|
||||
for k, v := range want {
|
||||
if d.Limits[k] != v {
|
||||
t.Errorf("limits[%s] = %d, want %d", k, d.Limits[k], v)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestLookup_NotFound(t *testing.T) {
|
||||
db := openTestDB(t)
|
||||
for _, domain := range []string{"gmail.com", "example.com", "example.org", ""} {
|
||||
_, err := db.Lookup(domain)
|
||||
if !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("Lookup(%q) err = %v, want ErrNotFound", domain, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestLookup_NormalizesCase(t *testing.T) {
|
||||
db := openTestDB(t)
|
||||
d := mustLookup(t, db, " YagoLivas.COM ")
|
||||
if d.OrgID != 1049 || d.Name != "yagolivas.com" {
|
||||
t.Errorf("got %+v", d)
|
||||
}
|
||||
}
|
||||
|
||||
// boxmailhere.com carries a 193 KB value with 9 798 per-address overrides.
|
||||
func TestLookup_HugeRecord(t *testing.T) {
|
||||
db := openTestDB(t)
|
||||
d := mustLookup(t, db, "boxmailhere.com")
|
||||
if d.OrgID != 87716 {
|
||||
t.Errorf("OrgID = %d, want 87716", d.OrgID)
|
||||
}
|
||||
if len(d.Limits) != 9804 {
|
||||
t.Errorf("len(Limits) = %d, want 9804 (6 base + 9798 overrides)", len(d.Limits))
|
||||
}
|
||||
checkLimit(t, d.Limits, AddressLevel, FlowIn, "gangstaelmo7", 500, true)
|
||||
checkLimit(t, d.Limits, AddressLevel, FlowOut, "gangstaelmo7", 30, true)
|
||||
checkLimit(t, d.Limits, AddressLevel, FlowIn, "nobody-with-override", 250000, true)
|
||||
checkLimit(t, d.Limits, OrganizationLevel, FlowOut, "", 4000, true)
|
||||
}
|
||||
|
||||
func checkLimit(t *testing.T, l Limits, level, flow, local string, want int64, wantOK bool) {
|
||||
t.Helper()
|
||||
got, ok := l.Limit(level, flow, local)
|
||||
if got != want || ok != wantOK {
|
||||
t.Errorf("Limit(%s, %s, %q) = (%d, %v), want (%d, %v)", level, flow, local, got, ok, want, wantOK)
|
||||
}
|
||||
}
|
||||
|
||||
// crisdanshopping.com.br: A/i=500,A/o=100 plus contato@ and admin@ overrides.
|
||||
func TestLimit_AddressOverride(t *testing.T) {
|
||||
db := openTestDB(t)
|
||||
d := mustLookup(t, db, "crisdanshopping.com.br")
|
||||
if d.OrgID != 56 {
|
||||
t.Fatalf("OrgID = %d", d.OrgID)
|
||||
}
|
||||
l := d.Limits
|
||||
checkLimit(t, l, AddressLevel, FlowIn, "contato", 300, true)
|
||||
checkLimit(t, l, AddressLevel, FlowOut, "contato", 150, true)
|
||||
checkLimit(t, l, AddressLevel, FlowIn, "admin", 300, true)
|
||||
checkLimit(t, l, AddressLevel, FlowOut, "admin", 150, true)
|
||||
checkLimit(t, l, AddressLevel, FlowIn, "someone", 500, true) // falls back to A/i
|
||||
checkLimit(t, l, AddressLevel, FlowOut, "someone", 100, true)
|
||||
checkLimit(t, l, DomainLevel, FlowIn, "", 1000, true)
|
||||
checkLimit(t, l, DomainLevel, FlowOut, "", 120, true)
|
||||
checkLimit(t, l, OrganizationLevel, FlowIn, "", 1000, true)
|
||||
checkLimit(t, l, OrganizationLevel, FlowOut, "", 120, true)
|
||||
// local part is irrelevant for D and O
|
||||
checkLimit(t, l, DomainLevel, FlowIn, "contato", 1000, true)
|
||||
}
|
||||
|
||||
// meu-servidor.com has overrides whose local parts contain dots.
|
||||
func TestLimit_DottedLocalPart(t *testing.T) {
|
||||
db := openTestDB(t)
|
||||
d := mustLookup(t, db, "meu-servidor.com")
|
||||
if d.OrgID != 726 {
|
||||
t.Fatalf("OrgID = %d", d.OrgID)
|
||||
}
|
||||
checkLimit(t, d.Limits, AddressLevel, FlowIn, "sexprime.xxx", 100, true)
|
||||
checkLimit(t, d.Limits, AddressLevel, FlowIn, "cf-tgamesandroid.com", 1000, true)
|
||||
checkLimit(t, d.Limits, AddressLevel, FlowOut, "cf-tgamesandroid.com", 100, true)
|
||||
checkLimit(t, d.Limits, AddressLevel, FlowIn, "alerta", 1000, true)
|
||||
checkLimit(t, d.Limits, AddressLevel, FlowOut, "alerta", 100, true)
|
||||
checkLimit(t, d.Limits, AddressLevel, FlowIn, "nobody", 500, true)
|
||||
}
|
||||
|
||||
func TestLimit_MigaduSelf(t *testing.T) {
|
||||
db := openTestDB(t)
|
||||
d := mustLookup(t, db, "migadu.com")
|
||||
if d.OrgID != 3743 {
|
||||
t.Fatalf("OrgID = %d", d.OrgID)
|
||||
}
|
||||
checkLimit(t, d.Limits, AddressLevel, FlowIn, "admin", 10000, true)
|
||||
checkLimit(t, d.Limits, AddressLevel, FlowIn, "someone", 100000, true)
|
||||
}
|
||||
|
||||
func TestLimit_Unlimited(t *testing.T) {
|
||||
l := ParseLimits("O/i=5")
|
||||
checkLimit(t, l, OrganizationLevel, FlowIn, "", 5, true)
|
||||
checkLimit(t, l, OrganizationLevel, FlowOut, "", 0, false)
|
||||
checkLimit(t, l, DomainLevel, FlowIn, "", 0, false)
|
||||
checkLimit(t, l, AddressLevel, FlowIn, "x", 0, false)
|
||||
checkLimit(t, Limits(nil), AddressLevel, FlowIn, "x", 0, false)
|
||||
checkLimit(t, l, "Z", FlowIn, "", 0, false) // unknown level
|
||||
}
|
||||
|
||||
func TestParseLimits(t *testing.T) {
|
||||
l := ParseLimits(" O/i = 5000,O/o=1000, broken ,x=notanumber,=3,A/o=,,alice@/o=7")
|
||||
want := Limits{"O/i": 5000, "O/o": 1000, "alice@/o": 7}
|
||||
if len(l) != len(want) {
|
||||
t.Fatalf("got %v, want %v", l, want)
|
||||
}
|
||||
for k, v := range want {
|
||||
if l[k] != v {
|
||||
t.Errorf("%s = %d, want %d", k, l[k], v)
|
||||
}
|
||||
}
|
||||
if got := ParseLimits(""); len(got) != 0 {
|
||||
t.Errorf("ParseLimits(\"\") = %v, want empty", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseRecord(t *testing.T) {
|
||||
tests := []struct {
|
||||
name, value string
|
||||
wantOrg int64
|
||||
wantLimits int
|
||||
wantErr bool
|
||||
}{
|
||||
{"normal", "1049|O/i=200,O/o=20", 1049, 2, false},
|
||||
{"spaces", " 7 | A/i=3 ", 7, 1, false},
|
||||
{"no pipe", "42", 42, 0, false}, // must not panic (the Lua plugin did)
|
||||
{"empty limits", "42|", 42, 0, false},
|
||||
{"non-numeric org", "abc|O/i=1", 0, 0, true},
|
||||
{"empty", "", 0, 0, true},
|
||||
{"negative org", "-1|O/i=1", 0, 0, true},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
d, err := ParseRecord("x.com", tt.value)
|
||||
if tt.wantErr {
|
||||
if err == nil {
|
||||
t.Fatalf("expected error, got %+v", d)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if d.Name != "x.com" || d.OrgID != tt.wantOrg || len(d.Limits) != tt.wantLimits {
|
||||
t.Errorf("got %+v, want org=%d limits=%d", d, tt.wantOrg, tt.wantLimits)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// String renders the Lua-like limits_map used by the transaction log, sorted.
|
||||
func TestLimits_String(t *testing.T) {
|
||||
l := ParseLimits("O/o=20,A/i=200,A/o=20,O/i=200,D/i=200,D/o=20,info@/o=5")
|
||||
want := "{A/i=200, A/o=20, D/i=200, D/o=20, O/i=200, O/o=20, info@/o=5}"
|
||||
if got := l.String(); got != want {
|
||||
t.Errorf("String() = %q, want %q", got, want)
|
||||
}
|
||||
if got := (Limits{}).String(); got != "{}" {
|
||||
t.Errorf("empty String() = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDB_Ping(t *testing.T) {
|
||||
db, err := Open(testCDB)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := db.Ping(); err != nil {
|
||||
t.Errorf("Ping() = %v", err)
|
||||
}
|
||||
db.Close()
|
||||
if err := db.Ping(); err == nil {
|
||||
t.Error("Ping() after Close() should fail")
|
||||
}
|
||||
}
|
||||
|
||||
// A CDB written with colinmarc/cdb's Writer (as tests and tooling do) must be
|
||||
// readable through the same path as the agent-built file.
|
||||
func TestLookup_WriterBuiltCDB(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "small.cdb")
|
||||
w, err := cdb.Create(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w.Put([]byte("newdomain.test"), []byte("5|A/i=1,A/o=1,D/i=2,D/o=2,O/i=3,O/o=3")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
db, err := Open(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
if db.Count() != 1 {
|
||||
t.Errorf("Count() = %d, want 1", db.Count())
|
||||
}
|
||||
d, err := db.Lookup("newdomain.test")
|
||||
if err != nil {
|
||||
t.Fatalf("Lookup: %v", err)
|
||||
}
|
||||
if d.OrgID != 5 {
|
||||
t.Errorf("OrgID = %d", d.OrgID)
|
||||
}
|
||||
if _, err := db.Lookup("other.test"); !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("Lookup(other) err = %v, want ErrNotFound", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Parsed records are cached per loaded table: a second Lookup of the same
|
||||
// domain returns the very same *Domain (no re-parse), and is safe because
|
||||
// callers never mutate it.
|
||||
func TestLookup_CachesParsedRecord(t *testing.T) {
|
||||
db := openTestDB(t)
|
||||
first := mustLookup(t, db, "boxmailhere.com")
|
||||
second := mustLookup(t, db, "BoxMailHere.com ")
|
||||
if first != second {
|
||||
t.Error("second lookup re-parsed the record instead of serving the cached *Domain")
|
||||
}
|
||||
if db.Cached() != 1 {
|
||||
t.Errorf("Cached() = %d, want 1", db.Cached())
|
||||
}
|
||||
if _, err := db.Lookup("nobody.invalid"); !errors.Is(err, ErrNotFound) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if db.Cached() != 1 {
|
||||
t.Errorf("not-found lookups must not be cached: Cached() = %d", db.Cached())
|
||||
}
|
||||
}
|
||||
|
||||
// The cache is bounded: beyond max entries lookups still work but parse
|
||||
// without caching; 0 disables the cache entirely.
|
||||
func TestLookup_CacheBounded(t *testing.T) {
|
||||
db, err := OpenWithCache(testCDB, 1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
a1 := mustLookup(t, db, "yagolivas.com")
|
||||
a2 := mustLookup(t, db, "yagolivas.com")
|
||||
b1 := mustLookup(t, db, "boxmailhere.com")
|
||||
b2 := mustLookup(t, db, "boxmailhere.com")
|
||||
if a1 != a2 {
|
||||
t.Error("first domain should be cached")
|
||||
}
|
||||
if b1 == b2 {
|
||||
t.Error("second domain exceeds max_cached_domains=1 and must not be cached")
|
||||
}
|
||||
if b1.OrgID != 87716 || db.Cached() != 1 {
|
||||
t.Errorf("uncached lookup still parses: org=%d cached=%d", b1.OrgID, db.Cached())
|
||||
}
|
||||
|
||||
off, err := OpenWithCache(testCDB, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer off.Close()
|
||||
if mustLookup(t, off, "yagolivas.com") == mustLookup(t, off, "yagolivas.com") {
|
||||
t.Error("cache disabled (0) must parse every time")
|
||||
}
|
||||
}
|
||||
|
||||
func benchLookup(b *testing.B, dom string) {
|
||||
db, err := Open(testCDB)
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
if _, err := db.Lookup(dom); err != nil { // warm the cache
|
||||
b.Fatal(err)
|
||||
}
|
||||
b.ReportAllocs()
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
if _, err := db.Lookup(dom); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// With the cache warm, the 193 KB boxmailhere.com record costs the same as a
|
||||
// small one (review finding #5: 0.9 ms and 1.4 MB per lookup before).
|
||||
func BenchmarkLookupSmall(b *testing.B) { benchLookup(b, "yagolivas.com") }
|
||||
func BenchmarkLookupHuge(b *testing.B) { benchLookup(b, "boxmailhere.com") }
|
||||
|
||||
// A row with a non-numeric org id is reported as malformed, which callers
|
||||
// treat like "not hosted" (review #8) — it must never fail the request.
|
||||
func TestLookup_MalformedRow(t *testing.T) {
|
||||
db, err := Open("../../review/_proof_tests/b/bad.cdb")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
_, err = db.Lookup("bad.com")
|
||||
if !errors.Is(err, ErrMalformed) {
|
||||
t.Fatalf("Lookup(bad.com) err = %v, want ErrMalformed", err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), `invalid org id "x12"`) {
|
||||
t.Errorf("error should name the problem: %v", err)
|
||||
}
|
||||
if d, err := db.Lookup("good.com"); err != nil || d.OrgID != 7 {
|
||||
t.Errorf("good.com = %+v, %v", d, err)
|
||||
}
|
||||
if db.Cached() != 1 {
|
||||
t.Errorf("malformed rows must not be cached: Cached() = %d", db.Cached())
|
||||
}
|
||||
}
|
||||
|
||||
func TestMailbox(t *testing.T) {
|
||||
for in, want := range map[string]string{
|
||||
"b+1@example.com": "b@example.com",
|
||||
"b+tag+more@example.com": "b@example.com",
|
||||
"b@example.com": "b@example.com",
|
||||
"+lead@example.com": "+lead@example.com", // empty local part would be nonsense: keep as is
|
||||
"nodomain": "nodomain",
|
||||
} {
|
||||
if got := Mailbox(in); got != want {
|
||||
t.Errorf("Mailbox(%q) = %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
// Package metrics defines the Prometheus metrics served on the admin port.
|
||||
package metrics
|
||||
|
||||
import (
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
)
|
||||
|
||||
// Metrics holds every collector; fields are exported for tests.
|
||||
type Metrics struct {
|
||||
Requests *prometheus.CounterVec // route, status
|
||||
Duration *prometheus.HistogramVec // route
|
||||
Increments *prometheus.CounterVec // level, flow
|
||||
LimitReached *prometheus.CounterVec // level, flow
|
||||
Notifications *prometheus.CounterVec // level, result (sent|failed|disabled)
|
||||
RedisErrors *prometheus.CounterVec // op
|
||||
CDBDomains prometheus.Gauge
|
||||
CDBReloads *prometheus.CounterVec // result (success|failure)
|
||||
Dropped *prometheus.CounterVec // route: requests refused because the counting queue was full
|
||||
QueueDepth prometheus.Gauge
|
||||
QueueWait prometheus.Histogram
|
||||
}
|
||||
|
||||
// New creates the collectors and registers them with reg.
|
||||
func New(reg prometheus.Registerer) *Metrics {
|
||||
m := &Metrics{
|
||||
Requests: prometheus.NewCounterVec(prometheus.CounterOpts{
|
||||
Name: "migadu_counters_requests_total",
|
||||
Help: "HTTP requests by route and status code.",
|
||||
}, []string{"route", "status"}),
|
||||
Duration: prometheus.NewHistogramVec(prometheus.HistogramOpts{
|
||||
Name: "migadu_counters_request_duration_seconds",
|
||||
Help: "Request latency by route.",
|
||||
Buckets: []float64{.0005, .001, .0025, .005, .01, .025, .05, .1, .25, .5, 1, 2.5},
|
||||
}, []string{"route"}),
|
||||
Increments: prometheus.NewCounterVec(prometheus.CounterOpts{
|
||||
Name: "migadu_counters_increments_total",
|
||||
Help: "Counter increments by level (A, D, O) and flow (i, o).",
|
||||
}, []string{"level", "flow"}),
|
||||
LimitReached: prometheus.NewCounterVec(prometheus.CounterOpts{
|
||||
Name: "migadu_counters_limit_reached_total",
|
||||
Help: "Increments that crossed a configured limit, by level and flow.",
|
||||
}, []string{"level", "flow"}),
|
||||
Notifications: prometheus.NewCounterVec(prometheus.CounterOpts{
|
||||
Name: "migadu_counters_notifications_total",
|
||||
Help: "Web app notifications by level and result (sent, failed, disabled).",
|
||||
}, []string{"level", "result"}),
|
||||
RedisErrors: prometheus.NewCounterVec(prometheus.CounterOpts{
|
||||
Name: "migadu_counters_redis_errors_total",
|
||||
Help: "Failed Redis operations by op (incr, mget, ping).",
|
||||
}, []string{"op"}),
|
||||
CDBDomains: prometheus.NewGauge(prometheus.GaugeOpts{
|
||||
Name: "migadu_counters_cdb_domains",
|
||||
Help: "Domains in the currently loaded domains CDB.",
|
||||
}),
|
||||
CDBReloads: prometheus.NewCounterVec(prometheus.CounterOpts{
|
||||
Name: "migadu_counters_cdb_reloads_total",
|
||||
Help: "Domains CDB reload attempts by result (success, failure).",
|
||||
}, []string{"result"}),
|
||||
Dropped: prometheus.NewCounterVec(prometheus.CounterOpts{
|
||||
Name: "migadu_counters_dropped_total",
|
||||
Help: "Validated requests refused with 503 because the counting queue was full, by route.",
|
||||
}, []string{"route"}),
|
||||
QueueDepth: prometheus.NewGauge(prometheus.GaugeOpts{
|
||||
Name: "migadu_counters_queue_depth",
|
||||
Help: "Jobs waiting in the counting queue.",
|
||||
}),
|
||||
QueueWait: prometheus.NewHistogram(prometheus.HistogramOpts{
|
||||
Name: "migadu_counters_queue_wait_seconds",
|
||||
Help: "Time a job waited in the counting queue before a worker took it.",
|
||||
Buckets: []float64{.0001, .0005, .001, .0025, .005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5},
|
||||
}),
|
||||
}
|
||||
reg.MustRegister(m.Requests, m.Duration, m.Increments, m.LimitReached, m.Notifications, m.RedisErrors, m.CDBDomains, m.CDBReloads,
|
||||
m.Dropped, m.QueueDepth, m.QueueWait)
|
||||
return m
|
||||
}
|
||||
|
||||
// RecordDropped counts a request refused because the queue was full.
|
||||
func (m *Metrics) RecordDropped(route string) { m.Dropped.WithLabelValues(route).Inc() }
|
||||
|
||||
// SetQueueDepth publishes the current queue depth.
|
||||
func (m *Metrics) SetQueueDepth(n int) { m.QueueDepth.Set(float64(n)) }
|
||||
|
||||
// ObserveQueueWait records how long a job waited for a worker.
|
||||
func (m *Metrics) ObserveQueueWait(seconds float64) { m.QueueWait.Observe(seconds) }
|
||||
|
||||
// RecordRequest counts one HTTP request and observes its latency.
|
||||
func (m *Metrics) RecordRequest(route, status string, seconds float64) {
|
||||
m.Requests.WithLabelValues(route, status).Inc()
|
||||
m.Duration.WithLabelValues(route).Observe(seconds)
|
||||
}
|
||||
|
||||
// RecordIncrement counts one counter increment.
|
||||
func (m *Metrics) RecordIncrement(level, flow string) {
|
||||
m.Increments.WithLabelValues(level, flow).Inc()
|
||||
}
|
||||
|
||||
// RecordLimitReached counts one limit crossing.
|
||||
func (m *Metrics) RecordLimitReached(level, flow string) {
|
||||
m.LimitReached.WithLabelValues(level, flow).Inc()
|
||||
}
|
||||
|
||||
// RecordNotification counts one notification outcome.
|
||||
func (m *Metrics) RecordNotification(level, result string) {
|
||||
m.Notifications.WithLabelValues(level, result).Inc()
|
||||
}
|
||||
|
||||
// RecordRedisError counts one failed Redis operation.
|
||||
func (m *Metrics) RecordRedisError(op string) { m.RedisErrors.WithLabelValues(op).Inc() }
|
||||
|
||||
// SetCDBDomains publishes the record count of the loaded CDB.
|
||||
func (m *Metrics) SetCDBDomains(n int) { m.CDBDomains.Set(float64(n)) }
|
||||
|
||||
// RecordReload counts one CDB reload attempt.
|
||||
func (m *Metrics) RecordReload(ok bool) {
|
||||
result := "failure"
|
||||
if ok {
|
||||
result = "success"
|
||||
}
|
||||
m.CDBReloads.WithLabelValues(result).Inc()
|
||||
}
|
||||
@@ -0,0 +1,111 @@
|
||||
package metrics
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"github.com/prometheus/client_golang/prometheus/testutil"
|
||||
)
|
||||
|
||||
func TestMetrics(t *testing.T) {
|
||||
reg := prometheus.NewRegistry()
|
||||
m := New(reg)
|
||||
|
||||
m.RecordRequest("out", "200", 0.001)
|
||||
m.RecordRequest("out", "200", 0.002)
|
||||
m.RecordRequest("in", "401", 0.0005)
|
||||
m.RecordIncrement("A", "o")
|
||||
m.RecordIncrement("A", "o")
|
||||
m.RecordIncrement("O", "i")
|
||||
m.RecordLimitReached("O", "o")
|
||||
m.RecordNotification("O", "sent")
|
||||
m.RecordNotification("A", "failed")
|
||||
m.RecordRedisError("incr")
|
||||
m.SetCDBDomains(93549)
|
||||
m.RecordReload(true)
|
||||
m.RecordReload(false)
|
||||
m.RecordReload(false)
|
||||
m.RecordDropped("/in")
|
||||
m.RecordDropped("/in")
|
||||
m.SetQueueDepth(7)
|
||||
m.ObserveQueueWait(0.01)
|
||||
|
||||
checks := []struct {
|
||||
name string
|
||||
c prometheus.Collector
|
||||
want float64
|
||||
}{
|
||||
{"requests out 200", m.Requests.WithLabelValues("out", "200"), 2},
|
||||
{"requests in 401", m.Requests.WithLabelValues("in", "401"), 1},
|
||||
{"increments A o", m.Increments.WithLabelValues("A", "o"), 2},
|
||||
{"increments O i", m.Increments.WithLabelValues("O", "i"), 1},
|
||||
{"limit reached O o", m.LimitReached.WithLabelValues("O", "o"), 1},
|
||||
{"notifications O sent", m.Notifications.WithLabelValues("O", "sent"), 1},
|
||||
{"notifications A failed", m.Notifications.WithLabelValues("A", "failed"), 1},
|
||||
{"redis errors incr", m.RedisErrors.WithLabelValues("incr"), 1},
|
||||
{"cdb domains", m.CDBDomains, 93549},
|
||||
{"reload success", m.CDBReloads.WithLabelValues("success"), 1},
|
||||
{"reload failure", m.CDBReloads.WithLabelValues("failure"), 2},
|
||||
{"dropped /in", m.Dropped.WithLabelValues("/in"), 2},
|
||||
{"queue depth", m.QueueDepth, 7},
|
||||
}
|
||||
for _, c := range checks {
|
||||
if got := testutil.ToFloat64(c.c); got != c.want {
|
||||
t.Errorf("%s = %v, want %v", c.name, got, c.want)
|
||||
}
|
||||
}
|
||||
|
||||
families, err := reg.Gather()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, f := range families {
|
||||
seen[f.GetName()] = true
|
||||
if f.GetName() == "migadu_counters_request_duration_seconds" {
|
||||
for _, metric := range f.GetMetric() {
|
||||
for _, l := range metric.GetLabel() {
|
||||
if l.GetName() == "route" && l.GetValue() == "out" && metric.GetHistogram().GetSampleCount() != 2 {
|
||||
t.Errorf("duration histogram for out has %d samples, want 2", metric.GetHistogram().GetSampleCount())
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, name := range []string{
|
||||
"migadu_counters_requests_total",
|
||||
"migadu_counters_request_duration_seconds",
|
||||
"migadu_counters_increments_total",
|
||||
"migadu_counters_limit_reached_total",
|
||||
"migadu_counters_notifications_total",
|
||||
"migadu_counters_redis_errors_total",
|
||||
"migadu_counters_cdb_domains",
|
||||
"migadu_counters_cdb_reloads_total",
|
||||
"migadu_counters_dropped_total",
|
||||
"migadu_counters_queue_depth",
|
||||
"migadu_counters_queue_wait_seconds",
|
||||
} {
|
||||
if !seen[name] {
|
||||
t.Errorf("metric %s not registered (have %v)", name, keys(seen))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func keys(m map[string]bool) []string {
|
||||
out := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
out = append(out, k)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Two daemons' worth of metrics (tests, counterctl) must not collide: each
|
||||
// registry gets its own collectors.
|
||||
func TestNew_SeparateRegistries(t *testing.T) {
|
||||
a := New(prometheus.NewRegistry())
|
||||
b := New(prometheus.NewRegistry())
|
||||
a.RecordRequest("in", "200", 0.1)
|
||||
if testutil.ToFloat64(b.Requests.WithLabelValues("in", "200")) != 0 {
|
||||
t.Error("registries share state")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,236 @@
|
||||
// Package notify tells the web app that a counter reached its limit.
|
||||
//
|
||||
// One URL template per level (org_url, address_url, domain_url) is filled
|
||||
// with the event and fetched with GET + basic auth, in the background, with
|
||||
// retries. The web app (migadu_web GET /api/limits) decides what happens to
|
||||
// the customer; this package only delivers the event.
|
||||
package notify
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Config holds the notification settings.
|
||||
type Config struct {
|
||||
OrgURL string // $LEVEL=O
|
||||
AddressURL string // $LEVEL=A
|
||||
DomainURL string // $LEVEL=D
|
||||
User string // basic auth; empty = no Authorization header
|
||||
Password string
|
||||
|
||||
Timeout time.Duration // per attempt (default 5s)
|
||||
Attempts int // total attempts (default 5)
|
||||
Backoff time.Duration // first retry delay, doubled each retry (default 1s)
|
||||
MaxConcurrent int // notifications in flight (default 8)
|
||||
}
|
||||
|
||||
// URLFor returns the template for a level, "" when that level is disabled.
|
||||
func (c Config) URLFor(level string) string {
|
||||
switch level {
|
||||
case "O":
|
||||
return c.OrgURL
|
||||
case "A":
|
||||
return c.AddressURL
|
||||
case "D":
|
||||
return c.DomainURL
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// URLName returns the config key that serves a level, for log lines.
|
||||
func URLName(level string) string {
|
||||
switch level {
|
||||
case "O":
|
||||
return "org_url"
|
||||
case "A":
|
||||
return "address_url"
|
||||
case "D":
|
||||
return "domain_url"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// Event is one "limit reached" occurrence.
|
||||
type Event struct {
|
||||
TraceID string
|
||||
Route string // in / out, for log correlation
|
||||
Level string // A, D or O
|
||||
ID string // address, domain or organisation id
|
||||
Flow string // i or o
|
||||
Address string // the address that triggered it ($ADDRESS / by=)
|
||||
Count int64 // counter value after the increment ($COUNT)
|
||||
Limit int64 // configured limit ($LIMIT)
|
||||
}
|
||||
|
||||
// BuildURL fills the template placeholders with URL-escaped event values.
|
||||
// $PERIOD is always "d" (per day).
|
||||
func BuildURL(tmpl string, ev Event) string {
|
||||
esc := url.QueryEscape
|
||||
return strings.NewReplacer(
|
||||
"$LEVEL", esc(ev.Level),
|
||||
"$LIMIT", strconv.FormatInt(ev.Limit, 10),
|
||||
"$ID", esc(ev.ID),
|
||||
"$FLOW", esc(ev.Flow),
|
||||
"$PERIOD", "d",
|
||||
"$ADDRESS", esc(ev.Address),
|
||||
"$COUNT", strconv.FormatInt(ev.Count, 10),
|
||||
).Replace(tmpl)
|
||||
}
|
||||
|
||||
// Hooks receive the outcome of every dispatched notification (transaction
|
||||
// log, metrics). Nil hooks are ignored.
|
||||
type Hooks struct {
|
||||
Sent func(ev Event, url string, attempt int)
|
||||
Failed func(ev Event, url string, attempts int, err error)
|
||||
}
|
||||
|
||||
// Notifier sends events in the background.
|
||||
type Notifier struct {
|
||||
cfg Config
|
||||
hooks Hooks
|
||||
client *http.Client
|
||||
sem chan struct{}
|
||||
wg sync.WaitGroup
|
||||
quit chan struct{}
|
||||
|
||||
mu sync.Mutex
|
||||
closed bool
|
||||
}
|
||||
|
||||
// New applies defaults and returns a ready Notifier.
|
||||
func New(cfg Config, hooks Hooks) *Notifier {
|
||||
if cfg.Timeout <= 0 {
|
||||
cfg.Timeout = 5 * time.Second
|
||||
}
|
||||
if cfg.Attempts <= 0 {
|
||||
cfg.Attempts = 5
|
||||
}
|
||||
if cfg.Backoff <= 0 {
|
||||
cfg.Backoff = time.Second
|
||||
}
|
||||
if cfg.MaxConcurrent <= 0 {
|
||||
cfg.MaxConcurrent = 8
|
||||
}
|
||||
return &Notifier{
|
||||
cfg: cfg,
|
||||
hooks: hooks,
|
||||
client: &http.Client{Timeout: cfg.Timeout},
|
||||
sem: make(chan struct{}, cfg.MaxConcurrent),
|
||||
quit: make(chan struct{}),
|
||||
}
|
||||
}
|
||||
|
||||
// Notify dispatches ev in the background. It returns false, without doing
|
||||
// anything, when no URL is configured for the level or after Shutdown.
|
||||
func (n *Notifier) Notify(ev Event) bool {
|
||||
tmpl := n.cfg.URLFor(ev.Level)
|
||||
if tmpl == "" {
|
||||
return false
|
||||
}
|
||||
n.mu.Lock()
|
||||
if n.closed {
|
||||
n.mu.Unlock()
|
||||
return false
|
||||
}
|
||||
n.wg.Add(1)
|
||||
n.mu.Unlock()
|
||||
|
||||
u := BuildURL(tmpl, ev)
|
||||
go func() {
|
||||
defer n.wg.Done()
|
||||
n.sem <- struct{}{}
|
||||
defer func() { <-n.sem }()
|
||||
n.send(ev, u)
|
||||
}()
|
||||
return true
|
||||
}
|
||||
|
||||
func (n *Notifier) send(ev Event, u string) {
|
||||
var lastErr error
|
||||
for attempt := 1; attempt <= n.cfg.Attempts; attempt++ {
|
||||
err, retry := n.attempt(u)
|
||||
if err == nil {
|
||||
if n.hooks.Sent != nil {
|
||||
n.hooks.Sent(ev, u, attempt)
|
||||
}
|
||||
return
|
||||
}
|
||||
lastErr = err
|
||||
if !retry {
|
||||
if n.hooks.Failed != nil {
|
||||
n.hooks.Failed(ev, u, attempt, err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if attempt < n.cfg.Attempts {
|
||||
select {
|
||||
case <-time.After(n.cfg.Backoff << (attempt - 1)):
|
||||
case <-n.quit:
|
||||
lastErr = fmt.Errorf("shutting down after attempt %d: %w", attempt, err)
|
||||
attempt = n.cfg.Attempts // stop retrying
|
||||
}
|
||||
}
|
||||
}
|
||||
if n.hooks.Failed != nil {
|
||||
n.hooks.Failed(ev, u, n.cfg.Attempts, lastErr)
|
||||
}
|
||||
}
|
||||
|
||||
// attempt performs one GET. retry is false for 4xx: wrong credentials or a
|
||||
// wrong URL will not fix themselves.
|
||||
func (n *Notifier) attempt(u string) (err error, retry bool) {
|
||||
req, err := http.NewRequest(http.MethodGet, u, nil)
|
||||
if err != nil {
|
||||
return err, false
|
||||
}
|
||||
if n.cfg.User != "" {
|
||||
req.SetBasicAuth(n.cfg.User, n.cfg.Password)
|
||||
}
|
||||
req.Header.Set("User-Agent", "migadu_counters")
|
||||
resp, err := n.client.Do(req)
|
||||
if err != nil {
|
||||
return err, true
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 4096))
|
||||
switch {
|
||||
case resp.StatusCode >= 200 && resp.StatusCode < 300:
|
||||
return nil, false
|
||||
case resp.StatusCode == http.StatusTooManyRequests || resp.StatusCode >= 500:
|
||||
return fmt.Errorf("unexpected status %d", resp.StatusCode), true
|
||||
default:
|
||||
return fmt.Errorf("unexpected status %d", resp.StatusCode), false
|
||||
}
|
||||
}
|
||||
|
||||
// Shutdown refuses new events, stops pending retries and waits for in-flight
|
||||
// requests until ctx expires. Safe to call more than once.
|
||||
func (n *Notifier) Shutdown(ctx context.Context) error {
|
||||
n.mu.Lock()
|
||||
if !n.closed {
|
||||
n.closed = true
|
||||
close(n.quit)
|
||||
}
|
||||
n.mu.Unlock()
|
||||
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
n.wg.Wait()
|
||||
close(done)
|
||||
}()
|
||||
select {
|
||||
case <-done:
|
||||
return nil
|
||||
case <-ctx.Done():
|
||||
return errors.Join(ctx.Err(), errors.New("notifications still in flight"))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,332 @@
|
||||
package notify
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
const tmpl = "https://admin.example/api/limits?level=$LEVEL&id=$ID&flow=$FLOW&period=$PERIOD&by=$ADDRESS&count=$COUNT&limit=$LIMIT"
|
||||
|
||||
func TestBuildURL(t *testing.T) {
|
||||
ev := Event{Level: "A", ID: "me@example.com", Flow: "o", Address: "me@example.com", Count: 100, Limit: 100}
|
||||
got := BuildURL(tmpl, ev)
|
||||
want := "https://admin.example/api/limits?level=A&id=me%40example.com&flow=o&period=d&by=me%40example.com&count=100&limit=100"
|
||||
if got != want {
|
||||
t.Errorf("BuildURL =\n %s\nwant\n %s", got, want)
|
||||
}
|
||||
// Dangerous characters in values are escaped, templates without placeholders pass through.
|
||||
ev.ID = "a b&c=d@x.y"
|
||||
u, err := url.Parse(BuildURL(tmpl, ev))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if u.Query().Get("id") != "a b&c=d@x.y" {
|
||||
t.Errorf("id round-trip = %q", u.Query().Get("id"))
|
||||
}
|
||||
if got := BuildURL("https://x/static", ev); got != "https://x/static" {
|
||||
t.Errorf("static template changed: %s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestURLFor(t *testing.T) {
|
||||
cfg := Config{OrgURL: "o", AddressURL: "a", DomainURL: "d"}
|
||||
for level, want := range map[string]string{"O": "o", "A": "a", "D": "d", "X": ""} {
|
||||
if got := cfg.URLFor(level); got != want {
|
||||
t.Errorf("URLFor(%s) = %q, want %q", level, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// capture is an httptest handler recording requests.
|
||||
type capture struct {
|
||||
mu sync.Mutex
|
||||
requests []*http.Request
|
||||
status []int // status to answer per call; last entry repeats
|
||||
calls atomic.Int32
|
||||
delay time.Duration
|
||||
inflight atomic.Int32
|
||||
maxSeen atomic.Int32
|
||||
}
|
||||
|
||||
func (c *capture) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
n := c.inflight.Add(1)
|
||||
for {
|
||||
m := c.maxSeen.Load()
|
||||
if n <= m || c.maxSeen.CompareAndSwap(m, n) {
|
||||
break
|
||||
}
|
||||
}
|
||||
defer c.inflight.Add(-1)
|
||||
if c.delay > 0 {
|
||||
time.Sleep(c.delay)
|
||||
}
|
||||
i := int(c.calls.Add(1)) - 1
|
||||
c.mu.Lock()
|
||||
c.requests = append(c.requests, r.Clone(context.Background()))
|
||||
c.mu.Unlock()
|
||||
status := http.StatusOK
|
||||
if len(c.status) > 0 {
|
||||
if i < len(c.status) {
|
||||
status = c.status[i]
|
||||
} else {
|
||||
status = c.status[len(c.status)-1]
|
||||
}
|
||||
}
|
||||
w.WriteHeader(status)
|
||||
}
|
||||
|
||||
type hooks struct {
|
||||
mu sync.Mutex
|
||||
sent []sentCall
|
||||
failed []failedCall
|
||||
done chan struct{}
|
||||
}
|
||||
type sentCall struct {
|
||||
ev Event
|
||||
url string
|
||||
attempt int
|
||||
}
|
||||
type failedCall struct {
|
||||
ev Event
|
||||
url string
|
||||
attempts int
|
||||
err error
|
||||
}
|
||||
|
||||
func newHooks() *hooks { return &hooks{done: make(chan struct{}, 64)} }
|
||||
|
||||
func (h *hooks) Hooks() Hooks {
|
||||
return Hooks{
|
||||
Sent: func(ev Event, url string, attempt int) {
|
||||
h.mu.Lock()
|
||||
h.sent = append(h.sent, sentCall{ev, url, attempt})
|
||||
h.mu.Unlock()
|
||||
h.done <- struct{}{}
|
||||
},
|
||||
Failed: func(ev Event, url string, attempts int, err error) {
|
||||
h.mu.Lock()
|
||||
h.failed = append(h.failed, failedCall{ev, url, attempts, err})
|
||||
h.mu.Unlock()
|
||||
h.done <- struct{}{}
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func (h *hooks) wait(t *testing.T, n int) {
|
||||
t.Helper()
|
||||
for i := 0; i < n; i++ {
|
||||
select {
|
||||
case <-h.done:
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatalf("timeout waiting for notification outcome %d/%d", i+1, n)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func fastConfig(orgURL, addrURL, domURL string) Config {
|
||||
return Config{
|
||||
OrgURL: orgURL, AddressURL: addrURL, DomainURL: domURL,
|
||||
User: "api", Password: "secret",
|
||||
Timeout: 2 * time.Second, Attempts: 5, Backoff: time.Millisecond, MaxConcurrent: 8,
|
||||
}
|
||||
}
|
||||
|
||||
func TestNotify_SendsToLevelURLWithBasicAuth(t *testing.T) {
|
||||
org, addr, dom := &capture{}, &capture{}, &capture{}
|
||||
so, sa, sd := httptest.NewServer(org), httptest.NewServer(addr), httptest.NewServer(dom)
|
||||
defer so.Close()
|
||||
defer sa.Close()
|
||||
defer sd.Close()
|
||||
q := "/api/limits?level=$LEVEL&id=$ID&flow=$FLOW&period=$PERIOD&by=$ADDRESS&count=$COUNT"
|
||||
h := newHooks()
|
||||
n := New(fastConfig(so.URL+q, sa.URL+q, sd.URL+q), h.Hooks())
|
||||
defer n.Shutdown(context.Background())
|
||||
|
||||
ev := Event{TraceID: "tr", Level: "A", ID: "me@example.com", Flow: "o", Address: "me@example.com", Count: 100, Limit: 100}
|
||||
if !n.Notify(ev) {
|
||||
t.Fatal("Notify returned false for a configured level")
|
||||
}
|
||||
h.wait(t, 1)
|
||||
|
||||
if org.calls.Load() != 0 || dom.calls.Load() != 0 || addr.calls.Load() != 1 {
|
||||
t.Fatalf("calls org=%d addr=%d dom=%d, want only addr", org.calls.Load(), addr.calls.Load(), dom.calls.Load())
|
||||
}
|
||||
r := addr.requests[0]
|
||||
if r.Method != http.MethodGet {
|
||||
t.Errorf("method = %s", r.Method)
|
||||
}
|
||||
if u, p, ok := r.BasicAuth(); !ok || u != "api" || p != "secret" {
|
||||
t.Errorf("basic auth = %q %q %v", u, p, ok)
|
||||
}
|
||||
qv := r.URL.Query()
|
||||
for k, want := range map[string]string{"level": "A", "id": "me@example.com", "flow": "o", "period": "d", "by": "me@example.com", "count": "100"} {
|
||||
if qv.Get(k) != want {
|
||||
t.Errorf("query %s = %q, want %q", k, qv.Get(k), want)
|
||||
}
|
||||
}
|
||||
if len(h.sent) != 1 || h.sent[0].attempt != 1 || h.sent[0].ev != ev || !strings.HasPrefix(h.sent[0].url, sa.URL) {
|
||||
t.Errorf("sent hook = %+v", h.sent)
|
||||
}
|
||||
|
||||
// Org and domain levels route to their own URLs.
|
||||
n.Notify(Event{Level: "O", ID: "7"})
|
||||
n.Notify(Event{Level: "D", ID: "example.com"})
|
||||
h.wait(t, 2)
|
||||
if org.calls.Load() != 1 || dom.calls.Load() != 1 {
|
||||
t.Errorf("calls org=%d dom=%d, want 1 each", org.calls.Load(), dom.calls.Load())
|
||||
}
|
||||
}
|
||||
|
||||
func TestNotify_EmptyURLSkips(t *testing.T) {
|
||||
h := newHooks()
|
||||
n := New(fastConfig("", "", ""), h.Hooks())
|
||||
if n.Notify(Event{Level: "A"}) || n.Notify(Event{Level: "Z"}) {
|
||||
t.Error("Notify should return false when no URL is configured for the level")
|
||||
}
|
||||
if err := n.Shutdown(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(h.sent)+len(h.failed) != 0 {
|
||||
t.Errorf("no hooks expected, got sent=%v failed=%v", h.sent, h.failed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNotify_NoBasicAuthWithoutUser(t *testing.T) {
|
||||
c := &capture{}
|
||||
s := httptest.NewServer(c)
|
||||
defer s.Close()
|
||||
h := newHooks()
|
||||
cfg := fastConfig(s.URL+"/l?level=$LEVEL", "", "")
|
||||
cfg.User, cfg.Password = "", ""
|
||||
n := New(cfg, h.Hooks())
|
||||
n.Notify(Event{Level: "O"})
|
||||
h.wait(t, 1)
|
||||
n.Shutdown(context.Background())
|
||||
if _, _, ok := c.requests[0].BasicAuth(); ok {
|
||||
t.Error("Authorization header sent although no user is configured")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNotify_RetriesThenSucceeds(t *testing.T) {
|
||||
c := &capture{status: []int{500, 503, 200}}
|
||||
s := httptest.NewServer(c)
|
||||
defer s.Close()
|
||||
h := newHooks()
|
||||
n := New(fastConfig(s.URL+"/l?level=$LEVEL", "", ""), h.Hooks())
|
||||
n.Notify(Event{Level: "O", ID: "7"})
|
||||
h.wait(t, 1)
|
||||
n.Shutdown(context.Background())
|
||||
if c.calls.Load() != 3 || len(h.sent) != 1 || h.sent[0].attempt != 3 || len(h.failed) != 0 {
|
||||
t.Errorf("calls=%d sent=%+v failed=%+v; want success on attempt 3", c.calls.Load(), h.sent, h.failed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNotify_GivesUpAfterAttempts(t *testing.T) {
|
||||
c := &capture{status: []int{500}}
|
||||
s := httptest.NewServer(c)
|
||||
defer s.Close()
|
||||
h := newHooks()
|
||||
n := New(fastConfig(s.URL+"/l?level=$LEVEL", "", ""), h.Hooks())
|
||||
n.Notify(Event{Level: "O", ID: "7"})
|
||||
h.wait(t, 1)
|
||||
n.Shutdown(context.Background())
|
||||
if c.calls.Load() != 5 || len(h.failed) != 1 || h.failed[0].attempts != 5 || len(h.sent) != 0 {
|
||||
t.Errorf("calls=%d failed=%+v sent=%+v; want 5 attempts then failure", c.calls.Load(), h.failed, h.sent)
|
||||
}
|
||||
if !strings.Contains(h.failed[0].err.Error(), "500") {
|
||||
t.Errorf("failure error should carry the status: %v", h.failed[0].err)
|
||||
}
|
||||
}
|
||||
|
||||
// 4xx means a configuration problem (bad credentials, wrong URL): retrying is pointless.
|
||||
func TestNotify_NoRetryOn4xx(t *testing.T) {
|
||||
c := &capture{status: []int{401}}
|
||||
s := httptest.NewServer(c)
|
||||
defer s.Close()
|
||||
h := newHooks()
|
||||
n := New(fastConfig(s.URL+"/l?level=$LEVEL", "", ""), h.Hooks())
|
||||
n.Notify(Event{Level: "O", ID: "7"})
|
||||
h.wait(t, 1)
|
||||
n.Shutdown(context.Background())
|
||||
if c.calls.Load() != 1 || len(h.failed) != 1 || h.failed[0].attempts != 1 {
|
||||
t.Errorf("calls=%d failed=%+v; want a single attempt", c.calls.Load(), h.failed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNotify_TimeoutIsRetried(t *testing.T) {
|
||||
c := &capture{delay: 300 * time.Millisecond}
|
||||
s := httptest.NewServer(c)
|
||||
defer s.Close()
|
||||
h := newHooks()
|
||||
cfg := fastConfig(s.URL+"/l?level=$LEVEL", "", "")
|
||||
cfg.Timeout = 50 * time.Millisecond
|
||||
cfg.Attempts = 2
|
||||
n := New(cfg, h.Hooks())
|
||||
n.Notify(Event{Level: "O", ID: "7"})
|
||||
h.wait(t, 1)
|
||||
n.Shutdown(context.Background())
|
||||
if len(h.failed) != 1 || h.failed[0].attempts != 2 {
|
||||
t.Errorf("failed=%+v; want failure after 2 timed-out attempts", h.failed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNotify_BoundedConcurrency(t *testing.T) {
|
||||
c := &capture{delay: 50 * time.Millisecond}
|
||||
s := httptest.NewServer(c)
|
||||
defer s.Close()
|
||||
h := newHooks()
|
||||
cfg := fastConfig(s.URL+"/l?level=$LEVEL", "", "")
|
||||
cfg.MaxConcurrent = 2
|
||||
n := New(cfg, h.Hooks())
|
||||
for i := 0; i < 10; i++ {
|
||||
n.Notify(Event{Level: "O", ID: "7"})
|
||||
}
|
||||
h.wait(t, 10)
|
||||
n.Shutdown(context.Background())
|
||||
if c.maxSeen.Load() > 2 {
|
||||
t.Errorf("max concurrent requests = %d, want <= 2", c.maxSeen.Load())
|
||||
}
|
||||
if c.calls.Load() != 10 {
|
||||
t.Errorf("calls = %d, want 10", c.calls.Load())
|
||||
}
|
||||
}
|
||||
|
||||
// Shutdown waits for in-flight notifications, bounded by the context.
|
||||
func TestNotify_ShutdownDrainsAndRespectsContext(t *testing.T) {
|
||||
c := &capture{delay: 200 * time.Millisecond}
|
||||
s := httptest.NewServer(c)
|
||||
defer s.Close()
|
||||
h := newHooks()
|
||||
n := New(fastConfig(s.URL+"/l?level=$LEVEL", "", ""), h.Hooks())
|
||||
n.Notify(Event{Level: "O", ID: "7"})
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
|
||||
defer cancel()
|
||||
if err := n.Shutdown(ctx); err == nil {
|
||||
t.Error("Shutdown with an expired context should return its error")
|
||||
}
|
||||
if err := n.Shutdown(context.Background()); err != nil {
|
||||
t.Errorf("second Shutdown: %v", err)
|
||||
}
|
||||
if c.calls.Load() != 1 || len(h.sent) != 1 {
|
||||
t.Errorf("in-flight notification was not completed: calls=%d sent=%d", c.calls.Load(), len(h.sent))
|
||||
}
|
||||
if n.Notify(Event{Level: "O"}) {
|
||||
t.Error("Notify after Shutdown should be refused")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNew_Defaults(t *testing.T) {
|
||||
n := New(Config{OrgURL: "https://x/?level=$LEVEL"}, Hooks{})
|
||||
defer n.Shutdown(context.Background())
|
||||
if n.cfg.Attempts != 5 || n.cfg.Backoff != time.Second || n.cfg.MaxConcurrent != 8 || n.cfg.Timeout != 5*time.Second {
|
||||
t.Errorf("defaults not applied: %+v", n.cfg)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,212 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"crypto/subtle"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"mime"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"runtime/debug"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"migadu_counters/internal/api"
|
||||
)
|
||||
|
||||
// maxBody bounds every request body (the biggest legitimate body is a few
|
||||
// hundred bytes).
|
||||
const maxBody = 64 << 10
|
||||
|
||||
// httpError is a rejection by the validation layer.
|
||||
type httpError struct {
|
||||
status int
|
||||
msg string
|
||||
}
|
||||
|
||||
func (e *httpError) Error() string { return e.msg }
|
||||
|
||||
func writeJSON(w http.ResponseWriter, status int, v any) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(status)
|
||||
_ = json.NewEncoder(w).Encode(v)
|
||||
}
|
||||
|
||||
func writeError(w http.ResponseWriter, status int, msg string) {
|
||||
writeJSON(w, status, api.ErrorResponse{Error: msg})
|
||||
}
|
||||
|
||||
// reject answers an httpError and logs it: a burst of 4xx from mailqueuer
|
||||
// means contract drift or a token mismatch.
|
||||
func (s *Server) reject(w http.ResponseWriter, r *http.Request, route string, e *httpError) {
|
||||
writeError(w, e.status, e.msg)
|
||||
s.d.Logger.Warn("request rejected", "route", route, "remote", r.RemoteAddr, "status", e.status, "reason", e.msg)
|
||||
}
|
||||
|
||||
// statusWriter captures the status code for metrics.
|
||||
type statusWriter struct {
|
||||
http.ResponseWriter
|
||||
status int
|
||||
}
|
||||
|
||||
func (w *statusWriter) WriteHeader(code int) {
|
||||
if w.status == 0 {
|
||||
w.status = code
|
||||
}
|
||||
w.ResponseWriter.WriteHeader(code)
|
||||
}
|
||||
|
||||
func (w *statusWriter) Write(p []byte) (int, error) {
|
||||
if w.status == 0 {
|
||||
w.status = http.StatusOK
|
||||
}
|
||||
return w.ResponseWriter.Write(p)
|
||||
}
|
||||
|
||||
// recoverer keeps the daemon alive when a handler panics. It sits outside
|
||||
// http.TimeoutHandler, which re-panics in the caller's goroutine.
|
||||
func (s *Server) recoverer(next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
defer func() {
|
||||
if p := recover(); p != nil {
|
||||
s.d.Logger.Error("panic in handler", "panic", fmt.Sprint(p), "path", r.URL.Path, "stack", string(debug.Stack()))
|
||||
writeError(w, http.StatusInternalServerError, "internal error")
|
||||
}
|
||||
}()
|
||||
next.ServeHTTP(w, r)
|
||||
})
|
||||
}
|
||||
|
||||
// observe records request metrics per route and status.
|
||||
func (s *Server) observe(next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
start := time.Now()
|
||||
sw := &statusWriter{ResponseWriter: w}
|
||||
next.ServeHTTP(sw, r)
|
||||
if sw.status == 0 {
|
||||
sw.status = http.StatusOK
|
||||
}
|
||||
route := r.URL.Path
|
||||
switch route {
|
||||
case "/in", "/out", "/counters":
|
||||
default:
|
||||
route = "other"
|
||||
}
|
||||
if s.d.Metrics != nil {
|
||||
s.d.Metrics.RecordRequest(route, strconv.Itoa(sw.status), time.Since(start).Seconds())
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// route applies the validation steps common to every endpoint, in order:
|
||||
// method, declared body size, bearer token, source IP, content type (POST).
|
||||
func (s *Server) route(path, method string, h http.HandlerFunc) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != method {
|
||||
w.Header().Set("Allow", method)
|
||||
s.reject(w, r, path, &httpError{http.StatusMethodNotAllowed, "method not allowed"})
|
||||
return
|
||||
}
|
||||
if r.ContentLength > maxBody {
|
||||
s.reject(w, r, path, &httpError{http.StatusRequestEntityTooLarge, "request body too large"})
|
||||
return
|
||||
}
|
||||
if e := s.checkAuth(r); e != nil {
|
||||
w.Header().Set("WWW-Authenticate", "Bearer")
|
||||
s.reject(w, r, path, e)
|
||||
return
|
||||
}
|
||||
if e := s.checkSource(r); e != nil {
|
||||
s.reject(w, r, path, e)
|
||||
return
|
||||
}
|
||||
if method == http.MethodPost {
|
||||
if e := checkContentType(r); e != nil {
|
||||
s.reject(w, r, path, e)
|
||||
return
|
||||
}
|
||||
r.Body = http.MaxBytesReader(w, r.Body, maxBody)
|
||||
}
|
||||
h(w, r)
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) checkAuth(r *http.Request) *httpError {
|
||||
if s.d.AuthToken == "" {
|
||||
return nil // config.Warnings() flags this at startup
|
||||
}
|
||||
const prefix = "Bearer "
|
||||
h := r.Header.Get("Authorization")
|
||||
if !strings.HasPrefix(h, prefix) {
|
||||
return &httpError{http.StatusUnauthorized, "unauthorized"}
|
||||
}
|
||||
if subtle.ConstantTimeCompare([]byte(strings.TrimSpace(h[len(prefix):])), []byte(s.d.AuthToken)) != 1 {
|
||||
return &httpError{http.StatusUnauthorized, "unauthorized"}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// checkSource matches RemoteAddr (never X-Forwarded-For: nothing proxies
|
||||
// mailqueuer's calls) against allowed_cidrs when configured.
|
||||
func (s *Server) checkSource(r *http.Request) *httpError {
|
||||
if len(s.d.AllowedCIDRs) == 0 {
|
||||
return nil
|
||||
}
|
||||
host, _, err := net.SplitHostPort(r.RemoteAddr)
|
||||
if err != nil {
|
||||
host = r.RemoteAddr
|
||||
}
|
||||
addr, err := netip.ParseAddr(host)
|
||||
if err != nil {
|
||||
return &httpError{http.StatusForbidden, "source address not allowed"}
|
||||
}
|
||||
addr = addr.Unmap()
|
||||
for _, p := range s.d.AllowedCIDRs {
|
||||
if p.Contains(addr) {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return &httpError{http.StatusForbidden, "source address not allowed"}
|
||||
}
|
||||
|
||||
func checkContentType(r *http.Request) *httpError {
|
||||
mt, _, err := mime.ParseMediaType(r.Header.Get("Content-Type"))
|
||||
if err != nil || mt != "application/json" {
|
||||
return &httpError{http.StatusUnsupportedMediaType, "Content-Type must be application/json"}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// decodeJSON reads exactly one JSON object into dst. Unknown fields are
|
||||
// ignored so mailqueuer can add fields without breaking counting.
|
||||
func decodeJSON(r *http.Request, dst any) *httpError {
|
||||
dec := json.NewDecoder(r.Body)
|
||||
if err := dec.Decode(dst); err != nil {
|
||||
var tooBig *http.MaxBytesError
|
||||
if errors.As(err, &tooBig) {
|
||||
return &httpError{http.StatusRequestEntityTooLarge, "request body too large"}
|
||||
}
|
||||
return &httpError{http.StatusBadRequest, "invalid json: " + err.Error()}
|
||||
}
|
||||
if err := dec.Decode(&struct{}{}); err != io.EOF {
|
||||
var tooBig *http.MaxBytesError
|
||||
if errors.As(err, &tooBig) {
|
||||
return &httpError{http.StatusRequestEntityTooLarge, "request body too large"}
|
||||
}
|
||||
return &httpError{http.StatusBadRequest, "body must contain exactly one JSON object"}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// logger returns the configured logger or slog's default.
|
||||
func (d Deps) logger() *slog.Logger {
|
||||
if d.Logger != nil {
|
||||
return d.Logger
|
||||
}
|
||||
return slog.Default()
|
||||
}
|
||||
@@ -0,0 +1,371 @@
|
||||
// Package server serves the API (/in, /out, /counters) behind the HTTP
|
||||
// validation layer, and the admin endpoints (/health, /metrics).
|
||||
package server
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/netip"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"github.com/prometheus/client_golang/prometheus/promhttp"
|
||||
|
||||
"migadu_counters/internal/api"
|
||||
"migadu_counters/internal/counting"
|
||||
"migadu_counters/internal/domains"
|
||||
"migadu_counters/internal/metrics"
|
||||
"migadu_counters/internal/notify"
|
||||
"migadu_counters/internal/store"
|
||||
"migadu_counters/internal/txlog"
|
||||
)
|
||||
|
||||
// Deps wires the server to the rest of the daemon.
|
||||
type Deps struct {
|
||||
AuthToken string
|
||||
AllowedCIDRs []netip.Prefix
|
||||
HandlerTimeout time.Duration // bounds the synchronous part (validation) and each queued job
|
||||
QueueSize int // counting queue capacity (default 10 000)
|
||||
Workers int // counting workers (default 8)
|
||||
Exempt map[string]bool
|
||||
|
||||
Counter *counting.Counter
|
||||
Store *store.Store
|
||||
Domains func() *domains.DB // current CDB (swapped on reload); nil = not loaded
|
||||
Notifier *notify.Notifier // nil = never notify
|
||||
TxLog *txlog.Logger // nil = transaction log disabled
|
||||
Metrics *metrics.Metrics
|
||||
Gatherer prometheus.Gatherer // served on /metrics; nil = no endpoint
|
||||
Logger *slog.Logger
|
||||
Now func() time.Time
|
||||
}
|
||||
|
||||
// Server holds the handlers and the counting queue.
|
||||
type Server struct {
|
||||
d Deps
|
||||
queue *counting.Queue
|
||||
|
||||
panicHook func() // tests: injected into the synchronous path
|
||||
}
|
||||
|
||||
// New applies defaults and starts the counting workers.
|
||||
func New(d Deps) *Server {
|
||||
if d.Now == nil {
|
||||
d.Now = time.Now
|
||||
}
|
||||
if d.HandlerTimeout <= 0 {
|
||||
d.HandlerTimeout = 4 * time.Second
|
||||
}
|
||||
d.Logger = d.logger()
|
||||
s := &Server{d: d}
|
||||
obs := counting.Observer{
|
||||
Panic: func(job counting.Job, p any, stack []byte) {
|
||||
s.d.Logger.Error("panic while counting", "panic", fmt.Sprint(p), "route", job.Route, "trace_id", job.TraceID, "stack", string(stack))
|
||||
},
|
||||
}
|
||||
if d.Metrics != nil {
|
||||
obs.Depth = d.Metrics.SetQueueDepth
|
||||
obs.Wait = func(w time.Duration) { d.Metrics.ObserveQueueWait(w.Seconds()) }
|
||||
}
|
||||
s.queue = counting.NewQueue(counting.QueueConfig{Size: d.QueueSize, Workers: d.Workers, Timeout: d.HandlerTimeout}, s.process, obs)
|
||||
return s
|
||||
}
|
||||
|
||||
// Drain stops accepting jobs and waits for the queued ones, bounded by ctx.
|
||||
// Call it after the HTTP server stopped and before the notifier shuts down.
|
||||
func (s *Server) Drain(ctx context.Context) error { return s.queue.Drain(ctx) }
|
||||
|
||||
// Pending is the number of accepted jobs not yet counted.
|
||||
func (s *Server) Pending() int { return s.queue.Pending() }
|
||||
|
||||
// Depth is the number of jobs waiting for a worker.
|
||||
func (s *Server) Depth() int { return s.queue.Depth() }
|
||||
|
||||
// NotifyHooks wires notification outcomes into the transaction log and the
|
||||
// metrics; used by the daemon and by tests.
|
||||
func NotifyHooks(tx *txlog.Logger, m *metrics.Metrics) notify.Hooks {
|
||||
return notify.Hooks{
|
||||
Sent: func(ev notify.Event, url string, attempt int) {
|
||||
if m != nil {
|
||||
m.RecordNotification(ev.Level, "sent")
|
||||
}
|
||||
if tx != nil {
|
||||
tx.NotificationSent(ev, url, attempt)
|
||||
}
|
||||
},
|
||||
Failed: func(ev notify.Event, url string, attempts int, err error) {
|
||||
if m != nil {
|
||||
m.RecordNotification(ev.Level, "failed")
|
||||
}
|
||||
if tx != nil {
|
||||
tx.NotificationFailed(ev, url, attempts, err)
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// Handler returns the API handler: recover → metrics → timeout → routes.
|
||||
func (s *Server) Handler() http.Handler {
|
||||
mux := http.NewServeMux()
|
||||
mux.Handle("/out", s.route("/out", http.MethodPost, s.handleOut))
|
||||
mux.Handle("/in", s.route("/in", http.MethodPost, s.handleIn))
|
||||
mux.Handle("/counters", s.route("/counters", http.MethodGet, s.handleCounters))
|
||||
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
|
||||
writeError(w, http.StatusNotFound, "not found")
|
||||
})
|
||||
timed := http.TimeoutHandler(mux, s.d.HandlerTimeout, `{"error":"request timeout"}`)
|
||||
return s.recoverer(s.observe(timed))
|
||||
}
|
||||
|
||||
// AdminHandler returns /health and /metrics (no auth; bind admin_addr locally).
|
||||
func (s *Server) AdminHandler() http.Handler {
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("/health", s.handleHealth)
|
||||
if s.d.Gatherer != nil {
|
||||
mux.Handle("/metrics", promhttp.HandlerFor(s.d.Gatherer, promhttp.HandlerOpts{}))
|
||||
}
|
||||
return mux
|
||||
}
|
||||
|
||||
func (s *Server) handleOut(w http.ResponseWriter, r *http.Request) {
|
||||
var req api.OutRequest
|
||||
if e := decodeJSON(r, &req); e != nil {
|
||||
s.reject(w, r, "/out", e)
|
||||
return
|
||||
}
|
||||
v, err := api.ValidateOut(req)
|
||||
if err != nil {
|
||||
s.reject(w, r, "/out", &httpError{http.StatusBadRequest, err.Error()})
|
||||
return
|
||||
}
|
||||
job := counting.Job{
|
||||
Route: counting.RouteOut, TraceID: v.TraceID, Received: s.d.Now(),
|
||||
AuthUser: v.AuthUser.Value, LoginSuffix: v.LoginSuffix,
|
||||
MailFrom: v.MailFrom, MailFromUsable: v.MailFromUsable,
|
||||
Recipient: v.Recipient.Value,
|
||||
}
|
||||
if !v.AuthUser.Usable() {
|
||||
job.SenderUnusable = counting.Unusable{Role: "sender", Value: v.AuthUser.Raw, Reason: v.AuthUser.Reason}
|
||||
}
|
||||
if !v.Recipient.Usable() {
|
||||
job.RecipientUnusable = counting.Unusable{Role: "recipient", Value: v.Recipient.Raw, Reason: v.Recipient.Reason}
|
||||
}
|
||||
s.enqueue(w, r, "/out", job)
|
||||
}
|
||||
|
||||
func (s *Server) handleIn(w http.ResponseWriter, r *http.Request) {
|
||||
var req api.InRequest
|
||||
if e := decodeJSON(r, &req); e != nil {
|
||||
s.reject(w, r, "/in", e)
|
||||
return
|
||||
}
|
||||
v, err := api.ValidateIn(req)
|
||||
if err != nil {
|
||||
s.reject(w, r, "/in", &httpError{http.StatusBadRequest, err.Error()})
|
||||
return
|
||||
}
|
||||
s.enqueue(w, r, "/in", counting.Job{
|
||||
Route: counting.RouteIn, TraceID: v.TraceID, Received: s.d.Now(),
|
||||
MailFrom: v.MailFrom, MailFromUsable: v.MailFromUsable, Recipient: v.Recipient, Junk: v.Junk,
|
||||
})
|
||||
}
|
||||
|
||||
// enqueue hands the validated job to the workers and answers 202; a full
|
||||
// queue answers 503 and the drop is counted and logged.
|
||||
func (s *Server) enqueue(w http.ResponseWriter, r *http.Request, route string, job counting.Job) {
|
||||
if s.panicHook != nil {
|
||||
s.panicHook()
|
||||
}
|
||||
if !s.queue.Enqueue(job) {
|
||||
if s.d.Metrics != nil {
|
||||
s.d.Metrics.RecordDropped(route)
|
||||
}
|
||||
if s.d.TxLog != nil {
|
||||
s.d.TxLog.Dropped(job.Route, job.TraceID)
|
||||
}
|
||||
s.d.Logger.Error("counting queue full, request dropped", "route", route, "trace_id", job.TraceID, "remote", r.RemoteAddr)
|
||||
writeError(w, http.StatusServiceUnavailable, "queue full")
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusAccepted, api.CountResponse{Accepted: true})
|
||||
}
|
||||
|
||||
// process is the worker body: plan, execute, dispatch notifications for
|
||||
// crossed limits and write the transaction log. ctx is the queue's own
|
||||
// bounded context, never a request's.
|
||||
func (s *Server) process(ctx context.Context, job counting.Job) {
|
||||
plan, err := s.d.Counter.Planner.Plan(job)
|
||||
if err != nil {
|
||||
s.d.Logger.Error("domain lookup failed", "route", job.Route, "trace_id", job.TraceID, "error", err)
|
||||
if s.d.TxLog != nil {
|
||||
s.d.TxLog.Error(job.Route, job.TraceID, err)
|
||||
}
|
||||
return
|
||||
}
|
||||
for _, lk := range plan.Lookups {
|
||||
if errors.Is(lk.Err, domains.ErrMalformed) {
|
||||
s.d.Logger.Warn("malformed row in domains db, domain treated as not hosted", "domain", lk.DomainName, "route", job.Route, "trace_id", job.TraceID, "error", lk.Err)
|
||||
}
|
||||
}
|
||||
out, err := s.d.Counter.Execute(ctx, plan)
|
||||
if err != nil {
|
||||
if s.d.Metrics != nil {
|
||||
s.d.Metrics.RecordRedisError("incr")
|
||||
}
|
||||
if s.d.TxLog != nil {
|
||||
s.d.TxLog.Failed(plan, err)
|
||||
}
|
||||
s.d.Logger.Error("counting failed", "route", plan.Route, "trace_id", plan.TraceID, "error", err)
|
||||
return
|
||||
}
|
||||
if out.Unknown != nil {
|
||||
if s.d.Metrics != nil {
|
||||
s.d.Metrics.RecordRedisError("incr_unknown")
|
||||
}
|
||||
s.d.Logger.Warn("redis outcome unknown, counts re-read from master", "route", plan.Route, "trace_id", plan.TraceID, "error", out.Unknown)
|
||||
}
|
||||
|
||||
dispatched := make([]bool, len(out.Results))
|
||||
for i, res := range out.Results {
|
||||
if res.Err != nil {
|
||||
if s.d.Metrics != nil {
|
||||
s.d.Metrics.RecordRedisError("incr")
|
||||
}
|
||||
s.d.Logger.Error("redis error on key", "route", plan.Route, "trace_id", plan.TraceID, "key", res.Key, "error", res.Err)
|
||||
continue
|
||||
}
|
||||
if s.d.Metrics != nil {
|
||||
s.d.Metrics.RecordIncrement(res.Level, res.Flow)
|
||||
}
|
||||
if res.Reached {
|
||||
if s.d.Metrics != nil {
|
||||
s.d.Metrics.RecordLimitReached(res.Level, res.Flow)
|
||||
}
|
||||
ev := notify.Event{TraceID: plan.TraceID, Route: plan.Route, Level: res.Level, ID: res.ID,
|
||||
Flow: res.Flow, Address: res.By, Count: res.Count, Limit: res.Limit}
|
||||
dispatched[i] = s.d.Notifier != nil && s.d.Notifier.Notify(ev)
|
||||
if !dispatched[i] && s.d.Metrics != nil {
|
||||
s.d.Metrics.RecordNotification(res.Level, "disabled")
|
||||
}
|
||||
}
|
||||
}
|
||||
if s.d.TxLog != nil {
|
||||
s.d.TxLog.Outcome(out, dispatched)
|
||||
}
|
||||
}
|
||||
|
||||
// handleCounters answers GET /counters?address=|domain=[&date=] from the
|
||||
// read side.
|
||||
func (s *Server) handleCounters(w http.ResponseWriter, r *http.Request) {
|
||||
q := r.URL.Query()
|
||||
address, domain := strings.TrimSpace(q.Get("address")), strings.ToLower(strings.TrimSpace(q.Get("domain")))
|
||||
if (address == "") == (domain == "") {
|
||||
s.reject(w, r, "/counters", &httpError{http.StatusBadRequest, "exactly one of address or domain is required"})
|
||||
return
|
||||
}
|
||||
date, err := api.ParseDate(q.Get("date"), s.d.Now())
|
||||
if err != nil {
|
||||
s.reject(w, r, "/counters", &httpError{http.StatusBadRequest, "date: " + err.Error()})
|
||||
return
|
||||
}
|
||||
|
||||
resp := api.CountersResponse{Date: date, Levels: []api.CountersLevel{}}
|
||||
var local string
|
||||
if address != "" {
|
||||
addr, err := api.NormalizeEmail(address)
|
||||
if err != nil {
|
||||
s.reject(w, r, "/counters", &httpError{http.StatusBadRequest, "address: invalid e-mail address"})
|
||||
return
|
||||
}
|
||||
addr = domains.Mailbox(addr)
|
||||
resp.Query, resp.Mode, resp.Exempt = addr, "address", s.d.Exempt[addr]
|
||||
at := strings.LastIndex(addr, "@")
|
||||
local, domain = addr[:at], addr[at+1:]
|
||||
} else {
|
||||
resp.Query, resp.Mode = domain, "domain"
|
||||
}
|
||||
|
||||
db := s.d.Domains()
|
||||
if db == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "domains db not loaded")
|
||||
return
|
||||
}
|
||||
d, err := db.Lookup(domain)
|
||||
if errors.Is(err, domains.ErrMalformed) {
|
||||
s.d.Logger.Warn("malformed row in domains db, domain treated as not hosted", "domain", domain, "route", "/counters", "error", err)
|
||||
} else if err != nil && !errors.Is(err, domains.ErrNotFound) {
|
||||
writeError(w, http.StatusServiceUnavailable, "domain lookup failed: "+err.Error())
|
||||
return
|
||||
}
|
||||
managed := err == nil
|
||||
if managed {
|
||||
org := d.OrgID
|
||||
resp.Managed, resp.Domain, resp.OrgID = true, d.Name, &org
|
||||
}
|
||||
|
||||
type row struct{ level, id string }
|
||||
var rows []row
|
||||
if resp.Mode == "address" {
|
||||
rows = append(rows, row{domains.AddressLevel, resp.Query})
|
||||
}
|
||||
rows = append(rows, row{domains.DomainLevel, domain})
|
||||
if managed {
|
||||
rows = append(rows, row{domains.OrganizationLevel, strconv.FormatInt(d.OrgID, 10)})
|
||||
}
|
||||
keys := make([]string, 0, 2*len(rows))
|
||||
for _, rw := range rows {
|
||||
keys = append(keys, store.Key(rw.level, rw.id, date, domains.FlowIn), store.Key(rw.level, rw.id, date, domains.FlowOut))
|
||||
}
|
||||
values, err := s.d.Store.Get(r.Context(), keys)
|
||||
if err != nil {
|
||||
if s.d.Metrics != nil {
|
||||
s.d.Metrics.RecordRedisError("mget")
|
||||
}
|
||||
writeError(w, http.StatusServiceUnavailable, "redis unavailable: "+err.Error())
|
||||
return
|
||||
}
|
||||
limit := func(level, flow string) *int64 {
|
||||
if !managed {
|
||||
return nil
|
||||
}
|
||||
if n, ok := d.Limits.Limit(level, flow, local); ok {
|
||||
return &n
|
||||
}
|
||||
return nil
|
||||
}
|
||||
for i, rw := range rows {
|
||||
resp.Levels = append(resp.Levels, api.CountersLevel{
|
||||
Level: rw.level, ID: rw.id,
|
||||
In: values[2*i], InLimit: limit(rw.level, domains.FlowIn),
|
||||
Out: values[2*i+1], OutLimit: limit(rw.level, domains.FlowOut),
|
||||
})
|
||||
}
|
||||
writeJSON(w, http.StatusOK, resp)
|
||||
}
|
||||
|
||||
func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) {
|
||||
db := s.d.Domains()
|
||||
if db == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "domains db not loaded")
|
||||
return
|
||||
}
|
||||
if err := db.Ping(); err != nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "domains db not readable: "+err.Error())
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 200*time.Millisecond)
|
||||
defer cancel()
|
||||
if err := s.d.Store.Ping(ctx); err != nil {
|
||||
if s.d.Metrics != nil {
|
||||
s.d.Metrics.RecordRedisError("ping")
|
||||
}
|
||||
writeError(w, http.StatusServiceUnavailable, fmt.Sprintf("redis: %v", err))
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]string{"status": "ok"})
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,238 @@
|
||||
// Package store holds the Redis counters. Writes (INCR) go to the master,
|
||||
// reads (MGET for /counters) go to the replicas; in Sentinel mode go-redis
|
||||
// discovers both, in static mode the addresses come from the config.
|
||||
//
|
||||
// Key format, shared with the rspamd Lua plugin and read by migadu_web
|
||||
// (apps/core/lib/core/redis_sentinel.ex):
|
||||
//
|
||||
// cnt/<A|D|O>/<id>/d/<YYYY-MM-DD>/<i|o>
|
||||
//
|
||||
// id is the address, the domain or the organisation id; the day is the UTC
|
||||
// calendar day; every key lives 31 days so the web app can chart a month.
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/redis/go-redis/v9"
|
||||
|
||||
"migadu_counters/internal/config"
|
||||
)
|
||||
|
||||
const (
|
||||
// Prefix of every counter key.
|
||||
Prefix = "cnt"
|
||||
// TTL is refreshed on every increment, like the Lua plugin did.
|
||||
TTL = 31 * 24 * time.Hour
|
||||
)
|
||||
|
||||
// Key builds a counter key: Key("A", "x@example.com", "2026-10-13", "i").
|
||||
func Key(level, id, date, flow string) string {
|
||||
return Prefix + "/" + level + "/" + id + "/d/" + date + "/" + flow
|
||||
}
|
||||
|
||||
// DayKey returns the UTC calendar day of t, as used in the keys. The web app
|
||||
// builds its keys with Date.utc_today(), so this must never use local time.
|
||||
func DayKey(t time.Time) string { return t.UTC().Format("2006-01-02") }
|
||||
|
||||
// Store is a write client plus one or more read clients.
|
||||
type Store struct {
|
||||
write *redis.Client
|
||||
readers []*redis.Client
|
||||
next atomic.Uint64
|
||||
}
|
||||
|
||||
// New builds the clients from the config. Nothing is dialled until the first
|
||||
// command, so an unreachable Redis does not prevent the daemon from starting.
|
||||
func New(cfg config.RedisConfig) (*Store, error) {
|
||||
if cfg.DB != 0 {
|
||||
return nil, fmt.Errorf("redis.db must be 0 (got %d): migadu_web reads the message counters from DB 0", cfg.DB)
|
||||
}
|
||||
timeout := cfg.Timeout()
|
||||
|
||||
if cfg.SentinelMode() {
|
||||
base := redis.FailoverOptions{
|
||||
MasterName: cfg.MasterName,
|
||||
SentinelAddrs: cfg.Sentinels,
|
||||
SentinelPassword: cfg.SentinelPassword,
|
||||
Password: cfg.Password,
|
||||
DB: 0,
|
||||
PoolSize: cfg.PoolSize,
|
||||
DialTimeout: timeout,
|
||||
ReadTimeout: timeout,
|
||||
WriteTimeout: timeout,
|
||||
}
|
||||
write := redis.NewFailoverClient(&base)
|
||||
// ReplicaOnly picks a random replica and falls back to the master when
|
||||
// no replica is known.
|
||||
replica := base
|
||||
replica.ReplicaOnly = true
|
||||
read := redis.NewFailoverClient(&replica)
|
||||
return &Store{write: write, readers: []*redis.Client{read}}, nil
|
||||
}
|
||||
|
||||
if cfg.WriteAddr == "" {
|
||||
return nil, errors.New("redis.write_addr is required when no sentinels are configured")
|
||||
}
|
||||
opts := func(addr string) *redis.Options {
|
||||
return &redis.Options{
|
||||
Addr: addr,
|
||||
Password: cfg.Password,
|
||||
DB: 0,
|
||||
PoolSize: cfg.PoolSize,
|
||||
DialTimeout: timeout,
|
||||
ReadTimeout: timeout,
|
||||
WriteTimeout: timeout,
|
||||
}
|
||||
}
|
||||
addrs := cfg.ReadAddrs
|
||||
if len(addrs) == 0 {
|
||||
addrs = []string{cfg.WriteAddr}
|
||||
}
|
||||
readers := make([]*redis.Client, 0, len(addrs))
|
||||
for _, a := range addrs {
|
||||
readers = append(readers, redis.NewClient(opts(a)))
|
||||
}
|
||||
return &Store{write: redis.NewClient(opts(cfg.WriteAddr)), readers: readers}, nil
|
||||
}
|
||||
|
||||
// NewWithClients wires pre-built clients (tests). With no readers the write
|
||||
// client is read from.
|
||||
func NewWithClients(write *redis.Client, readers []*redis.Client) *Store {
|
||||
if len(readers) == 0 {
|
||||
readers = []*redis.Client{write}
|
||||
}
|
||||
return &Store{write: write, readers: readers}
|
||||
}
|
||||
|
||||
// reader returns the next read client, round-robin.
|
||||
func (s *Store) reader() *redis.Client {
|
||||
n := s.next.Add(1) - 1
|
||||
return s.readers[n%uint64(len(s.readers))]
|
||||
}
|
||||
|
||||
// IncrResult is the outcome of one key's INCR inside the transaction.
|
||||
type IncrResult struct {
|
||||
Count int64 // value after the increment; meaningless when Err != nil
|
||||
Err error // this key's own failure (e.g. it holds a non-number)
|
||||
}
|
||||
|
||||
// Incr increments every key by one and refreshes its TTL in one MULTI/EXEC,
|
||||
// returning one result per key in order. EXEC's replies are the
|
||||
// authoritative counts, so no read-after-write is needed and replica lag
|
||||
// does not matter. A per-key failure (a key holding a non-number) is reported
|
||||
// in that key's result; the other keys are still incremented. The returned
|
||||
// error is set only when the transaction as a whole failed, in which case
|
||||
// OutcomeUnknown tells whether it may nevertheless have been applied.
|
||||
func (s *Store) Incr(ctx context.Context, keys []string) ([]IncrResult, error) {
|
||||
if len(keys) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
cmds := make([]*redis.IntCmd, len(keys))
|
||||
_, err := s.write.TxPipelined(ctx, func(pipe redis.Pipeliner) error {
|
||||
for i, k := range keys {
|
||||
cmds[i] = pipe.Incr(ctx, k)
|
||||
pipe.Expire(ctx, k, TTL)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
out := make([]IncrResult, len(keys))
|
||||
if err != nil {
|
||||
// TxPipelined returns the first failed command's error. A Redis reply
|
||||
// error (redis.Error) means EXEC answered and every command has a
|
||||
// definite result: value or its own error. Anything else (dial, write,
|
||||
// read timeout) is a transport failure of the whole transaction.
|
||||
var rerr redis.Error
|
||||
if !errors.As(err, &rerr) {
|
||||
return nil, fmt.Errorf("redis incr: %w", err)
|
||||
}
|
||||
}
|
||||
for i, c := range cmds {
|
||||
if cerr := c.Err(); cerr != nil {
|
||||
out[i].Err = fmt.Errorf("redis incr %s: %w", keys[i], cerr)
|
||||
continue
|
||||
}
|
||||
out[i].Count = c.Val()
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// OutcomeUnknown reports whether a failed Incr may still have been applied
|
||||
// by Redis: the request may have been written but the reply lost (timeout,
|
||||
// EOF, reset). A refused connection, a dial failure or a cancelled context
|
||||
// never reached Redis and is a definite failure.
|
||||
func OutcomeUnknown(err error) bool {
|
||||
if err == nil || errors.Is(err, context.Canceled) {
|
||||
return false
|
||||
}
|
||||
var opErr *net.OpError
|
||||
if errors.As(err, &opErr) && opErr.Op == "dial" {
|
||||
return false
|
||||
}
|
||||
if errors.Is(err, context.DeadlineExceeded) || errors.Is(err, io.EOF) || errors.Is(err, io.ErrUnexpectedEOF) {
|
||||
return true
|
||||
}
|
||||
var nerr net.Error
|
||||
if errors.As(err, &nerr) && nerr.Timeout() {
|
||||
return true
|
||||
}
|
||||
msg := err.Error()
|
||||
return strings.Contains(msg, "timeout") || strings.Contains(msg, "reset") || strings.Contains(msg, "broken pipe") || strings.Contains(msg, "EOF")
|
||||
}
|
||||
|
||||
// GetMaster reads the keys from the write side (the master) with one MGET;
|
||||
// used to establish the outcome of an Incr whose reply was lost, when a
|
||||
// replica might not have the increment yet. A missing key reads as 0.
|
||||
func (s *Store) GetMaster(ctx context.Context, keys []string) ([]int64, error) {
|
||||
return mget(ctx, s.write, keys)
|
||||
}
|
||||
|
||||
// Get reads the keys from a replica with one MGET; a missing key reads as 0.
|
||||
func (s *Store) Get(ctx context.Context, keys []string) ([]int64, error) {
|
||||
return mget(ctx, s.reader(), keys)
|
||||
}
|
||||
|
||||
func mget(ctx context.Context, c *redis.Client, keys []string) ([]int64, error) {
|
||||
if len(keys) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
vals, err := c.MGet(ctx, keys...).Result()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("redis mget: %w", err)
|
||||
}
|
||||
out := make([]int64, len(keys))
|
||||
for i, v := range vals {
|
||||
str, ok := v.(string)
|
||||
if !ok {
|
||||
continue // nil = missing key
|
||||
}
|
||||
n, err := strconv.ParseInt(str, 10, 64)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("redis key %s holds non-integer value %q", keys[i], str)
|
||||
}
|
||||
out[i] = n
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Ping checks the write side (the master), which is what /health cares about.
|
||||
func (s *Store) Ping(ctx context.Context) error { return s.write.Ping(ctx).Err() }
|
||||
|
||||
// Close closes every client.
|
||||
func (s *Store) Close() error {
|
||||
errs := []error{s.write.Close()}
|
||||
for _, r := range s.readers {
|
||||
if r != s.write {
|
||||
errs = append(errs, r.Close())
|
||||
}
|
||||
}
|
||||
return errors.Join(errs...)
|
||||
}
|
||||
@@ -0,0 +1,314 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/alicebob/miniredis/v2"
|
||||
"github.com/redis/go-redis/v9"
|
||||
|
||||
"migadu_counters/internal/config"
|
||||
)
|
||||
|
||||
func newMini(t *testing.T) *miniredis.Miniredis {
|
||||
t.Helper()
|
||||
m := miniredis.RunT(t)
|
||||
return m
|
||||
}
|
||||
|
||||
// testStore wires a Store to two separate miniredis instances so tests can
|
||||
// prove which side (write master vs read replica) a method talks to.
|
||||
func testStore(t *testing.T) (*Store, *miniredis.Miniredis, *miniredis.Miniredis) {
|
||||
t.Helper()
|
||||
w, r := newMini(t), newMini(t)
|
||||
s := NewWithClients(
|
||||
redis.NewClient(&redis.Options{Addr: w.Addr()}),
|
||||
[]*redis.Client{redis.NewClient(&redis.Options{Addr: r.Addr()})},
|
||||
)
|
||||
t.Cleanup(func() { s.Close() })
|
||||
return s, w, r
|
||||
}
|
||||
|
||||
func TestKey(t *testing.T) {
|
||||
tests := []struct {
|
||||
level, id, date, flow, want string
|
||||
}{
|
||||
{"A", "x@example.com", "2026-10-13", "i", "cnt/A/x@example.com/d/2026-10-13/i"},
|
||||
{"D", "example.com", "2026-10-13", "o", "cnt/D/example.com/d/2026-10-13/o"},
|
||||
{"O", "1234", "2026-01-01", "i", "cnt/O/1234/d/2026-01-01/i"},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
if got := Key(tt.level, tt.id, tt.date, tt.flow); got != tt.want {
|
||||
t.Errorf("Key(%s,%s,%s,%s) = %q, want %q", tt.level, tt.id, tt.date, tt.flow, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The day stamp is always the UTC calendar day, matching migadu_web's
|
||||
// Date.utc_today() — never the server's local day.
|
||||
func TestDayKey_UTC(t *testing.T) {
|
||||
cest := time.FixedZone("CEST", 2*3600)
|
||||
tests := []struct {
|
||||
in time.Time
|
||||
want string
|
||||
}{
|
||||
{time.Date(2026, 10, 13, 23, 59, 0, 0, cest), "2026-10-13"}, // 21:59 UTC
|
||||
{time.Date(2026, 10, 14, 1, 30, 0, 0, cest), "2026-10-13"}, // 23:30 UTC the day before
|
||||
{time.Date(2026, 10, 14, 2, 0, 0, 0, cest), "2026-10-14"}, // 00:00 UTC
|
||||
{time.Date(2026, 3, 5, 12, 0, 0, 0, time.UTC), "2026-03-05"}, // zero padded
|
||||
}
|
||||
for _, tt := range tests {
|
||||
if got := DayKey(tt.in); got != tt.want {
|
||||
t.Errorf("DayKey(%v) = %q, want %q", tt.in, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestIncr(t *testing.T) {
|
||||
s, w, _ := testStore(t)
|
||||
ctx := context.Background()
|
||||
keys := []string{"cnt/A/x@example.com/d/2026-10-13/o", "cnt/O/1234/d/2026-10-13/o"}
|
||||
|
||||
got, err := s.Incr(ctx, keys)
|
||||
if err != nil {
|
||||
t.Fatalf("Incr: %v", err)
|
||||
}
|
||||
if len(got) != 2 || got[0].Count != 1 || got[1].Count != 1 || got[0].Err != nil || got[1].Err != nil {
|
||||
t.Fatalf("first Incr = %+v, want counts [1 1]", got)
|
||||
}
|
||||
got, err = s.Incr(ctx, keys)
|
||||
if err != nil || got[0].Count != 2 || got[1].Count != 2 {
|
||||
t.Fatalf("second Incr = %+v, %v; want [2 2]", got, err)
|
||||
}
|
||||
for _, k := range keys {
|
||||
if v, _ := w.Get(k); v != "2" {
|
||||
t.Errorf("stored %s = %q, want \"2\"", k, v)
|
||||
}
|
||||
if ttl := w.TTL(k); ttl <= 30*24*time.Hour || ttl > TTL {
|
||||
t.Errorf("TTL(%s) = %v, want (30d, 31d]", k, ttl)
|
||||
}
|
||||
}
|
||||
if got, err := s.Incr(ctx, nil); err != nil || len(got) != 0 {
|
||||
t.Errorf("Incr(no keys) = %v, %v; want empty, nil", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
// One key holding a non-number fails its own INCR inside EXEC; the other
|
||||
// keys are still incremented and their counts reported (review #4).
|
||||
func TestIncr_PartialFailure(t *testing.T) {
|
||||
s, w, _ := testStore(t)
|
||||
w.Set("bad", "garbage")
|
||||
got, err := s.Incr(context.Background(), []string{"k1", "bad", "k3"})
|
||||
if err != nil {
|
||||
t.Fatalf("a per-key failure must not fail the whole Incr: %v", err)
|
||||
}
|
||||
if got[0].Count != 1 || got[0].Err != nil || got[2].Count != 1 || got[2].Err != nil {
|
||||
t.Errorf("good keys = %+v %+v, want count 1 without error", got[0], got[2])
|
||||
}
|
||||
if got[1].Err == nil || !strings.Contains(got[1].Err.Error(), "not an integer") {
|
||||
t.Errorf("bad key err = %v, want the INCR error", got[1].Err)
|
||||
}
|
||||
if v, _ := w.Get("bad"); v != "garbage" {
|
||||
t.Errorf("bad key = %q, must be untouched", v)
|
||||
}
|
||||
if ttl := w.TTL("k1"); ttl <= 30*24*time.Hour {
|
||||
t.Errorf("TTL(k1) = %v, EXPIRE should still have run", ttl)
|
||||
}
|
||||
}
|
||||
|
||||
// GetMaster reads from the write side (used after a lost reply, when the
|
||||
// replica may not have the increment yet).
|
||||
func TestGetMaster(t *testing.T) {
|
||||
s, w, r := testStore(t)
|
||||
w.Set("k", "19")
|
||||
r.Set("k", "3")
|
||||
got, err := s.GetMaster(context.Background(), []string{"k", "missing"})
|
||||
if err != nil || len(got) != 2 || got[0] != 19 || got[1] != 0 {
|
||||
t.Errorf("GetMaster = %v, %v; want [19 0] from the master", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
// OutcomeUnknown tells whether a failed Incr may still have been applied: a
|
||||
// refused connection never reached Redis, a timeout or reset after the write
|
||||
// might have.
|
||||
func TestOutcomeUnknown(t *testing.T) {
|
||||
s, w, _ := testStore(t)
|
||||
w.Close()
|
||||
_, err := s.Incr(context.Background(), []string{"k"})
|
||||
if err == nil {
|
||||
t.Fatal("expected an error with the write side down")
|
||||
}
|
||||
if OutcomeUnknown(err) {
|
||||
t.Errorf("connection refused must be a known failure, got unknown: %v", err)
|
||||
}
|
||||
if OutcomeUnknown(nil) || OutcomeUnknown(context.Canceled) {
|
||||
t.Error("nil / canceled are not unknown outcomes")
|
||||
}
|
||||
for _, e := range []error{context.DeadlineExceeded, io.EOF, &net.OpError{Op: "read", Err: timeoutErr{}}, errors.New("i/o timeout")} {
|
||||
if !OutcomeUnknown(e) {
|
||||
t.Errorf("%v should be an unknown outcome", e)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
type timeoutErr struct{}
|
||||
|
||||
func (timeoutErr) Error() string { return "i/o timeout" }
|
||||
func (timeoutErr) Timeout() bool { return true }
|
||||
func (timeoutErr) Temporary() bool { return true }
|
||||
|
||||
// Fifty concurrent increments of one key must end at exactly 50 — the
|
||||
// pipeline relies on INCRBY being atomic, not on read-then-write.
|
||||
func TestIncr_Concurrent(t *testing.T) {
|
||||
s, w, _ := testStore(t)
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < 50; i++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
if _, err := s.Incr(context.Background(), []string{"k"}); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
if v, _ := w.Get("k"); v != "50" {
|
||||
t.Errorf("k = %q, want 50", v)
|
||||
}
|
||||
}
|
||||
|
||||
// Get reads from the replica side: a key present only on the write instance
|
||||
// is invisible, a key present on the read instance is returned, missing = 0.
|
||||
func TestGet_UsesReadSide(t *testing.T) {
|
||||
s, w, r := testStore(t)
|
||||
w.Set("only-on-write", "9")
|
||||
r.Set("on-read", "5")
|
||||
got, err := s.Get(context.Background(), []string{"on-read", "only-on-write", "missing"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) != 3 || got[0] != 5 || got[1] != 0 || got[2] != 0 {
|
||||
t.Errorf("Get = %v, want [5 0 0]", got)
|
||||
}
|
||||
if got, err := s.Get(context.Background(), nil); err != nil || len(got) != 0 {
|
||||
t.Errorf("Get(no keys) = %v, %v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGet_RoundRobinReaders(t *testing.T) {
|
||||
w, r1, r2 := newMini(t), newMini(t), newMini(t)
|
||||
r1.Set("k", "1")
|
||||
r2.Set("k", "2")
|
||||
s := NewWithClients(redis.NewClient(&redis.Options{Addr: w.Addr()}), []*redis.Client{
|
||||
redis.NewClient(&redis.Options{Addr: r1.Addr()}),
|
||||
redis.NewClient(&redis.Options{Addr: r2.Addr()}),
|
||||
})
|
||||
defer s.Close()
|
||||
seen := map[int64]int{}
|
||||
for i := 0; i < 4; i++ {
|
||||
got, err := s.Get(context.Background(), []string{"k"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
seen[got[0]]++
|
||||
}
|
||||
if seen[1] != 2 || seen[2] != 2 {
|
||||
t.Errorf("reads not alternated between replicas: %v", seen)
|
||||
}
|
||||
}
|
||||
|
||||
func TestErrors_RedisDown(t *testing.T) {
|
||||
s, w, r := testStore(t)
|
||||
ctx := context.Background()
|
||||
if err := s.Ping(ctx); err != nil {
|
||||
t.Fatalf("Ping before close: %v", err)
|
||||
}
|
||||
w.Close()
|
||||
if _, err := s.Incr(ctx, []string{"k"}); err == nil {
|
||||
t.Error("Incr with write side down should fail")
|
||||
}
|
||||
if err := s.Ping(ctx); err == nil {
|
||||
t.Error("Ping with write side down should fail")
|
||||
}
|
||||
r.Close()
|
||||
if _, err := s.Get(ctx, []string{"k"}); err == nil {
|
||||
t.Error("Get with read side down should fail")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNew_Static(t *testing.T) {
|
||||
w, r := newMini(t), newMini(t)
|
||||
s, err := New(config.RedisConfig{WriteAddr: w.Addr(), ReadAddrs: []string{r.Addr()}, PoolSize: 2, TimeoutMS: 200})
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
defer s.Close()
|
||||
ctx := context.Background()
|
||||
if err := s.Ping(ctx); err != nil {
|
||||
t.Fatalf("Ping: %v", err)
|
||||
}
|
||||
if _, err := s.Incr(ctx, []string{"k"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if v, _ := w.Get("k"); v != "1" {
|
||||
t.Errorf("write went to %q, want the write_addr instance", v)
|
||||
}
|
||||
r.Set("k", "7")
|
||||
if got, _ := s.Get(ctx, []string{"k"}); got[0] != 7 {
|
||||
t.Errorf("Get = %v, want 7 from read_addrs", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNew_StaticDefaultsReadToWrite(t *testing.T) {
|
||||
w := newMini(t)
|
||||
s, err := New(config.RedisConfig{WriteAddr: w.Addr(), PoolSize: 1, TimeoutMS: 200})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer s.Close()
|
||||
w.Set("k", "3")
|
||||
if got, err := s.Get(context.Background(), []string{"k"}); err != nil || got[0] != 3 {
|
||||
t.Errorf("Get = %v, %v; want 3 from the write instance", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNew_RejectsNonZeroDB(t *testing.T) {
|
||||
if _, err := New(config.RedisConfig{WriteAddr: "127.0.0.1:6379", DB: 1, PoolSize: 1, TimeoutMS: 100}); err == nil {
|
||||
t.Error("New with db=1 should fail: migadu_web reads DB 0")
|
||||
}
|
||||
}
|
||||
|
||||
// Sentinel mode constructs lazily; an unreachable sentinel surfaces on the
|
||||
// first command, bounded by the configured timeout.
|
||||
func TestNew_SentinelUnreachable(t *testing.T) {
|
||||
s, err := New(config.RedisConfig{MasterName: "mymaster", Sentinels: []string{"127.0.0.1:1"}, PoolSize: 1, TimeoutMS: 200})
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
defer s.Close()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
|
||||
defer cancel()
|
||||
start := time.Now()
|
||||
if err := s.Ping(ctx); err == nil {
|
||||
t.Error("Ping against an unreachable sentinel should fail")
|
||||
}
|
||||
if time.Since(start) > 2500*time.Millisecond {
|
||||
t.Errorf("Ping took %v, should fail fast", time.Since(start))
|
||||
}
|
||||
}
|
||||
|
||||
func TestClose(t *testing.T) {
|
||||
s, _, _ := testStore(t)
|
||||
if err := s.Close(); err != nil {
|
||||
t.Fatalf("Close: %v", err)
|
||||
}
|
||||
if err := s.Ping(context.Background()); err == nil {
|
||||
t.Error("Ping after Close should fail")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,300 @@
|
||||
// Package txlog writes the counting transaction log: one block of lines per
|
||||
// request, in the shape of the rspamd Lua plugin's log so existing grep
|
||||
// habits keep working, with rspamd's task id replaced by the trace id:
|
||||
//
|
||||
// 2026-10-06 01:12:06.412 <ad8525> out; local domain: example.com account_id=7
|
||||
// 2026-10-06 01:12:06.412 <ad8525> out; determined limit (level=[A] flow=[o] key=[cnt/A/x@example.com/d/2026-10-06/o]) => 100
|
||||
// 2026-10-06 01:12:06.412 <ad8525> out; tracked address counters for x@example.com => count=12 limit=100 reached=false
|
||||
//
|
||||
// Each block is written with a single Write so concurrent requests never
|
||||
// interleave. With format "json" the same events become one JSON object per
|
||||
// line.
|
||||
package txlog
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"migadu_counters/internal/counting"
|
||||
"migadu_counters/internal/domains"
|
||||
"migadu_counters/internal/notify"
|
||||
)
|
||||
|
||||
// Logger writes transaction blocks to w.
|
||||
type Logger struct {
|
||||
w io.Writer
|
||||
json bool
|
||||
now func() time.Time
|
||||
}
|
||||
|
||||
// New creates a Logger. format is "text" (default) or "json"; now may be nil.
|
||||
func New(w io.Writer, format string, now func() time.Time) *Logger {
|
||||
if now == nil {
|
||||
now = time.Now
|
||||
}
|
||||
return &Logger{w: w, json: format == "json", now: now}
|
||||
}
|
||||
|
||||
// block accumulates the lines of one request.
|
||||
type block struct {
|
||||
l *Logger
|
||||
ts time.Time
|
||||
trace string
|
||||
route string
|
||||
buf bytes.Buffer
|
||||
}
|
||||
|
||||
func (l *Logger) begin(trace, route string) *block {
|
||||
return &block{l: l, ts: l.now().UTC(), trace: trace, route: route}
|
||||
}
|
||||
|
||||
// line appends one event: msg in text mode, fields in json mode.
|
||||
func (b *block) line(event, msg string, fields map[string]any) {
|
||||
if b.l.json {
|
||||
rec := map[string]any{
|
||||
"ts": b.ts.Format("2006-01-02T15:04:05.000Z07:00"),
|
||||
"trace_id": b.trace,
|
||||
"route": b.route,
|
||||
"event": event,
|
||||
}
|
||||
for k, v := range fields {
|
||||
rec[k] = v
|
||||
}
|
||||
data, err := json.Marshal(rec)
|
||||
if err != nil {
|
||||
data = []byte(fmt.Sprintf(`{"event":"marshal_error","error":%q}`, err.Error()))
|
||||
}
|
||||
b.buf.Write(data)
|
||||
b.buf.WriteByte('\n')
|
||||
return
|
||||
}
|
||||
trace := b.trace
|
||||
if trace == "" {
|
||||
trace = "-"
|
||||
}
|
||||
fmt.Fprintf(&b.buf, "%s <%s> %s; %s\n", b.ts.Format("2006-01-02 15:04:05.000"), trace, b.route, msg)
|
||||
}
|
||||
|
||||
func (b *block) flush() { _, _ = b.l.w.Write(b.buf.Bytes()) }
|
||||
|
||||
// Outcome logs an executed plan. dispatched[i] tells, for a Results[i] that
|
||||
// reached its limit, whether a notification was handed to the notifier.
|
||||
func (l *Logger) Outcome(out *counting.Outcome, dispatched []bool) {
|
||||
b := l.begin(out.TraceID, out.Route)
|
||||
b.request(out.Plan)
|
||||
b.trail(out.Plan)
|
||||
if out.Unknown != nil {
|
||||
b.line("redis_unknown", fmt.Sprintf("redis outcome unknown: %v, re-read from master", out.Unknown),
|
||||
map[string]any{"error": out.Unknown.Error(), "reread": true})
|
||||
}
|
||||
for _, r := range out.Results {
|
||||
if r.Err != nil {
|
||||
b.line("redis_key_error", fmt.Sprintf("redis error on key %s: %v (not counted)", r.Key, r.Err),
|
||||
map[string]any{"level": r.Level, "id": r.ID, "flow": r.Flow, "key": r.Key, "error": r.Err.Error()})
|
||||
continue
|
||||
}
|
||||
b.tracked(r, out.Unknown != nil)
|
||||
}
|
||||
for i, r := range out.Results {
|
||||
if r.Reached {
|
||||
b.reached(r, i < len(dispatched) && dispatched[i])
|
||||
}
|
||||
}
|
||||
if out.Skipped != "" {
|
||||
b.line("skipped", "skipped: "+out.Skipped, map[string]any{"reason": out.Skipped})
|
||||
}
|
||||
b.flush()
|
||||
}
|
||||
|
||||
// Failed logs a plan whose increments could not be written to Redis.
|
||||
func (l *Logger) Failed(plan *counting.Plan, err error) {
|
||||
b := l.begin(plan.TraceID, plan.Route)
|
||||
b.request(plan)
|
||||
b.trail(plan)
|
||||
b.line("redis_error", fmt.Sprintf("redis error: %v (nothing counted)", err), map[string]any{"error": err.Error()})
|
||||
b.flush()
|
||||
}
|
||||
|
||||
// Dropped logs a request that was rejected because the counting queue was full.
|
||||
func (l *Logger) Dropped(route, traceID string) {
|
||||
b := l.begin(traceID, route)
|
||||
b.line("dropped", "dropped: queue full (not counted)", map[string]any{"reason": "queue full"})
|
||||
b.flush()
|
||||
}
|
||||
|
||||
// Error logs a request that failed before a plan existed (domain lookup error).
|
||||
func (l *Logger) Error(route, traceID string, err error) {
|
||||
b := l.begin(traceID, route)
|
||||
b.line("error", fmt.Sprintf("error: %v (nothing counted)", err), map[string]any{"error": err.Error()})
|
||||
b.flush()
|
||||
}
|
||||
|
||||
// NotificationSent logs a delivered web app notification (notify hook).
|
||||
func (l *Logger) NotificationSent(ev notify.Event, url string, attempt int) {
|
||||
b := l.begin(ev.TraceID, ev.Route)
|
||||
b.line("notification_sent", fmt.Sprintf("sent limit exceeded notification to url %q", url),
|
||||
map[string]any{"url": url, "attempt": attempt, "level": ev.Level, "id": ev.ID, "flow": ev.Flow})
|
||||
b.flush()
|
||||
}
|
||||
|
||||
// NotificationFailed logs a notification given up on (notify hook).
|
||||
func (l *Logger) NotificationFailed(ev notify.Event, url string, attempts int, err error) {
|
||||
b := l.begin(ev.TraceID, ev.Route)
|
||||
b.line("notification_failed", fmt.Sprintf("failed limit exceeded notification to url %q after %d attempts: %v", url, attempts, err),
|
||||
map[string]any{"url": url, "attempts": attempts, "error": err.Error(), "level": ev.Level, "id": ev.ID, "flow": ev.Flow})
|
||||
b.flush()
|
||||
}
|
||||
|
||||
func (b *block) request(p *counting.Plan) {
|
||||
if p.Route == counting.RouteOut {
|
||||
msg := fmt.Sprintf("request auth_user=%s recipient=%s mail_from=%s", p.AuthUser, p.Recipient, p.MailFrom)
|
||||
fields := map[string]any{"auth_user": p.AuthUser, "recipient": p.Recipient, "mail_from": p.MailFrom}
|
||||
if p.LoginSuffix != "" {
|
||||
msg += " login_suffix=" + p.LoginSuffix
|
||||
fields["login_suffix"] = p.LoginSuffix
|
||||
}
|
||||
b.line("request", msg, fields)
|
||||
} else {
|
||||
b.line("request", fmt.Sprintf("request mail_from=%s recipient=%s junk=%t", p.MailFrom, p.Recipient, p.Junk),
|
||||
map[string]any{"mail_from": p.MailFrom, "recipient": p.Recipient, "junk": p.Junk})
|
||||
}
|
||||
for _, u := range p.Unusable {
|
||||
shown := u.Value
|
||||
if shown == "" {
|
||||
shown = "<empty>"
|
||||
}
|
||||
b.line("not_usable", fmt.Sprintf("%s address not usable: %s (%s, not counted)", u.Role, shown, u.Reason),
|
||||
map[string]any{"role": u.Role, "value": u.Value, "reason": u.Reason})
|
||||
}
|
||||
}
|
||||
|
||||
// trail logs the domain lookups and the limits determined for each.
|
||||
func (b *block) trail(p *counting.Plan) {
|
||||
for _, lk := range p.Lookups {
|
||||
if lk.Domain == nil {
|
||||
if lk.Err == nil || errors.Is(lk.Err, domains.ErrNotFound) {
|
||||
b.line("not_in_db", fmt.Sprintf("%s domain not in domains db: %s (not counted)", lk.Role, lk.DomainName),
|
||||
map[string]any{"role": lk.Role, "domain": lk.DomainName})
|
||||
} else if errors.Is(lk.Err, domains.ErrMalformed) {
|
||||
detail := strings.TrimPrefix(lk.Err.Error(), domains.ErrMalformed.Error()+": ")
|
||||
b.line("malformed_row", fmt.Sprintf("%s domain row malformed in domains db: %s: %s (not counted)", lk.Role, lk.DomainName, detail),
|
||||
map[string]any{"role": lk.Role, "domain": lk.DomainName, "error": detail})
|
||||
} else {
|
||||
b.line("lookup_error", fmt.Sprintf("%s domain lookup failed: %s: %v", lk.Role, lk.DomainName, lk.Err),
|
||||
map[string]any{"role": lk.Role, "domain": lk.DomainName, "error": lk.Err.Error()})
|
||||
}
|
||||
continue
|
||||
}
|
||||
d := lk.Domain
|
||||
b.line("local_domain", fmt.Sprintf("local domain: %s account_id=%d", d.Name, d.OrgID),
|
||||
map[string]any{"role": lk.Role, "domain": d.Name, "org_id": d.OrgID})
|
||||
local := lk.Address
|
||||
if i := strings.LastIndex(local, "@"); i >= 0 {
|
||||
local = local[:i]
|
||||
}
|
||||
shown, other := summarize(d.Limits, local)
|
||||
msg := fmt.Sprintf("domain: %s limits_map=%s", d.Name, renderLimits(shown, d.Limits))
|
||||
if other > 0 {
|
||||
msg += fmt.Sprintf(" (+%d other overrides)", other)
|
||||
}
|
||||
limitsJSON := make(map[string]int64, len(shown))
|
||||
for _, k := range shown {
|
||||
limitsJSON[k] = d.Limits[k]
|
||||
}
|
||||
b.line("limits", msg, map[string]any{"domain": d.Name, "limits": limitsJSON, "other_overrides": other})
|
||||
|
||||
flow := domains.FlowIn
|
||||
if lk.Role == "sender" {
|
||||
flow = domains.FlowOut
|
||||
}
|
||||
for _, inc := range p.Increments {
|
||||
if inc.By != lk.Address || inc.Flow != flow {
|
||||
continue
|
||||
}
|
||||
b.line("determined_limit",
|
||||
fmt.Sprintf("determined limit (level=[%s] flow=[%s] key=[%s]) => %s", inc.Level, inc.Flow, inc.Key, limitText(inc)),
|
||||
map[string]any{"level": inc.Level, "flow": inc.Flow, "key": inc.Key, "limit": limitJSON(inc), "limited": inc.Limited})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (b *block) tracked(r counting.Result, reread bool) {
|
||||
msg := fmt.Sprintf("tracked %s counters for %s => count=%d limit=%s reached=%t", levelWord(r.Level), r.ID, r.Count, limitText(r.Increment), r.Reached)
|
||||
fields := map[string]any{"level": r.Level, "id": r.ID, "flow": r.Flow, "key": r.Key, "count": r.Count, "limit": limitJSON(r.Increment), "reached": r.Reached}
|
||||
if reread {
|
||||
msg += " (re-read)"
|
||||
fields["reread"] = true
|
||||
}
|
||||
b.line("tracked", msg, fields)
|
||||
}
|
||||
|
||||
func (b *block) reached(r counting.Result, dispatched bool) {
|
||||
name := notify.URLName(r.Level)
|
||||
verdict := "notifying " + name
|
||||
if !dispatched {
|
||||
verdict = "no " + name + " configured, not notifying"
|
||||
}
|
||||
b.line("limit_reached",
|
||||
fmt.Sprintf("limit reached (level=[%s] id=[%s] flow=[%s] by=[%s]) => %s", r.Level, r.ID, r.Flow, r.By, verdict),
|
||||
map[string]any{"level": r.Level, "id": r.ID, "flow": r.Flow, "by": r.By, "count": r.Count, "limit": r.Limit, "notifying": dispatched, "url_name": name})
|
||||
}
|
||||
|
||||
// summarize picks the limit keys worth printing for one address: the base
|
||||
// keys and the address's own overrides. Some domains carry thousands of
|
||||
// per-mailbox overrides (boxmailhere.com: 9 798); dumping them all would make
|
||||
// a 190 KB log line.
|
||||
func summarize(l domains.Limits, local string) (shown []string, other int) {
|
||||
keys := make([]string, 0, len(l))
|
||||
for k := range l {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, k := range keys {
|
||||
if !strings.Contains(k, "@/") || strings.HasPrefix(k, local+"@/") {
|
||||
shown = append(shown, k)
|
||||
} else {
|
||||
other++
|
||||
}
|
||||
}
|
||||
return shown, other
|
||||
}
|
||||
|
||||
func renderLimits(keys []string, l domains.Limits) string {
|
||||
parts := make([]string, len(keys))
|
||||
for i, k := range keys {
|
||||
parts[i] = fmt.Sprintf("%s=%d", k, l[k])
|
||||
}
|
||||
return "{" + strings.Join(parts, ", ") + "}"
|
||||
}
|
||||
|
||||
func levelWord(level string) string {
|
||||
switch level {
|
||||
case domains.AddressLevel:
|
||||
return "address"
|
||||
case domains.DomainLevel:
|
||||
return "domain"
|
||||
case domains.OrganizationLevel:
|
||||
return "account" // the Lua plugin's word for the organisation level
|
||||
}
|
||||
return level
|
||||
}
|
||||
|
||||
func limitText(inc counting.Increment) string {
|
||||
if !inc.Limited {
|
||||
return "unlimited"
|
||||
}
|
||||
return fmt.Sprintf("%d", inc.Limit)
|
||||
}
|
||||
|
||||
func limitJSON(inc counting.Increment) any {
|
||||
if !inc.Limited {
|
||||
return nil
|
||||
}
|
||||
return inc.Limit
|
||||
}
|
||||
@@ -0,0 +1,413 @@
|
||||
package txlog
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"migadu_counters/internal/counting"
|
||||
"migadu_counters/internal/domains"
|
||||
"migadu_counters/internal/notify"
|
||||
)
|
||||
|
||||
var clock = time.Date(2026, 10, 6, 1, 12, 6, 412_000_000, time.UTC)
|
||||
|
||||
const ts = "2026-10-06 01:12:06.412"
|
||||
|
||||
type table map[string]*domains.Domain
|
||||
|
||||
func (t table) Lookup(d string) (*domains.Domain, error) {
|
||||
if r, ok := t[d]; ok {
|
||||
return r, nil
|
||||
}
|
||||
return nil, domains.ErrNotFound
|
||||
}
|
||||
|
||||
func planner() *counting.Planner {
|
||||
return &counting.Planner{
|
||||
Lookup: table{
|
||||
"wearemiraclesnow.com": {Name: "wearemiraclesnow.com", OrgID: 116681,
|
||||
Limits: domains.ParseLimits("O/i=10000,O/o=2200,D/i=10000,D/o=2200,A/i=500,A/o=100")},
|
||||
"nolimits.net": {Name: "nolimits.net", OrgID: 11, Limits: domains.Limits{}},
|
||||
"many.example": {Name: "many.example", OrgID: 5,
|
||||
Limits: domains.ParseLimits("A/i=1,A/o=2,bob@/i=3,bob@/o=4,carol@/i=5,dave@/o=6")},
|
||||
},
|
||||
Exempt: map[string]bool{"support@migadu.com": true},
|
||||
Now: func() time.Time { return clock },
|
||||
}
|
||||
}
|
||||
|
||||
// outcome executes nothing: it attaches the given counts to the plan, with
|
||||
// Reached derived exactly as Counter does.
|
||||
func outcome(plan *counting.Plan, counts ...int64) *counting.Outcome {
|
||||
out := &counting.Outcome{Plan: plan}
|
||||
for i, inc := range plan.Increments {
|
||||
out.Results = append(out.Results, counting.Result{
|
||||
Increment: inc, Count: counts[i],
|
||||
Reached: inc.Limited && counting.Crossed(counts[i], 1, inc.Limit),
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func newLogger(format string) (*Logger, *bytes.Buffer) {
|
||||
var buf bytes.Buffer
|
||||
return New(&buf, format, func() time.Time { return clock }), &buf
|
||||
}
|
||||
|
||||
func TestOutcome_OutText(t *testing.T) {
|
||||
plan, err := planner().PlanOut("ad8525", "noah@wearemiraclesnow.com", "noah@wearemiraclesnow.com", "y@example.org")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
l, buf := newLogger("text")
|
||||
l.Outcome(outcome(plan, 12, 1830, 2200), []bool{false, false, true})
|
||||
|
||||
want := strings.Join([]string{
|
||||
ts + " <ad8525> out; request auth_user=noah@wearemiraclesnow.com recipient=y@example.org mail_from=noah@wearemiraclesnow.com",
|
||||
ts + " <ad8525> out; local domain: wearemiraclesnow.com account_id=116681",
|
||||
ts + " <ad8525> out; domain: wearemiraclesnow.com limits_map={A/i=500, A/o=100, D/i=10000, D/o=2200, O/i=10000, O/o=2200}",
|
||||
ts + " <ad8525> out; determined limit (level=[A] flow=[o] key=[cnt/A/noah@wearemiraclesnow.com/d/2026-10-06/o]) => 100",
|
||||
ts + " <ad8525> out; determined limit (level=[D] flow=[o] key=[cnt/D/wearemiraclesnow.com/d/2026-10-06/o]) => 2200",
|
||||
ts + " <ad8525> out; determined limit (level=[O] flow=[o] key=[cnt/O/116681/d/2026-10-06/o]) => 2200",
|
||||
ts + " <ad8525> out; recipient domain not in domains db: example.org (not counted)",
|
||||
ts + " <ad8525> out; tracked address counters for noah@wearemiraclesnow.com => count=12 limit=100 reached=false",
|
||||
ts + " <ad8525> out; tracked domain counters for wearemiraclesnow.com => count=1830 limit=2200 reached=false",
|
||||
ts + " <ad8525> out; tracked account counters for 116681 => count=2200 limit=2200 reached=true",
|
||||
ts + " <ad8525> out; limit reached (level=[O] id=[116681] flow=[o] by=[noah@wearemiraclesnow.com]) => notifying org_url",
|
||||
"",
|
||||
}, "\n")
|
||||
if got := buf.String(); got != want {
|
||||
t.Errorf("block mismatch\n--- got ---\n%s--- want ---\n%s", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOutcome_InTextUnlimited(t *testing.T) {
|
||||
plan, err := planner().PlanIn("tr2", "s@gmail.com", "q@nolimits.net", false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
l, buf := newLogger("text")
|
||||
l.Outcome(outcome(plan, 1, 1, 1), nil)
|
||||
want := strings.Join([]string{
|
||||
ts + " <tr2> in; request mail_from=s@gmail.com recipient=q@nolimits.net junk=false",
|
||||
ts + " <tr2> in; local domain: nolimits.net account_id=11",
|
||||
ts + " <tr2> in; domain: nolimits.net limits_map={}",
|
||||
ts + " <tr2> in; determined limit (level=[A] flow=[i] key=[cnt/A/q@nolimits.net/d/2026-10-06/i]) => unlimited",
|
||||
ts + " <tr2> in; determined limit (level=[D] flow=[i] key=[cnt/D/nolimits.net/d/2026-10-06/i]) => unlimited",
|
||||
ts + " <tr2> in; determined limit (level=[O] flow=[i] key=[cnt/O/11/d/2026-10-06/i]) => unlimited",
|
||||
ts + " <tr2> in; tracked address counters for q@nolimits.net => count=1 limit=unlimited reached=false",
|
||||
ts + " <tr2> in; tracked domain counters for nolimits.net => count=1 limit=unlimited reached=false",
|
||||
ts + " <tr2> in; tracked account counters for 11 => count=1 limit=unlimited reached=false",
|
||||
"",
|
||||
}, "\n")
|
||||
if got := buf.String(); got != want {
|
||||
t.Errorf("block mismatch\n--- got ---\n%s--- want ---\n%s", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOutcome_ReachedWithoutURL(t *testing.T) {
|
||||
plan, _ := planner().PlanIn("tr", "s@gmail.com", "y@wearemiraclesnow.com", false)
|
||||
l, buf := newLogger("text")
|
||||
// D/i=10000 reached, A and O not; domain_url not configured -> dispatched false.
|
||||
l.Outcome(outcome(plan, 3, 10000, 9), []bool{false, false, false})
|
||||
want := ts + " <tr> in; limit reached (level=[D] id=[wearemiraclesnow.com] flow=[i] by=[y@wearemiraclesnow.com]) => no domain_url configured, not notifying\n"
|
||||
if got := buf.String(); !strings.HasSuffix(got, want) {
|
||||
t.Errorf("missing not-notifying line\n--- got ---\n%s--- want suffix ---\n%s", got, want)
|
||||
}
|
||||
if strings.Count(buf.String(), "limit reached") != 1 {
|
||||
t.Errorf("want exactly one limit reached line:\n%s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestOutcome_Skips(t *testing.T) {
|
||||
p := planner()
|
||||
cases := []struct {
|
||||
name string
|
||||
plan func() (*counting.Plan, error)
|
||||
want []string
|
||||
}{
|
||||
{"exempt", func() (*counting.Plan, error) {
|
||||
return p.PlanIn("t", "support@migadu.com", "y@wearemiraclesnow.com", false)
|
||||
},
|
||||
[]string{
|
||||
ts + " <t> in; request mail_from=support@migadu.com recipient=y@wearemiraclesnow.com junk=false",
|
||||
ts + " <t> in; skipped: exempt",
|
||||
}},
|
||||
{"junk", func() (*counting.Plan, error) { return p.PlanIn("t", "s@gmail.com", "y@wearemiraclesnow.com", true) },
|
||||
[]string{
|
||||
ts + " <t> in; request mail_from=s@gmail.com recipient=y@wearemiraclesnow.com junk=true",
|
||||
ts + " <t> in; skipped: junk",
|
||||
}},
|
||||
{"unknown", func() (*counting.Plan, error) { return p.PlanOut("t", "a@unknown.tld", "", "b@gmail.com") },
|
||||
[]string{
|
||||
ts + " <t> out; request auth_user=a@unknown.tld recipient=b@gmail.com mail_from=",
|
||||
ts + " <t> out; sender domain not in domains db: unknown.tld (not counted)",
|
||||
ts + " <t> out; recipient domain not in domains db: gmail.com (not counted)",
|
||||
ts + " <t> out; skipped: unknown_domain",
|
||||
}},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
plan, err := c.plan()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
l, buf := newLogger("text")
|
||||
l.Outcome(outcome(plan), nil)
|
||||
if got, want := buf.String(), strings.Join(c.want, "\n")+"\n"; got != want {
|
||||
t.Errorf("--- got ---\n%s--- want ---\n%s", got, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestFailed_RedisError(t *testing.T) {
|
||||
plan, _ := planner().PlanIn("tr", "s@gmail.com", "y@wearemiraclesnow.com", false)
|
||||
l, buf := newLogger("text")
|
||||
l.Failed(plan, errors.New("redis incr: dial tcp 127.0.0.1:6379: connect: connection refused"))
|
||||
got := buf.String()
|
||||
if !strings.HasPrefix(got, ts+" <tr> in; request mail_from=s@gmail.com") {
|
||||
t.Errorf("missing request line:\n%s", got)
|
||||
}
|
||||
if !strings.Contains(got, "determined limit (level=[O]") {
|
||||
t.Errorf("lookup trail missing before the error:\n%s", got)
|
||||
}
|
||||
if !strings.HasSuffix(got, ts+" <tr> in; redis error: redis incr: dial tcp 127.0.0.1:6379: connect: connection refused (nothing counted)\n") {
|
||||
t.Errorf("missing error line:\n%s", got)
|
||||
}
|
||||
if strings.Contains(got, "tracked") {
|
||||
t.Errorf("no tracked lines expected when nothing was counted:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNotificationLines(t *testing.T) {
|
||||
l, buf := newLogger("text")
|
||||
ev := notify.Event{TraceID: "ad8525", Route: "out", Level: "O", ID: "116681", Flow: "o", Address: "noah@wearemiraclesnow.com", Count: 2200, Limit: 2200}
|
||||
u := "https://admin.migadu.com/api/limits?level=O&id=116681&flow=o&period=d&by=noah%40wearemiraclesnow.com&count=2200"
|
||||
l.NotificationSent(ev, u, 1)
|
||||
l.NotificationFailed(ev, u, 5, errors.New("unexpected status 500"))
|
||||
want := ts + ` <ad8525> out; sent limit exceeded notification to url "` + u + `"` + "\n" +
|
||||
ts + ` <ad8525> out; failed limit exceeded notification to url "` + u + `" after 5 attempts: unexpected status 500` + "\n"
|
||||
if got := buf.String(); got != want {
|
||||
t.Errorf("--- got ---\n%s--- want ---\n%s", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEmptyTraceID(t *testing.T) {
|
||||
plan, _ := planner().PlanIn("", "s@gmail.com", "y@wearemiraclesnow.com", true)
|
||||
l, buf := newLogger("text")
|
||||
l.Outcome(outcome(plan), nil)
|
||||
if !strings.HasPrefix(buf.String(), ts+" <-> in; request") {
|
||||
t.Errorf("empty trace id should print as <->:\n%s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
// Domains with thousands of per-address overrides (boxmailhere.com has 9 798)
|
||||
// must not dump them all: base keys + the address's own overrides + a count.
|
||||
func TestLimitsMapSummarised(t *testing.T) {
|
||||
plan, _ := planner().PlanOut("t", "bob@many.example", "", "ext@gmail.com")
|
||||
l, buf := newLogger("text")
|
||||
l.Outcome(outcome(plan, 1, 1, 1), nil)
|
||||
want := ts + " <t> out; domain: many.example limits_map={A/i=1, A/o=2, bob@/i=3, bob@/o=4} (+2 other overrides)\n"
|
||||
if !strings.Contains(buf.String(), want) {
|
||||
t.Errorf("--- got ---\n%s--- want line ---\n%s", buf.String(), want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOutcome_JSON(t *testing.T) {
|
||||
plan, _ := planner().PlanOut("ad8525", "noah@wearemiraclesnow.com", "noah@wearemiraclesnow.com", "y@example.org")
|
||||
l, buf := newLogger("json")
|
||||
l.Outcome(outcome(plan, 12, 1830, 2200), []bool{false, false, true})
|
||||
|
||||
lines := strings.Split(strings.TrimSpace(buf.String()), "\n")
|
||||
var events []string
|
||||
var records []map[string]any
|
||||
for _, line := range lines {
|
||||
var rec map[string]any
|
||||
if err := json.Unmarshal([]byte(line), &rec); err != nil {
|
||||
t.Fatalf("line is not JSON: %q: %v", line, err)
|
||||
}
|
||||
if rec["trace_id"] != "ad8525" || rec["route"] != "out" || rec["ts"] != "2026-10-06T01:12:06.412Z" {
|
||||
t.Errorf("common fields wrong: %v", rec)
|
||||
}
|
||||
events = append(events, rec["event"].(string))
|
||||
records = append(records, rec)
|
||||
}
|
||||
wantEvents := []string{"request", "local_domain", "limits", "determined_limit", "determined_limit", "determined_limit",
|
||||
"not_in_db", "tracked", "tracked", "tracked", "limit_reached"}
|
||||
if strings.Join(events, ",") != strings.Join(wantEvents, ",") {
|
||||
t.Errorf("events = %v\nwant %v", events, wantEvents)
|
||||
}
|
||||
o := records[9]
|
||||
if o["level"] != "O" || o["id"] != "116681" || o["count"].(float64) != 2200 || o["limit"].(float64) != 2200 || o["reached"] != true ||
|
||||
o["key"] != "cnt/O/116681/d/2026-10-06/o" || o["flow"] != "o" {
|
||||
t.Errorf("tracked O record = %v", o)
|
||||
}
|
||||
if r := records[10]; r["notifying"] != true || r["url_name"] != "org_url" || r["by"] != "noah@wearemiraclesnow.com" {
|
||||
t.Errorf("limit_reached record = %v", r)
|
||||
}
|
||||
if r := records[1]; r["domain"] != "wearemiraclesnow.com" || r["org_id"].(float64) != 116681 {
|
||||
t.Errorf("local_domain record = %v", r)
|
||||
}
|
||||
}
|
||||
|
||||
// countingWriter counts Write calls: each block must be one write so
|
||||
// concurrent requests never interleave their lines.
|
||||
type countingWriter struct {
|
||||
mu sync.Mutex
|
||||
writes int
|
||||
buf bytes.Buffer
|
||||
}
|
||||
|
||||
func (w *countingWriter) Write(p []byte) (int, error) {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
w.writes++
|
||||
return w.buf.Write(p)
|
||||
}
|
||||
|
||||
func TestBlockIsSingleWrite(t *testing.T) {
|
||||
w := &countingWriter{}
|
||||
l := New(w, "text", func() time.Time { return clock })
|
||||
plan, _ := planner().PlanOut("t", "noah@wearemiraclesnow.com", "", "y@wearemiraclesnow.com")
|
||||
l.Outcome(outcome(plan, 1, 1, 1, 1, 1, 1), nil)
|
||||
l.Failed(plan, errors.New("x"))
|
||||
l.NotificationSent(notify.Event{}, "u", 1)
|
||||
if w.writes != 3 {
|
||||
t.Errorf("writes = %d, want 3 (one per block/line)", w.writes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestErrorLine(t *testing.T) {
|
||||
l, buf := newLogger("text")
|
||||
l.Error("out", "tr", errors.New("lookup sender domain x.com: cdb read error"))
|
||||
want := ts + " <tr> out; error: lookup sender domain x.com: cdb read error (nothing counted)\n"
|
||||
if got := buf.String(); got != want {
|
||||
t.Errorf("got %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// --- review round 1 lines -----------------------------------------------------
|
||||
|
||||
func TestRequestLine_LoginSuffixAndUnusable(t *testing.T) {
|
||||
plan, err := planner().Plan(counting.Job{Route: counting.RouteOut, TraceID: "m1", AuthUser: "x@wearemiraclesnow.com", LoginSuffix: "SUPPORTTOKEN",
|
||||
MailFrom: "x@wearemiraclesnow.com", MailFromUsable: true,
|
||||
RecipientUnusable: counting.Unusable{Role: "recipient", Value: "u@[192.0.2.1]", Reason: "invalid e-mail address"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
l, buf := newLogger("text")
|
||||
l.Outcome(outcome(plan, 1, 1, 1), nil)
|
||||
got := buf.String()
|
||||
if !strings.Contains(got, ts+" <m1> out; request auth_user=x@wearemiraclesnow.com recipient= mail_from=x@wearemiraclesnow.com login_suffix=SUPPORTTOKEN\n") {
|
||||
t.Errorf("request line should show the login suffix:\n%s", got)
|
||||
}
|
||||
if !strings.Contains(got, ts+" <m1> out; recipient address not usable: u@[192.0.2.1] (invalid e-mail address, not counted)\n") {
|
||||
t.Errorf("missing unusable line:\n%s", got)
|
||||
}
|
||||
if !strings.Contains(got, "tracked address counters for x@wearemiraclesnow.com => count=1") {
|
||||
t.Errorf("sender should still be tracked:\n%s", got)
|
||||
}
|
||||
|
||||
plan, _ = planner().Plan(counting.Job{Route: counting.RouteOut, TraceID: "m2", Recipient: "y@wearemiraclesnow.com",
|
||||
SenderUnusable: counting.Unusable{Role: "sender", Reason: "missing"}})
|
||||
l, buf = newLogger("text")
|
||||
l.Outcome(outcome(plan, 1, 1, 1), nil)
|
||||
if got := buf.String(); !strings.Contains(got, " <m2> out; sender address not usable: <empty> (missing, not counted)\n") || strings.Contains(got, "login_suffix") {
|
||||
t.Errorf("missing sender line / unexpected suffix:\n%s", got)
|
||||
}
|
||||
|
||||
l, buf = newLogger("json")
|
||||
l.Outcome(outcome(plan, 1, 1, 1), nil)
|
||||
if got := buf.String(); !strings.Contains(got, `"event":"not_usable"`) || !strings.Contains(got, `"role":"sender"`) || !strings.Contains(got, `"reason":"missing"`) {
|
||||
t.Errorf("json not_usable event missing:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A key whose own INCR failed gets an error line instead of a tracked line;
|
||||
// the other keys are tracked normally.
|
||||
func TestOutcome_PerKeyError(t *testing.T) {
|
||||
plan, _ := planner().PlanOut("p1", "noah@wearemiraclesnow.com", "", "s@gmail.com")
|
||||
out := outcome(plan, 100, 1830, 0)
|
||||
out.Results[2].Err = errors.New("redis incr cnt/O/116681/d/2026-10-06/o: ERR value is not an integer or out of range")
|
||||
l, buf := newLogger("text")
|
||||
l.Outcome(out, []bool{true, false, false})
|
||||
got := buf.String()
|
||||
if !strings.Contains(got, ts+" <p1> out; tracked address counters for noah@wearemiraclesnow.com => count=100 limit=100 reached=true\n") {
|
||||
t.Errorf("A should be tracked and reached:\n%s", got)
|
||||
}
|
||||
if !strings.Contains(got, ts+" <p1> out; redis error on key cnt/O/116681/d/2026-10-06/o: redis incr cnt/O/116681/d/2026-10-06/o: ERR value is not an integer or out of range (not counted)\n") {
|
||||
t.Errorf("missing per-key error line:\n%s", got)
|
||||
}
|
||||
if strings.Contains(got, "tracked account counters") {
|
||||
t.Errorf("failed key must not be reported as tracked:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A lost EXEC reply: the counts come from a master re-read and the block says so.
|
||||
func TestOutcome_UnknownReRead(t *testing.T) {
|
||||
plan, _ := planner().PlanOut("u1", "noah@wearemiraclesnow.com", "", "s@gmail.com")
|
||||
out := outcome(plan, 100, 1830, 2200)
|
||||
out.Unknown = errors.New("redis incr: read tcp 10.0.0.1:1->10.0.0.2:6379: i/o timeout")
|
||||
l, buf := newLogger("text")
|
||||
l.Outcome(out, []bool{true, false, true})
|
||||
got := buf.String()
|
||||
if !strings.Contains(got, ts+" <u1> out; redis outcome unknown: redis incr: read tcp 10.0.0.1:1->10.0.0.2:6379: i/o timeout, re-read from master\n") {
|
||||
t.Errorf("missing unknown-outcome line:\n%s", got)
|
||||
}
|
||||
if !strings.Contains(got, "tracked address counters for noah@wearemiraclesnow.com => count=100 limit=100 reached=true (re-read)\n") {
|
||||
t.Errorf("tracked lines should be marked re-read:\n%s", got)
|
||||
}
|
||||
if !strings.Contains(got, "limit reached (level=[A]") {
|
||||
t.Errorf("crossing after re-read should still be reported:\n%s", got)
|
||||
}
|
||||
if strings.Contains(got, "nothing counted") {
|
||||
t.Errorf("an unknown outcome is not 'nothing counted':\n%s", got)
|
||||
}
|
||||
l, buf = newLogger("json")
|
||||
l.Outcome(out, nil)
|
||||
if got := buf.String(); !strings.Contains(got, `"event":"redis_unknown"`) || !strings.Contains(got, `"reread":true`) {
|
||||
t.Errorf("json unknown event missing:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDropped(t *testing.T) {
|
||||
l, buf := newLogger("text")
|
||||
l.Dropped(counting.RouteIn, "d1")
|
||||
if got := buf.String(); got != ts+" <d1> in; dropped: queue full (not counted)\n" {
|
||||
t.Errorf("got %q", got)
|
||||
}
|
||||
l, buf = newLogger("json")
|
||||
l.Dropped(counting.RouteOut, "d2")
|
||||
if got := buf.String(); !strings.Contains(got, `"event":"dropped"`) || !strings.Contains(got, `"route":"out"`) {
|
||||
t.Errorf("got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTrail_MalformedRow(t *testing.T) {
|
||||
bad := table{"good.com": {Name: "good.com", OrgID: 7, Limits: domains.ParseLimits("A/o=5")}}
|
||||
p := &counting.Planner{Lookup: malformedTable{bad}, Now: func() time.Time { return clock }}
|
||||
plan, err := p.PlanOut("mf", "a@good.com", "", "b@bad.com")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
l, buf := newLogger("text")
|
||||
l.Outcome(outcome(plan, 1, 1, 1), nil)
|
||||
if !strings.Contains(buf.String(), ts+" <mf> out; recipient domain row malformed in domains db: bad.com: "+`domain bad.com: invalid org id "x12"`+" (not counted)\n") {
|
||||
t.Errorf("missing malformed line:\n%s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
type malformedTable struct{ table }
|
||||
|
||||
func (m malformedTable) Lookup(d string) (*domains.Domain, error) {
|
||||
if d == "bad.com" {
|
||||
return domains.ParseRecord("bad.com", "x12|A/i=5")
|
||||
}
|
||||
return m.table.Lookup(d)
|
||||
}
|
||||
@@ -0,0 +1,151 @@
|
||||
// Package watcher reloads the domains CDB when the file changes on disk.
|
||||
//
|
||||
// It is the table mode of rcptd's watcher, trimmed: it watches the files
|
||||
// themselves (kqueue on FreeBSD/macOS only reports in-place writes through a
|
||||
// file watch) and their directories (an atomic `mv` over the file shows up as
|
||||
// a Create in the directory), debounces bursts and calls one reload function.
|
||||
package watcher
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/fsnotify/fsnotify"
|
||||
)
|
||||
|
||||
// ReloadFunc is called, debounced, after a watched file changed. Its error is
|
||||
// logged; the watcher keeps running either way.
|
||||
type ReloadFunc func() error
|
||||
|
||||
// Watcher watches a set of files and calls reload on change.
|
||||
type Watcher struct {
|
||||
reload ReloadFunc
|
||||
fsw *fsnotify.Watcher
|
||||
files map[string]bool // absolute paths we care about
|
||||
debounce time.Duration
|
||||
|
||||
done chan struct{}
|
||||
stopOnce sync.Once
|
||||
wg sync.WaitGroup
|
||||
|
||||
timerMu sync.Mutex
|
||||
timer *time.Timer
|
||||
runMu sync.Mutex // serialises reload calls
|
||||
}
|
||||
|
||||
// New watches paths. Their directories must exist; the files themselves may
|
||||
// appear later (the first deploy of the CDB triggers a reload).
|
||||
func New(paths []string, reload ReloadFunc) (*Watcher, error) {
|
||||
fsw, err := fsnotify.NewWatcher()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
w := &Watcher{
|
||||
reload: reload,
|
||||
fsw: fsw,
|
||||
files: make(map[string]bool, len(paths)),
|
||||
debounce: 500 * time.Millisecond,
|
||||
done: make(chan struct{}),
|
||||
}
|
||||
dirs := map[string]bool{}
|
||||
for _, p := range paths {
|
||||
abs, err := filepath.Abs(p)
|
||||
if err != nil {
|
||||
fsw.Close()
|
||||
return nil, fmt.Errorf("resolve path %s: %w", p, err)
|
||||
}
|
||||
w.files[abs] = true
|
||||
if dir := filepath.Dir(abs); !dirs[dir] {
|
||||
if err := fsw.Add(dir); err != nil {
|
||||
fsw.Close()
|
||||
return nil, fmt.Errorf("watch directory %s: %w", dir, err)
|
||||
}
|
||||
dirs[dir] = true
|
||||
}
|
||||
}
|
||||
for abs := range w.files {
|
||||
if _, err := os.Stat(abs); err == nil {
|
||||
_ = fsw.Add(abs)
|
||||
}
|
||||
}
|
||||
return w, nil
|
||||
}
|
||||
|
||||
// Start begins watching in a background goroutine.
|
||||
func (w *Watcher) Start() {
|
||||
w.wg.Add(1)
|
||||
go func() {
|
||||
defer w.wg.Done()
|
||||
for {
|
||||
select {
|
||||
case event, ok := <-w.fsw.Events:
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if event.Op&(fsnotify.Write|fsnotify.Create) == 0 {
|
||||
continue
|
||||
}
|
||||
abs, err := filepath.Abs(event.Name)
|
||||
if err != nil || !w.files[abs] {
|
||||
continue
|
||||
}
|
||||
// After a rename-over the old file watch is stale: re-add it so
|
||||
// in-place writes to the new inode are seen too.
|
||||
if event.Op&fsnotify.Create != 0 {
|
||||
_ = w.fsw.Add(abs)
|
||||
}
|
||||
w.schedule()
|
||||
case err, ok := <-w.fsw.Errors:
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
slog.Error("file watcher error", "error", err)
|
||||
case <-w.done:
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
// schedule (re)arms the debounce timer.
|
||||
func (w *Watcher) schedule() {
|
||||
w.timerMu.Lock()
|
||||
defer w.timerMu.Unlock()
|
||||
if w.timer != nil {
|
||||
w.timer.Stop()
|
||||
}
|
||||
w.timer = time.AfterFunc(w.debounce, w.run)
|
||||
}
|
||||
|
||||
func (w *Watcher) run() {
|
||||
select {
|
||||
case <-w.done:
|
||||
return
|
||||
default:
|
||||
}
|
||||
w.runMu.Lock()
|
||||
defer w.runMu.Unlock()
|
||||
if err := w.reload(); err != nil {
|
||||
slog.Error("reload after file change failed", "error", err)
|
||||
return
|
||||
}
|
||||
slog.Info("reloaded after file change")
|
||||
}
|
||||
|
||||
// Stop ends watching and waits for the goroutine; safe to call twice.
|
||||
func (w *Watcher) Stop() {
|
||||
w.stopOnce.Do(func() {
|
||||
w.timerMu.Lock()
|
||||
if w.timer != nil {
|
||||
w.timer.Stop()
|
||||
}
|
||||
w.timerMu.Unlock()
|
||||
close(w.done)
|
||||
w.fsw.Close()
|
||||
w.wg.Wait()
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,189 @@
|
||||
package watcher
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func write(t *testing.T, path, content string) {
|
||||
t.Helper()
|
||||
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// replace mimics the agent's atomic deploy: write a temp file next to the
|
||||
// target and rename it over the target.
|
||||
func replace(t *testing.T, path, content string) {
|
||||
t.Helper()
|
||||
tmp := path + ".tmp"
|
||||
write(t, tmp, content)
|
||||
if err := os.Rename(tmp, path); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
type counter struct {
|
||||
n atomic.Int32
|
||||
done chan struct{}
|
||||
err error
|
||||
}
|
||||
|
||||
func newCounter() *counter { return &counter{done: make(chan struct{}, 16)} }
|
||||
|
||||
func (c *counter) reload() error {
|
||||
c.n.Add(1)
|
||||
select {
|
||||
case c.done <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
return c.err
|
||||
}
|
||||
|
||||
func (c *counter) wait(t *testing.T) {
|
||||
t.Helper()
|
||||
select {
|
||||
case <-c.done:
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("timeout waiting for reload")
|
||||
}
|
||||
}
|
||||
|
||||
// start creates and starts a watcher; debounce must be set before Start so
|
||||
// the watcher goroutine never races with the test.
|
||||
func start(t *testing.T, path string, c *counter, debounce time.Duration) *Watcher {
|
||||
t.Helper()
|
||||
w, err := New([]string{path}, c.reload)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
w.debounce = debounce
|
||||
w.Start()
|
||||
t.Cleanup(w.Stop)
|
||||
time.Sleep(50 * time.Millisecond) // let the watches settle
|
||||
return w
|
||||
}
|
||||
|
||||
func TestWatcher_ReloadsOnInPlaceWrite(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "rspamd_domains_db")
|
||||
write(t, path, "v1")
|
||||
c := newCounter()
|
||||
start(t, path, c, 100*time.Millisecond)
|
||||
|
||||
write(t, path, "v2")
|
||||
c.wait(t)
|
||||
if c.n.Load() < 1 {
|
||||
t.Errorf("reloads = %d, want >= 1", c.n.Load())
|
||||
}
|
||||
}
|
||||
|
||||
// The agent deploys the CDB with mv; after the first replacement the old
|
||||
// kqueue file watch is stale, so a second replacement must still be seen.
|
||||
func TestWatcher_ReloadsOnAtomicReplaceTwice(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "rspamd_domains_db")
|
||||
write(t, path, "v1")
|
||||
c := newCounter()
|
||||
start(t, path, c, 100*time.Millisecond)
|
||||
|
||||
replace(t, path, "v2")
|
||||
c.wait(t)
|
||||
first := c.n.Load()
|
||||
|
||||
time.Sleep(150 * time.Millisecond)
|
||||
replace(t, path, "v3")
|
||||
c.wait(t)
|
||||
if c.n.Load() <= first {
|
||||
t.Errorf("second replacement not detected: reloads %d -> %d", first, c.n.Load())
|
||||
}
|
||||
}
|
||||
|
||||
func TestWatcher_IgnoresOtherFilesInDir(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "rspamd_domains_db")
|
||||
write(t, path, "v1")
|
||||
c := newCounter()
|
||||
start(t, path, c, 100*time.Millisecond)
|
||||
|
||||
write(t, filepath.Join(dir, "other_file"), "x")
|
||||
time.Sleep(400 * time.Millisecond)
|
||||
if c.n.Load() != 0 {
|
||||
t.Errorf("reloads = %d for an unrelated file, want 0", c.n.Load())
|
||||
}
|
||||
}
|
||||
|
||||
func TestWatcher_DebouncesBurst(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "rspamd_domains_db")
|
||||
write(t, path, "v1")
|
||||
c := newCounter()
|
||||
start(t, path, c, 200*time.Millisecond)
|
||||
|
||||
for i := 0; i < 5; i++ {
|
||||
write(t, path, "burst")
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
c.wait(t)
|
||||
time.Sleep(500 * time.Millisecond)
|
||||
if n := c.n.Load(); n > 2 {
|
||||
t.Errorf("reloads = %d, want 1-2 (debounced)", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWatcher_ContinuesAfterReloadError(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "rspamd_domains_db")
|
||||
write(t, path, "v1")
|
||||
c := newCounter()
|
||||
c.err = errors.New("boom")
|
||||
start(t, path, c, 100*time.Millisecond)
|
||||
|
||||
write(t, path, "v2")
|
||||
c.wait(t)
|
||||
time.Sleep(150 * time.Millisecond)
|
||||
write(t, path, "v3")
|
||||
c.wait(t)
|
||||
if c.n.Load() < 2 {
|
||||
t.Errorf("reloads = %d, want >= 2 (a failing reload must not stop the watcher)", c.n.Load())
|
||||
}
|
||||
}
|
||||
|
||||
func TestWatcher_StopPreventsReload(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "rspamd_domains_db")
|
||||
write(t, path, "v1")
|
||||
c := newCounter()
|
||||
w := start(t, path, c, 100*time.Millisecond)
|
||||
w.Stop()
|
||||
|
||||
write(t, path, "v2")
|
||||
time.Sleep(400 * time.Millisecond)
|
||||
if c.n.Load() != 0 {
|
||||
t.Errorf("reloads = %d after Stop, want 0", c.n.Load())
|
||||
}
|
||||
w.Stop() // idempotent
|
||||
}
|
||||
|
||||
func TestNew_MissingDirectory(t *testing.T) {
|
||||
if _, err := New([]string{"/nonexistent-dir-for-test/rspamd_domains_db"}, func() error { return nil }); err == nil {
|
||||
t.Fatal("expected error for a missing directory")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNew_MissingFileInExistingDirIsFine(t *testing.T) {
|
||||
// The CDB may be deployed after the daemon starts; the directory watch
|
||||
// picks up its creation.
|
||||
path := filepath.Join(t.TempDir(), "rspamd_domains_db")
|
||||
c := newCounter()
|
||||
w, err := New([]string{path}, c.reload)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
w.debounce = 100 * time.Millisecond
|
||||
w.Start()
|
||||
defer w.Stop()
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
|
||||
write(t, path, "v1")
|
||||
c.wait(t)
|
||||
}
|
||||
@@ -0,0 +1,480 @@
|
||||
# Plan: `migadu_counters` — real-time mail counting service (Go)
|
||||
|
||||
## Context
|
||||
|
||||
Daily send/receive limits used to be counted and enforced by the rspamd Lua plugin
|
||||
`ansible-freebsd2/roles/rspamd-postfix/templates/rspamd/plugins.d/migadu_counters.lua`
|
||||
(Redis counters per address + org, soft-reject/reject on breach, web app notified once).
|
||||
Mail now flows mizu → mailqueuer and bypasses rspamd, so nothing counts in real time.
|
||||
`stats-merger/internal/quota` recomputes usage from the delivery ledger after each merge, but it is
|
||||
reporting-only today: only fernrohr reads `/api/quota/{status,stats,overquota}`; nothing calls `/check`.
|
||||
|
||||
`migadu_counters` is a new standalone Go service that becomes the single counting source and the
|
||||
enforcement trigger:
|
||||
- mailqueuer calls it **fire-and-forget**, once per ingested recipient, from its publish loop;
|
||||
- one route per flow: `POST /in`, `POST /out` (`/rel` later, not planned here);
|
||||
- it increments Redis counters at **Address / Domain / Organization** level, in the key format the
|
||||
web app already reads. The domain level is turned back on (Lua defined it but never tracked it);
|
||||
- when an increment crosses a limit it **notifies the web app** (`migadu_web` `GET /api/limits`), which
|
||||
decides what to do (notify the user, remove sending/receiving);
|
||||
- a small CLI, `counterctl`, shows an address's incoming/outgoing counters for today or a given date
|
||||
(see "CLI: counterctl" below).
|
||||
|
||||
The service returns no verdicts and enforces nothing itself.
|
||||
Out of scope: `/rel`, the mailqueuer client change, retiring stats-merger quota, the web app's `D` branch,
|
||||
deployment (see prerequisites at the end).
|
||||
|
||||
## How it fits together
|
||||
|
||||
```
|
||||
sender ──SMTP──▶ mizu ──POST /ingest──▶ mailqueuer
|
||||
X-Trace-Id, X-Mail-From, │ one queue message per recipient;
|
||||
X-Mail-To, X-Auth-User, │ after each successful publish:
|
||||
X-Junk │ fire-and-forget HTTP call, reply ignored
|
||||
▼
|
||||
POST /out {"trace_id","auth_user","mail_from","recipient"} ← X-Auth-User set
|
||||
POST /in {"trace_id","mail_from","recipient","junk"} ← X-Auth-User empty
|
||||
Authorization: Bearer <token> (mailqueuer sends it, migadu_counters checks it; 401 if wrong)
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────┐
|
||||
│ migadu_counters (daemon :8095) │
|
||||
│ ┌─────────────────────────────────┐ │ rejected here, nothing counted:
|
||||
│ │ HTTP validation layer │ │ 405 method · 413 body > 64 KB
|
||||
│ │ method → size → bearer token → │ │ 401 bad/missing token · 403 source IP
|
||||
│ │ source IP → content-type → JSON │ │ 415 content-type · 400 json / fields /
|
||||
│ │ → fields → e-mail syntax │ │ e-mail syntax · 503 handler timeout
|
||||
│ └───────────────┬─────────────────┘ │
|
||||
│ ▼ validated request │
|
||||
rspamd_domains_db ───▶│ counting: 1. domain in CDB? → org_id, limits │
|
||||
(CDB: domain → │ 2. MULTI: INCR A/D/O keys, EXPIRE 31 d │
|
||||
org_id | limits) │ 3. did this INCR cross a limit? → notify │
|
||||
└───┬─────────────────────────┬───────┘ admin :9095 /health /metrics
|
||||
│ │
|
||||
Redis Sentinel ◀──┘ └──────▶ GET https://admin.migadu.com/api/limits
|
||||
writes → master ?level=&id=&flow=&period=d&by=&count=
|
||||
reads → replicas (basic auth) → migadu_web notifies the user
|
||||
DB 0: cnt/<A|D|O>/<id>/d/<YYYY-MM-DD>/<i|o> or disables sending/receiving
|
||||
|
||||
counterctl --address x@y [--date YYYY-MM-DD] ──GET /counters (Bearer)──▶ migadu_counters ──▶ Redis replicas
|
||||
```
|
||||
|
||||
## Existing pieces reused or matched
|
||||
|
||||
- **Limits**: the CDB the Lua plugin reads, `/usr/local/etc/rspamd/db/rspamd_domains_db`, built by
|
||||
`ansible-freebsd2/roles/agent/templates/extract.sh.j2:478` with tinycdb `cdb -m` (no NUL terminators,
|
||||
exact-byte lookups like rcptd). Key = `domain`, value = `<org_id>|O/i=..,O/o=..,D/i=..,A/o=..,alice@/o=..`.
|
||||
Parsing and limit resolution follow `stats-merger/internal/quota/quota.go` (`ParseLimits` L66,
|
||||
`DomainLimits.limit` L116): keys lowercased, malformed pairs skipped, `<local>@/<flow>` overrides
|
||||
`A/<flow>`, missing key = unlimited. A domain present in the CDB is "Migadu-hosted".
|
||||
- **Redis keys** — must match the web app's key builder (`migadu_web/apps/core/lib/core/redis_sentinel.ex:73-103`):
|
||||
`cnt/<A|D|O>/<id>/d/<YYYY-MM-DD>/<i|o>` with id = address / domain / org_id, **date in UTC**
|
||||
(web app uses `Date.utc_today()`), **Redis DB 0** (web app reads `:redis_sentinel_db0`; rspamd uses
|
||||
`redis_rspamd_db: 0`). TTL 31 days. DB 1 and the `/s` storage keys belong to the web app; never touched.
|
||||
The web app may DEL today's keys to reset a customer (`redis_sentinel.ex:145-153`); counting must keep
|
||||
working after that, and a second crossing must notify again.
|
||||
- **Web app notification**: `GET https://admin.migadu.com/api/limits?level=&id=&flow=&period=&by=&count=`
|
||||
with basic auth (`apps/admin_web/lib/admin_web/controllers/internal_api/notification_controller.ex:33`,
|
||||
`router.ex:99` `Plug.BasicAuth`). All six params are pattern-matched, so all must be present.
|
||||
`id` = address for A, domain name for D, org id for O. Credentials: `vault_migadu_web.internal_api_basic_auth_*`
|
||||
(see `rspamd-postfix/templates/rspamd/local.d/migadu_counters.conf.j2`).
|
||||
- **Redis topology**: `ansible-freebsd2/roles/redis-sentinel`, `inventory/hosts.ini:160`: hosts `redisSentinel0-2`,
|
||||
Redis :6379 + Sentinel :26379, master name `mymaster`, 1 master + 2 replicas, one password
|
||||
(`vault_redis_sentinel_password`) for Redis; Sentinel itself has no `requirepass`.
|
||||
- **Service conventions** from `~/Work/rcptd` (`cmd/rcptd/main.go`): TOML config via `-config` + default paths,
|
||||
bearer `auth_token` with env override, slog, CDB behind `atomic.Pointer` with a 30 s grace close,
|
||||
SIGHUP + fsnotify reload (`internal/watcher/watcher.go` table mode), admin port with `/health` + `/metrics`,
|
||||
graceful shutdown. No dependency on rcptd's `shared` module (not present locally).
|
||||
- **Libraries** (in the module cache): `github.com/colinmarc/cdb`, `github.com/BurntSushi/toml` v1.6.0,
|
||||
`github.com/fsnotify/fsnotify` v1.9.0, `github.com/redis/go-redis/v9` v9.7.3, `github.com/prometheus/client_golang`;
|
||||
tests: `github.com/alicebob/miniredis/v2` v2.34.0.
|
||||
|
||||
## API (called by mailqueuer, fire-and-forget, one request per recipient)
|
||||
|
||||
> **Revised 2026-10-14 (review round 1, `review/README.md` #1–#6):** `/in` and `/out` now validate, enqueue
|
||||
> and answer `202 {"accepted":true}`; counting runs in a worker pool (`queue_size`, `workers`) under its own
|
||||
> time budget, and the counted values appear only in the counting log and `GET /counters`. A full queue answers
|
||||
> `503 {"error":"queue full"}`. Only the counted sides are validated (`mail_from` never rejects; `auth_user` and
|
||||
> `recipient` independently), IDN domains are converted to punycode, `user@dom@SUFFIX` logins count under the
|
||||
> base address. `INCR` replaces `INCRBY`; per-key EXEC failures and lost replies are handled individually
|
||||
> (re-read from the master). Parsed CDB records are cached (`max_cached_domains`). `auth_token` is mandatory,
|
||||
> unknown config keys are fatal, the default API port is `:5555`. The `200 {"counted":…}` contract below is
|
||||
> the original design and is superseded by the README.
|
||||
|
||||
mailqueuer's ingest handler is already per recipient: it loops over `toPublish`
|
||||
(`mailqueuer/internal/handler/ingest.go:235-306`), publishes one queue message per recipient (own uuid `id`,
|
||||
L237) and marks dedup only after a successful publish (L263-272). Both routes are called from inside that loop,
|
||||
right after a successful `Publish`, with exactly one recipient. The route is chosen by `X-Auth-User`:
|
||||
non-empty → `/out`, empty → `/in`. mailqueuer never waits for or acts on the response.
|
||||
|
||||
**Authentication**: every request carries `Authorization: Bearer <token>` — mailqueuer sends it (the same
|
||||
header it already sends to rcptd, `internal/routing/routing.go:256-258`), migadu_counters validates it
|
||||
(constant-time compare against `auth_token`; missing/wrong → `401`, nothing counted). One shared token,
|
||||
set in both configs; no other auth. All routes take a JSON body ≤ 64 KB. Processing is **synchronous**: one Redis
|
||||
round trip, then the response. Only web-app notifications run in the background.
|
||||
- `200` `{"counted":[{"level":"A","id":"x@example.com","flow":"o","count":12,"limit":100,"reached":false},…]}`
|
||||
(`reached: true` on the increment that crossed the limit → notification triggered). Skipped requests return
|
||||
`{"counted":[],"skipped":"exempt|unknown_domain|junk"}`.
|
||||
- `400` invalid body/address, `401` bad token, `503` Redis error (nothing counted).
|
||||
|
||||
### `POST /out` — authenticated submission
|
||||
```json
|
||||
{ "trace_id": "…", "auth_user": "x@example.com", "mail_from": "x@example.com", "recipient": "y@example.com" }
|
||||
```
|
||||
1. **Sender outgoing +1** on A(`auth_user`) / D(its domain) / O(its org) — only if `auth_user`'s domain is in the CDB.
|
||||
2. **Recipient incoming +1** on A / D / O of the recipient's domain row — only if that domain is in the CDB
|
||||
(any Migadu-hosted domain, any org). That mail never passes `/in`.
|
||||
|
||||
Example: `x@example.com` → `y@example.com`, `s@gmail.com` = two `/out` calls.
|
||||
Call 1: out A x +1, D example.com +1, O +1; in A y +1, D example.com +1, O +1. Call 2: out +1 on each; gmail not
|
||||
in CDB → no incoming. Totals: outgoing +2 per level, incoming for y +1 per level.
|
||||
|
||||
### `POST /in` — inbound mail from outside
|
||||
```json
|
||||
{ "trace_id": "…", "mail_from": "s@gmail.com", "recipient": "y@example.com", "junk": false }
|
||||
```
|
||||
Recipient incoming +1 on A / D / O if its domain is in the CDB. The sender is never counted.
|
||||
`junk = true` is skipped unless `count_junk = true` (Lua only counted `no action` mail).
|
||||
|
||||
### `GET /counters?address=…` or `?domain=…` `[&date=YYYY-MM-DD]`
|
||||
Read-only view (used by `counterctl`), served from the **read client**. `date` is an ISO 8601 calendar date
|
||||
(`YYYY-MM-DD`, interpreted as a UTC day); default = today UTC; a future date is `400`. Keys live 31 days, so
|
||||
older dates return zeros. Exactly one of `address`/`domain` is required.
|
||||
```json
|
||||
{ "query": "me@example.com", "mode": "address", "date": "2026-10-13",
|
||||
"managed": true, "exempt": false, "domain": "example.com", "org_id": 1234,
|
||||
"levels": [
|
||||
{"level": "A", "id": "me@example.com", "in": 12, "in_limit": 500, "out": 3, "out_limit": 200},
|
||||
{"level": "D", "id": "example.com", "in": 40, "in_limit": 1000, "out": 9, "out_limit": 400},
|
||||
{"level": "O", "id": "1234", "in": 120, "in_limit": 5000, "out": 30, "out_limit": 1000} ] }
|
||||
```
|
||||
`in_limit`/`out_limit` are `null` when unlimited. `mode: "domain"` returns the D and O rows only.
|
||||
`managed: false` (domain not in the CDB) still returns the raw counters with `null` limits — Lua-era keys may exist.
|
||||
Values are read with one `MGET` of the ≤ 6 keys.
|
||||
|
||||
### Shared rules
|
||||
- Addresses are trimmed and lowercased; validation = one `@`, non-empty parts, ≤ 254 chars.
|
||||
- **Exempt list** (default `support@migadu.com`, `contact@migadu.com`): if `auth_user`, `mail_from` or
|
||||
`recipient` is exempt, nothing is counted for that request (Lua: mail to/from support never costs quota).
|
||||
- `trace_id` (mizu's `X-Trace-Id`, shared by all recipients of one message) is logged for correlation only;
|
||||
mailqueuer's internal per-recipient queue id is **not** sent. No Redis idempotency keys: mailqueuer already
|
||||
guarantees at-most-once per (trace, recipient) publish; the residual case (mailqueuer restart + mizu retry)
|
||||
also double-delivers the mail, so the count stays truthful.
|
||||
- `O` and `D` limits come from the domain row that triggered the increment (sender's row for outgoing,
|
||||
recipient's row for incoming) — same caveat as Lua if domains of one org carry different `O/*` values.
|
||||
|
||||
## HTTP validation layer (mailqueuer → migadu_counters)
|
||||
|
||||
Everything mailqueuer sends passes this layer **before** any CDB or Redis access; a request that fails any step is
|
||||
answered with a JSON error `{"error":"…"}` and nothing is counted. Two parts:
|
||||
|
||||
**1. Middleware chain** (`internal/server/middleware.go`, applied in this order to `/in`, `/out`; `/counters`
|
||||
gets the same chain with method GET and no body steps):
|
||||
|
||||
| # | Step | Rule | Failure |
|
||||
|---|------|------|---------|
|
||||
| 1 | recover | panic in a handler → log with stack, keep the daemon alive | `500` |
|
||||
| 2 | method | `POST` only (`GET` for `/counters`) | `405` + `Allow` header |
|
||||
| 3 | body size | `http.MaxBytesReader` 64 KB | `413` |
|
||||
| 4 | bearer token | `Authorization: Bearer <token>`, constant-time compare with `auth_token` | `401` + `WWW-Authenticate: Bearer` |
|
||||
| 5 | source IP | optional `allowed_cidrs` (config); matched on `RemoteAddr` only — `X-Forwarded-For` is never trusted (no proxy in front) | `403` |
|
||||
| 6 | content type | `Content-Type: application/json` (parameters such as `charset` ignored) | `415` |
|
||||
| 7 | JSON decode | exactly one JSON object; syntax errors rejected; **unknown fields ignored** so mailqueuer can add fields without breaking counting | `400` |
|
||||
| 8 | request validation | per-route field rules below | `400` |
|
||||
| 9 | timeout | `http.TimeoutHandler` (`handler_timeout`, default 4 s) around the counting work | `503` |
|
||||
|
||||
Every rejection is logged at `warn` with route, remote address, status, reason and `trace_id` (if parsable) and
|
||||
counted in `requests_total{route,status}` — a burst of `400`/`401` from mailqueuer means contract drift or a
|
||||
token mismatch and should alert.
|
||||
|
||||
**2. Request validators** (`internal/api/validate.go`, pure functions, table-tested):
|
||||
|
||||
| Field | `/out` | `/in` | Rule |
|
||||
|-------|--------|-------|------|
|
||||
| `trace_id` | optional | optional | ≤ 128 chars, printable ASCII; control characters/newlines stripped before it reaches any log line (log-injection guard) |
|
||||
| `auth_user` | **required** | — (ignored) | e-mail syntax |
|
||||
| `recipient` | **required** | **required** | e-mail syntax |
|
||||
| `mail_from` | optional | optional (`""` = bounce sender `<>`) | e-mail syntax when non-empty |
|
||||
| `junk` | — (ignored) | optional, default `false` | boolean |
|
||||
|
||||
E-mail syntax = trimmed, lowercased, ≤ 254 chars, exactly one `@`, non-empty local part and domain, no
|
||||
whitespace/control characters; the domain is ASCII (punycode accepted). Deliberately permissive: the CDB lookup
|
||||
decides whether an address is counted, the validator only guards against garbage and log injection.
|
||||
Errors name the field: `{"error":"recipient: missing"}`, `{"error":"auth_user: invalid e-mail address"}`.
|
||||
|
||||
Config: `allowed_cidrs = ["10.0.0.0/8"]` (optional; empty = any source), `handler_timeout` (see daemon section).
|
||||
The response codes mailqueuer may see are therefore: `200` counted/skipped, `4xx` rejected by this layer,
|
||||
`503` Redis unavailable or timeout — it ignores all of them (fire-and-forget), they exist for operators and tests.
|
||||
|
||||
## Redis (`internal/store`)
|
||||
|
||||
Config has a **write** side and a **read** side:
|
||||
- **Sentinel mode (production)**: `sentinels`, `master_name`, `password` (+ optional `sentinel_password`).
|
||||
Write = `redis.NewFailoverClient` (follows failover). Read = `NewFailoverClient{ReplicaOnly: true}`; go-redis
|
||||
picks a random replica and falls back to the master when none is known (`sentinel.go:526`).
|
||||
- **Static mode (dev/tests, when `sentinels` is empty)**: `write_addr` + `read_addrs` (round-robin; defaults to
|
||||
`write_addr`).
|
||||
- Both pooled (`pool_size`); one `timeout_ms` for dial/read/write. `db` must be `0` (validated, startup error otherwise).
|
||||
|
||||
Counting = one `TxPipeline` (MULTI/EXEC) per request: `INCRBY key 1` + `EXPIRE key 2678400` for every affected key
|
||||
(≤ 6 keys). EXEC returns the new values, so no read-after-write and replica lag is irrelevant. Increments are
|
||||
atomic (fixes the Lua GET-then-INCR race); counting continues past the limit.
|
||||
|
||||
**Limit crossing**: for each incremented key with a configured limit `L`, notify iff `new − amount < L ≤ new`
|
||||
(with amount 1: `new == L`). Exactly one increment can cross, across all workers/instances, with no marker key.
|
||||
After a web-app reset (DEL) the counter crosses again and notifies again. Accepted gap (same as Lua): a limit
|
||||
lowered below the current count mid-day produces no crossing.
|
||||
|
||||
Redis errors → `503`, log, `redis_errors_total{op}`.
|
||||
|
||||
## Notification (`internal/notify`)
|
||||
|
||||
- One URL per level in `[notify]`: `org_url` (O), `address_url` (A), `domain_url` (D). Empty URL = that level is
|
||||
counted and `limit_reached_total` is incremented, but no HTTP call (useful while the web app's `D` branch is a TODO).
|
||||
Basic auth and timeout are shared.
|
||||
- `GET <url>` with template vars `$LEVEL $ID $FLOW $PERIOD(=d) $ADDRESS $COUNT $LIMIT`, each `url.QueryEscape`d.
|
||||
`ADDRESS` = the address that triggered it (sender for outgoing, recipient for incoming). `COUNT` = new value.
|
||||
- Runs in a background goroutine (semaphore of 8 to not flood the web app): up to 5 attempts, exponential
|
||||
backoff 1 s → 16 s. Final failure → error log + `notifications_total{level,result="failed"}` (alertable).
|
||||
Shutdown waits ≤ 10 s for in-flight notifications.
|
||||
|
||||
## CLI: `counterctl`
|
||||
|
||||
```
|
||||
counterctl --address me@dimitrijedimitrijevic.com # today (UTC)
|
||||
counterctl --address me@dimitrijedimitrijevic.com --date 2026-10-12
|
||||
counterctl --domain dimitrijedimitrijevic.com --json
|
||||
```
|
||||
- Flags (Go `flag`, so `-x` and `--x` both work): `--address` | `--domain` (exactly one), `--date YYYY-MM-DD`
|
||||
(ISO 8601 calendar date; default today UTC; parsed with `time.Parse("2006-01-02")`; future date = error;
|
||||
older than 31 days = warning on stderr that keys have expired), `--json` (raw API response), `--config PATH`,
|
||||
`--url` (override), `--version`.
|
||||
- Talks to the running service's `GET /counters` instead of Redis directly — one code path for key building
|
||||
and limit resolution, no Redis/CDB access needed from the operator's shell, same approach as `rcptctl`.
|
||||
Config discovery mirrors the service and `rcptctl` (`cmd/rcptctl/main.go:65-75`): `--config`, else
|
||||
`/etc/migadu_counters/config.toml`, `/usr/local/etc/migadu_counters/config.toml`; reads `addr` and
|
||||
`auth_token` only (`MIGADU_COUNTERS_AUTH_TOKEN` overrides); `addr` like `:8095` becomes
|
||||
`http://localhost:8095` (`rcptctl/api.go:52-60`); warns when the host is not local (token over plaintext).
|
||||
- Output (`text/tabwriter`), limits shown as `-` when unlimited, `!` appended when value ≥ limit:
|
||||
```
|
||||
me@dimitrijedimitrijevic.com date 2026-10-13 (UTC) domain dimitrijedimitrijevic.com org 1234
|
||||
LEVEL ID IN LIMIT OUT LIMIT
|
||||
A me@dimitrijedimitrijevic.com 12 500 3 200
|
||||
D dimitrijedimitrijevic.com 40 1000 9 400
|
||||
O 1234 120 5000 30 1000
|
||||
```
|
||||
`managed: false` prints a `domain is not in rspamd_domains_db (not counted)` note after the table;
|
||||
`exempt: true` prints `address is exempt from counting`.
|
||||
- Exit codes: `0` success, `1` usage error / bad date / service unreachable / non-200 (message on stderr).
|
||||
|
||||
## Counting transaction log (`internal/txlog`)
|
||||
|
||||
A second logger, separate from the slog application log, that writes the Lua plugin's step-by-step lines for
|
||||
**every** counting transaction (the Lua `infox` lines from `migadu_counters.lua:217/235/266/268/271/283`),
|
||||
so existing grep habits keep working. rspamd's task id `<ad8525>` is replaced by `<trace_id>` (shared with
|
||||
mizu/mailqueuer logs via `X-Trace-Id`), and the route is added. `breach=` becomes `reached=` because the
|
||||
meaning changed (Lua: already over → rejected; here: this increment crossed the limit).
|
||||
|
||||
Text format (default), timestamps UTC with milliseconds, one block per request:
|
||||
```
|
||||
2026-10-06 01:12:06.412 <ad8525> out; request auth_user=noah@wearemiraclesnow.com recipient=y@example.org mail_from=noah@wearemiraclesnow.com
|
||||
2026-10-06 01:12:06.412 <ad8525> out; local domain: wearemiraclesnow.com account_id=116681
|
||||
2026-10-06 01:12:06.412 <ad8525> out; domain: wearemiraclesnow.com limits_map={A/i=500, A/o=100, D/i=10000, D/o=2200, O/i=10000, O/o=2200}
|
||||
2026-10-06 01:12:06.412 <ad8525> out; determined limit (level=[A] flow=[o] key=[cnt/A/noah@wearemiraclesnow.com/d/2026-10-06/o]) => 100
|
||||
2026-10-06 01:12:06.412 <ad8525> out; determined limit (level=[D] flow=[o] key=[cnt/D/wearemiraclesnow.com/d/2026-10-06/o]) => 2200
|
||||
2026-10-06 01:12:06.412 <ad8525> out; determined limit (level=[O] flow=[o] key=[cnt/O/116681/d/2026-10-06/o]) => 2200
|
||||
2026-10-06 01:12:06.412 <ad8525> out; recipient domain not in domains db: example.org (not counted)
|
||||
2026-10-06 01:12:06.414 <ad8525> out; tracked address counters for noah@wearemiraclesnow.com => count=12 limit=100 reached=false
|
||||
2026-10-06 01:12:06.414 <ad8525> out; tracked domain counters for wearemiraclesnow.com => count=1830 limit=2200 reached=false
|
||||
2026-10-06 01:12:06.414 <ad8525> out; tracked account counters for 116681 => count=2200 limit=2200 reached=true
|
||||
2026-10-06 01:12:06.414 <ad8525> out; limit reached (level=[O] id=[116681] flow=[o] by=[noah@wearemiraclesnow.com]) => notifying org_url
|
||||
2026-10-06 01:12:06.903 <ad8525> out; sent limit exceeded notification to url "https://admin.migadu.com/api/limits?level=O&id=116681&flow=o&period=d&by=noah%40wearemiraclesnow.com&count=2200"
|
||||
```
|
||||
- A local recipient produces the same `local domain / limits_map / determined limit / tracked …` block for
|
||||
`flow=[i]` under the recipient's domain row. `/in` requests log the recipient block only.
|
||||
- `determined limit … => unlimited` when no limit is configured. Skips are logged as one line:
|
||||
`skipped: exempt address <addr>`, `skipped: junk`, `recipient domain not in domains db: <d> (not counted)`,
|
||||
`sender domain not in domains db: <d> (not counted)`.
|
||||
- Notification outcome lines come from the background sender, tagged with the same `<trace_id>`:
|
||||
`sent limit exceeded notification to url "…"` or
|
||||
`failed limit exceeded notification to url "…" after 5 attempts: <err>`.
|
||||
- Redis failure: `redis error: <err> (nothing counted)`.
|
||||
- `format = "json"` switches the same lines to one JSON object per line (`ts, trace_id, route, event,
|
||||
level, flow, key, id, count, limit, reached, domain, org_id, limits, url, error`) for later ingestion.
|
||||
- The "tracked" lines are written after the pipeline EXEC (counts are the returned values); the earlier lines are
|
||||
written before the Redis call, so a Redis failure still leaves the lookup trail in the log.
|
||||
- Writer: the same `LogWriter` type as the app log (mutex, unbuffered for `tail -f`, `Reopen` on SIGHUP).
|
||||
Config: `[logging] counting_output` (default: same destination as `output`; `"off"` disables),
|
||||
`counting_format` (`text` | `json`). The per-request details live only here; the app log keeps startup,
|
||||
reloads, warnings and errors, to avoid duplication.
|
||||
|
||||
## Process model: standalone daemon
|
||||
|
||||
`cmd/migadu_counters` is a long-running **foreground** process:
|
||||
- Two `http.Server`s: the API (`/in`, `/out`, `/counters`, bearer auth, `http.TimeoutHandler`) on `addr`, and the
|
||||
admin server (`/health`, `/metrics`, no auth) on `admin_addr`. Server timeouts as in rcptd:
|
||||
`read_header_timeout=5`, `write_timeout=30`, `idle_timeout=60`, `handler_timeout=4` (seconds).
|
||||
- Startup: load config → logger → open CDB (fatal if unreadable) → create Redis clients + PING the write side
|
||||
(warn, not fatal: the service must come up while Redis is briefly away) → start fsnotify watcher → listen.
|
||||
- Signals: `SIGHUP` → reopen both log files (app + counting) + reload CDB (`atomic.Pointer` swap, old file
|
||||
closed after 30 s);
|
||||
`SIGTERM`/`SIGINT` → stop accepting, drain in-flight requests (≤ 10 s), wait for background notifications
|
||||
(≤ 10 s), close Redis clients, exit 0.
|
||||
- Supervision on FreeBSD mirrors rcptd (`ansible-freebsd3/roles/rcptd/templates/rcptd.rc.j2`): an rc.d script runs
|
||||
the binary under `/usr/sbin/daemon -f -H -p <pidfile> -o <logfile>`; daemon(8) owns the pidfile, restarts on
|
||||
exit and captures stderr; `service migadu_counters reload` = `kill -HUP $(cat pidfile)` (the child's pid).
|
||||
Therefore **no `daemon`/`pid_file` options** (rcptd's config template warns that writing the same pidfile from
|
||||
the process breaks the supervisor). Logging defaults to stderr for daemon(8) `-o`; a file path is supported
|
||||
with `Reopen` on SIGHUP for newsyslog.
|
||||
- `[logging]` is rcptd's `LoggingConfig` verbatim (`output` = stderr|stdout|path, `format` = console|json,
|
||||
`level`); copy `rcptd/internal/config/logger.go` (self-contained).
|
||||
- Binaries carry `version/commit/date` via ldflags (`-version` flag); Makefile mirrors rcptd's: `build`, `test`,
|
||||
`freebsd` (CGO_ENABLED=0, `migadu_counters-freebsd-amd64`, `counterctl-freebsd-amd64`), `build-linux`.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
migadu_counters/
|
||||
├── go.mod # module migadu_counters (exists)
|
||||
├── Makefile # build / test / freebsd / build-linux, ldflags version info
|
||||
├── cmd/migadu_counters/main.go # config, logger, CDB load/reload, redis clients, HTTP + admin servers, signals
|
||||
├── cmd/counterctl/main.go # CLI: flags, config discovery, GET /counters, table/json output
|
||||
├── internal/api/types.go # JSON types shared by the server (/in, /out, /counters) and counterctl
|
||||
├── internal/api/validate.go # per-route request validators, e-mail syntax, trace_id sanitizing
|
||||
├── internal/config/config.go # TOML structs, defaults, validation (sentinel vs static, db==0, urls)
|
||||
├── internal/config/logger.go # LoggingConfig + LogWriter.Reopen (from rcptd), counting_* fields
|
||||
├── internal/txlog/txlog.go # counting transaction logger: text (Lua-like) / json, per-request block
|
||||
├── internal/domains/domains.go # CDB open/lookup, ParseRecord/ParseLimits, Limit(level, flow, local)
|
||||
├── internal/watcher/watcher.go # fsnotify reload of the CDB (rcptd table mode, trimmed)
|
||||
├── internal/store/redis.go # write/read clients, Incr pipeline, Get for /counters
|
||||
├── internal/counting/counting.go # /in, /out request → increments (level, id, flow, limit), exempt/junk/unknown
|
||||
├── internal/notify/notify.go # per-level URL, templating/escaping, background sender with retries
|
||||
├── internal/server/middleware.go # recover, method, body size, bearer token, allowed_cidrs, content-type, decode, timeout
|
||||
├── internal/server/server.go # /in, /out, /counters handlers; admin /health, /metrics
|
||||
├── internal/metrics/metrics.go # prometheus definitions
|
||||
├── config.toml.example
|
||||
└── README.md # API contract for mailqueuer, key format, counting rules, ops
|
||||
```
|
||||
|
||||
Example config (`/usr/local/etc/migadu_counters/config.toml`; discovery order `-config`, `/etc/migadu_counters/config.toml`,
|
||||
`/usr/local/etc/migadu_counters/config.toml`):
|
||||
```toml
|
||||
addr = ":5555" # was :8095 before review round 1
|
||||
admin_addr = ":9095"
|
||||
auth_token = "…" # or MIGADU_COUNTERS_AUTH_TOKEN; same value in mailqueuer's config
|
||||
allowed_cidrs = [] # optional source-IP allowlist for /in, /out, /counters, e.g. ["10.0.0.0/8"]
|
||||
domains_cdb = "/usr/local/etc/rspamd/db/rspamd_domains_db"
|
||||
exempt = ["support@migadu.com", "contact@migadu.com"]
|
||||
count_junk = false
|
||||
|
||||
read_header_timeout = 5
|
||||
write_timeout = 30
|
||||
idle_timeout = 60
|
||||
handler_timeout = 4
|
||||
|
||||
[logging]
|
||||
output = "stderr" # daemon(8) -o captures it; or a file path (reopened on SIGHUP)
|
||||
format = "console" # or "json"
|
||||
level = "info"
|
||||
counting_output = "/var/log/migadu_counters/counting.log" # default: same as output; "off" disables
|
||||
counting_format = "text" # Lua-like lines (default) or "json"
|
||||
|
||||
[redis]
|
||||
db = 0 # must stay 0: migadu_web reads message counters from DB 0
|
||||
password = "…" # or MIGADU_COUNTERS_REDIS_PASSWORD
|
||||
pool_size = 32
|
||||
timeout_ms = 500
|
||||
# Sentinel mode (production): writes → master, reads → replicas
|
||||
master_name = "mymaster"
|
||||
sentinels = ["10.0.0.10:26379", "10.0.0.11:26379", "10.0.0.12:26379"]
|
||||
sentinel_password = ""
|
||||
# Static mode (used when sentinels is empty)
|
||||
# write_addr = "127.0.0.1:6379"
|
||||
# read_addrs = ["127.0.0.1:6379"]
|
||||
|
||||
[notify]
|
||||
# One URL per level; leave empty to disable notifications for that level.
|
||||
org_url = "https://admin.migadu.com/api/limits?level=$LEVEL&id=$ID&flow=$FLOW&period=$PERIOD&by=$ADDRESS&count=$COUNT"
|
||||
address_url = "https://admin.migadu.com/api/limits?level=$LEVEL&id=$ID&flow=$FLOW&period=$PERIOD&by=$ADDRESS&count=$COUNT"
|
||||
domain_url = ""
|
||||
user = "…"
|
||||
password = "…" # or MIGADU_COUNTERS_NOTIFY_PASSWORD
|
||||
timeout_seconds = 5
|
||||
```
|
||||
|
||||
Metrics: `migadu_counters_requests_total{route,status}`, `_request_duration_seconds{route}`,
|
||||
`_increments_total{level,flow}`, `_limit_reached_total{level,flow}`, `_notifications_total{level,result}`,
|
||||
`_redis_errors_total{op}`, `_cdb_domains` (gauge), `_cdb_reloads_total{result}`.
|
||||
`/health`: 200 when the CDB is readable and the write client answers PING within 200 ms; else 503 with reason.
|
||||
|
||||
## Behaviour changes vs the Lua plugin (intentional)
|
||||
|
||||
- Separate `/in` and `/out` routes, one request per recipient, instead of rspamd's `MIGADU_FLOW` symbol.
|
||||
- Outgoing counted **per recipient** (Lua: once per message, skipped when all recipients were same-domain).
|
||||
Existing `*/o` limits therefore become stricter — a 50-recipient message costs 50; limits may need re-tuning
|
||||
before the web app's `outgoing_limit_exceeded!` is relied upon.
|
||||
- Local recipients of authenticated mail get incoming counted via `/out`.
|
||||
- Domain (`D`) level counted; notified only when `domain_url` is set.
|
||||
- Atomic increments; counters don't stop at the limit; no reject/soft-reject anywhere.
|
||||
- Notification on the exact crossing increment; re-fires after a web-app reset.
|
||||
- UTC day keys (matches the web app; Lua used server local time). Malformed CDB rows are skipped, not fatal.
|
||||
|
||||
## Deployment prerequisites (not part of this repo)
|
||||
|
||||
- An Ansible role in `ansible-freebsd3` cloned from `roles/rcptd`: user/group, `config.toml.j2`, the rc.d script
|
||||
(daemon(8) with pidfile `/var/run/migadu_counters/migadu_counters.pid`, log `/var/log/migadu_counters/…`),
|
||||
newsyslog entry, binary `migadu_counters-freebsd-amd64` + `counterctl-freebsd-amd64` in `/usr/local/bin`.
|
||||
- The CDB must exist on the host running the service (the agent delivers it to rspamd hosts today;
|
||||
stats-merger pulls the text variant) — same file, so a copy/pull job is enough.
|
||||
- Firewall: mailqueuer hosts → service port; service → Sentinel hosts :26379/:6379 and admin.migadu.com:443.
|
||||
- Secrets: bearer token shared with mailqueuer, Redis password, web app basic-auth credentials.
|
||||
|
||||
## Implementation order
|
||||
|
||||
0. Copy this plan into the repo as `plans/2026-10-13-migadu-counters.md` (requested; kept as the design record).
|
||||
1. `go.mod` deps, `Makefile`, `config.toml.example`, `internal/config` (structs, defaults, validation, logger from rcptd).
|
||||
2. `internal/domains` (+ tests with a temp CDB), `internal/watcher` (trimmed from rcptd).
|
||||
3. `internal/store` (sentinel/static clients, Incr pipeline, MGET) + miniredis tests, incl. crossing detection.
|
||||
4. `internal/counting` (request → increments, exempt/junk/unknown) + tests for the `/out` and `/in` examples.
|
||||
5. `internal/txlog` (text + json, golden tests), `internal/notify` (per-level URLs, retries) + httptest tests.
|
||||
6. `internal/api` types, `internal/server` (`/in`, `/out`, `/counters`, admin), `internal/metrics`.
|
||||
7. `cmd/migadu_counters/main.go` (wiring, signals, shutdown), `cmd/counterctl/main.go` (+ tests).
|
||||
8. `README.md` (API contract for mailqueuer, key format, counting rules, ops), manual verification below.
|
||||
|
||||
## Verification
|
||||
|
||||
1. `go vet ./... && go test -race ./...`
|
||||
- `domains`: parse/override/unlimited/malformed rows via a temp CDB (`cdb.Create`); unknown domain.
|
||||
- `counting`: the `/out` example (two calls → out +2 per level, y in +1 per level); external-only recipient;
|
||||
sender domain not in CDB (only recipient counted); exempt sender/recipient → nothing; `/in` local/unknown/junk.
|
||||
- `store` + miniredis: key format, TTL set, DB 0, crossing fires exactly once (limit 3: 3rd increment
|
||||
`reached`, 4th not; after `DEL` the next crossing fires again); concurrent increments → one crossing;
|
||||
read-side round-robin / fallback; Redis down → 503.
|
||||
- `notify`: each level hits its own URL, empty URL skips, all vars escaped (`@` → `%40`), retries then failure metric.
|
||||
- `txlog`: golden-file tests with an injected clock for the `/out` example above (local + external recipient,
|
||||
one level reached), an `/in` request, each skip reason, a Redis failure, and the `json` format; `Reopen`
|
||||
after the file is moved writes to the new path.
|
||||
- `api/validate`: table tests per field and route (missing, empty, too long, two `@`, whitespace, control
|
||||
chars, uppercase → lowercased, punycode accepted, `mail_from` empty on `/in`, `trace_id` sanitized).
|
||||
- `server/middleware`: one test per step in chain order — 405, 413 (65 KB body), 401 (missing/wrong/`Basic`),
|
||||
403 (`allowed_cidrs` miss) vs pass, 415, 400 (syntax error, two objects, unknown field *accepted*), 503
|
||||
(handler timeout with a blocked Redis), 500 (panic recovered, daemon still serving); each rejection leaves
|
||||
Redis untouched (miniredis key count unchanged) and increments `requests_total{status}`.
|
||||
- `server`: 401, 400, 200 with `counted` payload; `/counters` with address/domain, default date = today UTC,
|
||||
explicit `date`, future date → 400, malformed date → 400, unmanaged domain → `managed:false` + null limits;
|
||||
notify targets are per-level `httptest.Server`s.
|
||||
- `counterctl`: date parsing/validation (valid ISO, malformed, future, > 31 days warning), exactly-one-of
|
||||
`--address`/`--domain`, `addr` → URL normalization, table rendering against a fixed API response
|
||||
(`httptest.Server`), `--json` passthrough, exit codes.
|
||||
2. Manual: local Redis in static mode, CDB built from a sample line (`cdb -m -c`, or the test helper),
|
||||
`go run ./cmd/migadu_counters -config config.toml.example` with `org_url`/`address_url` pointed at a local
|
||||
listener. `curl -XPOST localhost:8095/out -H 'Authorization: Bearer …' -d '{…}'` repeatedly; check
|
||||
`redis-cli -n 0 KEYS 'cnt/*'`, `GET /counters?address=…`, exactly one notification per level on crossing,
|
||||
`redis-cli DEL` the A key → crossing notifies again; edit the CDB + `kill -HUP` → limits reload.
|
||||
`go run ./cmd/counterctl --config config.toml.example --address x@example.com` shows the same numbers as
|
||||
`redis-cli`; `--date <yesterday>` shows zeros/older data; `--date 2099-01-01` is rejected.
|
||||
`tail -f counting.log` while sending the curl requests shows the full block per request, and
|
||||
`grep '<trace_id>' counting.log` returns one transaction. Daemon behaviour: `kill -TERM` exits 0 after
|
||||
draining; with `logging.output`/`counting_output` set to files, `kill -HUP` reopens them (rotate with `mv`
|
||||
first and confirm new lines land in the new files); `make freebsd` produces the two static binaries.
|
||||
Optionally a local Sentinel (1 master + 1 replica) to confirm reads hit the replica.
|
||||
@@ -0,0 +1,113 @@
|
||||
# Code review — 2026-10-14
|
||||
|
||||
Review of the first implementation of migadu_counters (see `../plans/2026-10-13-migadu-counters.md`).
|
||||
Nothing in the code was changed by the review.
|
||||
|
||||
## How it was done
|
||||
|
||||
- Two independent reviewers (agents) each read the whole repository. Neither saw the other's results.
|
||||
- **A** — correctness and runtime behaviour: [`reviewer-a-correctness.md`](reviewer-a-correctness.md)
|
||||
- **B** — security, contracts and operability: [`reviewer-b-security-contracts.md`](reviewer-b-security-contracts.md)
|
||||
- Each worked in its own scratch copy and proved findings with tests: [`_proof_tests/`](_proof_tests/README.md).
|
||||
- The coordinating agent then verified every finding (see "Verification" below), merged duplicates and ranked them.
|
||||
18 distinct issues remain; 4 of them were found by both reviewers.
|
||||
|
||||
## Findings
|
||||
|
||||
Who found it: A / B / both, with the reviewer's own IDs for traceability.
|
||||
|
||||
### High
|
||||
|
||||
**1. If mailqueuer disconnects or times out before the reply, the count is lost** — A (F1)
|
||||
- `internal/server/server.go:156` runs the Redis write with the request context. When the client hangs up, Go cancels
|
||||
the context and go-redis aborts before sending MULTI.
|
||||
- Re-run during verification: 20 of 20 delayed requests and 300 of 300 immediate ones were not counted. The most
|
||||
dangerous finding, because fire-and-forget clients often close early.
|
||||
- Fix: run the counting under `context.WithoutCancel(r.Context())` with its own time budget.
|
||||
|
||||
**2. A bad value in a field that isn't counted rejects the whole request** — both (B F1, A F3)
|
||||
- `internal/api/validate.go` strictly validates `mail_from` although it only feeds the exempt check; a 400 there means
|
||||
the recipient is never counted.
|
||||
- Senders trigger it easily: mizu's SMTP parser turns `"john smith"@x.com` into `john smith@x.com`, mizu runs with
|
||||
SMTPUTF8 (`info@bücher.de`), address literals and trailing dots pass, and mailqueuer forwards `X-Mail-From` unchecked.
|
||||
- On `/out`, an unusual external recipient (`x@[192.0.2.1]`, `x@host_name.example.com`) gives 400 and the sender's
|
||||
outgoing count is lost too.
|
||||
- Fix: never reject because of `mail_from`; validate sender and recipient sides separately and count whichever is
|
||||
valid; convert UTF-8 domains to punycode.
|
||||
|
||||
### Medium
|
||||
|
||||
**3. Support/token logins aren't counted** — A (F2)
|
||||
- mizu puts the raw SASL login into `X-Auth-User` (`pkg/smtp/auth_session.go:160` → `poster.go:186`), so master logins
|
||||
arrive as `user@dom@SUFFIX` and fail validation with 400.
|
||||
- Fix: strip the second `@SUFFIX` and count the base address.
|
||||
|
||||
**4. Redis failures with an unclear outcome lose the limit notification** — both (B F3, A F4)
|
||||
- If Redis applies MULTI/EXEC but the reply times out, or one command inside EXEC fails (a key holding a non-number)
|
||||
while the others succeed, the service answers 503 and logs "nothing counted" — but the counters moved.
|
||||
- If that increment reached the limit, no later increment can reach it again: no notification that day.
|
||||
- Fix: use each command's own result; after a timeout re-read the keys from the master; log "outcome unknown".
|
||||
|
||||
**5. Huge CDB records are re-parsed on every request** — A (F5)
|
||||
- `boxmailhere.com` (193 KB, 9,798 overrides): about 0.9 ms and 1.4 MB per lookup, 2.2 ms and 1.6 MB per request
|
||||
(re-measured).
|
||||
- Fix: cache parsed records per loaded CDB.
|
||||
|
||||
**6. An empty `auth_token` leaves the API open with only a warning** — B (F4)
|
||||
- A typo such as `auth-token =` starts the daemon unauthenticated; anyone who can reach it can inflate any customer's
|
||||
counters, which triggers the web app's limit actions.
|
||||
- Fix: make an empty token fatal unless explicitly opted out; make unknown config keys errors.
|
||||
|
||||
**7. Plus-addressed recipients get their own counters** — B (F2); A notes the Lua plugin did the same
|
||||
- Mail to `b+1@…` creates `cnt/A/b+1@…`; the mailbox's own counter (the one the web app charts) never moves. If such
|
||||
a counter reaches its limit, the web app answers "mailbox not found".
|
||||
- rcptd strips the `+tag`, so stripping it here too seems right. **Needs a product decision.**
|
||||
|
||||
**8. One malformed CDB row breaks counting for anything touching that domain** — both (B F5, A F6)
|
||||
- A non-numeric org id gives 503 and drops the other side's valid count; the plan says such rows are skipped.
|
||||
- Fix: treat the row as not hosted and log a warning.
|
||||
|
||||
### Low
|
||||
|
||||
9. **Notification retries can send customers duplicate emails** — B (F11), A (F9). Resolved disagreement: the
|
||||
organisation handler has a once-per-day guard (`migadu_web core/accounts.ex:1928-1956`), the mailbox handler
|
||||
(`core/email.ex:3572-3584`) emails on every call, so a retry after a slow-but-successful call means a second
|
||||
email. Also: shutdown cuts off pending retries, retries keep their concurrency slot while waiting, and the
|
||||
"after 5 attempts" log line can be wrong.
|
||||
10. **A SIGHUP during startup kills the daemon** — A (F8). The signal handler is installed only after the Redis
|
||||
ping, which can take up to 2 s.
|
||||
11. **Rejected requests leave no trace** — both (B F9, A F2 note). Nothing in the counting log, no trace_id in the app
|
||||
log; together with #2 the lost counts are invisible.
|
||||
12. **Override keys aren't lowercased** — A (F7). `Info@/o=5` would be ignored; none in today's data.
|
||||
13. **Credentials in the notify URL are written to the log** — B (F7); plain `http://` with basic auth isn't warned about.
|
||||
14. **Error responses expose internals** — B (F10): internal Redis addresses and paths; `admin_addr` defaults to all
|
||||
interfaces without auth.
|
||||
15. **Sentinel mode starts without a Redis password** — B (F12): every write then fails; only `/health` and metrics show it.
|
||||
16. **counterctl warns one day too early** — B (F6): a date exactly 31 days back is reported as expired.
|
||||
17. **A recovered panic isn't counted in `requests_total`** — B (F8).
|
||||
18. **Work keeps running after a handler timeout** — A (F10, not reproduced): not tracked by shutdown, a late panic is
|
||||
silently lost. Largely covered by the fix for #1.
|
||||
|
||||
**Test gaps** — B (F13): the suite misses most of the above; `TestValidation_JSONAndFields` asserts the 400 behind #2
|
||||
as correct; watcher tests can occasionally pass on a stale signal.
|
||||
|
||||
## Verification
|
||||
|
||||
Done by the coordinating agent after both reviews:
|
||||
|
||||
- Both scratch copies were checked to have source files identical to the repo, so the proof tests ran against the
|
||||
reviewed code. B's tests fail as described; A's demonstration tests reproduce their evidence (#1 was worse on
|
||||
re-run: 0 counted instead of 21 of 300).
|
||||
- External facts checked in the sibling repos:
|
||||
- mailqueuer reads `X-Mail-From` without validating it (`internal/handler/ingest.go`).
|
||||
- mizu's go-smtp fork unquotes quoted local parts and copies the domain verbatim (`parse.go` `parseMailbox`,
|
||||
`parseLocalPart`); mizu sets `EnableSMTPUTF8 = true` (`cmd/mizu-server/main.go:1363`).
|
||||
- mizu stores the raw SASL username as the authenticated user (`pkg/smtp/auth_session.go:160`) and sends it as
|
||||
`X-Auth-User`; `user@domain@SUFFIX` logins are documented in mizu's `CLAUDE.md`.
|
||||
- Web app: org-level limit handlers are guarded once per day, the mailbox-level ones are not (see #9).
|
||||
- Reviewer disagreements: #7 (A: Lua parity, B: defect) is left as a product decision; #9 settled as above.
|
||||
|
||||
## Suggested order
|
||||
|
||||
Fix #1–4, #6, #8 and #10 first (small, they protect the counts), then #5 and #7 (#7 after the decision), then the
|
||||
low items. Turn each proof test into a regression test as its finding is fixed.
|
||||
@@ -0,0 +1,71 @@
|
||||
# Proof tests from the 2026-10-14 code review
|
||||
|
||||
These are the tests the two reviewers wrote to demonstrate their findings. They live under a
|
||||
directory whose name starts with `_`, so `go build ./...`, `go test ./...` and `go vet ./...`
|
||||
ignore them. They are evidence, not part of the suite. When a finding is fixed, turn its proof
|
||||
test into a real regression test in the package it belongs to.
|
||||
|
||||
| Dir | Reviewer | Style |
|
||||
|---|---|---|
|
||||
| `a/` | A — correctness and runtime | Demonstrations: the tests **pass** and print the evidence with `t.Log` (run with `-v`). |
|
||||
| `b/` | B — security, contracts, ops | Assertions: the tests **fail** while the defect exists (a few `PASS`es only log evidence). |
|
||||
|
||||
## Running
|
||||
|
||||
Copy the files into a scratch copy of the repo, so the real packages stay clean:
|
||||
|
||||
```sh
|
||||
rm -rf /tmp/mc_proof && cp -R /Users/dice/Work/migadu_counters /tmp/mc_proof && cd /tmp/mc_proof
|
||||
cp -R review/_proof_tests/a/internal/. internal/ # reviewer A
|
||||
cp -R review/_proof_tests/b/internal/. internal/ # reviewer B
|
||||
cp -R review/_proof_tests/b/cmd/. cmd/
|
||||
go test -count=1 -v -run 'TestReview' ./internal/... ./cmd/...
|
||||
go test -run '^$' -bench Review -benchmem ./internal/txlog/ # A's CDB parsing benchmarks
|
||||
```
|
||||
|
||||
Requirements and notes:
|
||||
|
||||
- `a/internal/store/review_test.go` (`TestReview_TimeoutAfterWriteStillCounts`) needs a real Redis on
|
||||
port 16401: `redis-server --port 16401 --save '' --appendonly no`. The test uses `DEBUG SLEEP` to make the
|
||||
reply arrive after the client timeout. It fails to connect if no Redis is running.
|
||||
- `b/bad.cdb` is a three-row fixture with one malformed row (`bad.com x12|A/i=5`). It is used by
|
||||
`b/internal/server/review_b4_test.go`, whose path was updated to `../../review/_proof_tests/b/bad.cdb`.
|
||||
- `a/run/` holds the config and the app log from reviewer A's manual reproduction of "SIGHUP during startup
|
||||
kills the daemon" (merged item #10). The paths in `config.toml` point at the reviewer's former scratch copy,
|
||||
so edit them before reuse. The log stops after `domains db loaded`, which is the evidence.
|
||||
|
||||
## Map to the merged report (`../README.md`)
|
||||
|
||||
| Merged # | Proof |
|
||||
|---|---|
|
||||
| 1 | `a/internal/server/review_test.go` `TestReview_ClientDisconnectLosesCount`, `TestReview_ClientDisconnectNoDelay` |
|
||||
| 2 | `b/internal/server/review_b_test.go` `TestReviewB_InMailFromDropsRecipientCount`, `TestReviewB_OutRecipientDropsSenderCount`; `a/…` `TestReview_MasterLoginAndOddSenders` |
|
||||
| 3 | `a/internal/server/review_test.go` `TestReview_MasterLoginAndOddSenders` (first case) |
|
||||
| 4 | `b/internal/server/review_b2_test.go` `TestReviewB_LostCrossingOnReplyTimeout`; `a/internal/server/review_test.go` `TestReview_PartialExecFailure`; `a/internal/store/review_test.go` (real Redis) |
|
||||
| 5 | `a/internal/txlog/review_bench_test.go` |
|
||||
| 6 | `b/internal/config/review_b_test.go` `TestReviewB_MisconfigStarts`; `b/…/review_b_test.go` `TestReviewB_EmptyTokenOpen` |
|
||||
| 7 | `b/internal/server/review_b_test.go` `TestReviewB_PlusTagSeparateCounters` |
|
||||
| 8 | `b/internal/server/review_b4_test.go`; `a/internal/counting/review_test.go` |
|
||||
| 10 | `a/run/` (manual reproduction) |
|
||||
| 11 | `b/internal/server/review_b_test.go` `TestReviewB_RejectLogHasNoTraceID` |
|
||||
| 13 | `b/internal/server/review_b3_test.go` |
|
||||
| 14 | `b/internal/server/review_b_test.go` `TestReviewB_ErrorLeak` |
|
||||
| 15 | `b/internal/config/review_b_test.go` (case "sentinel without password") |
|
||||
| 16 | `b/cmd/counterctl/review_b_test.go` `TestReviewB_31DayWarningOffByOne` |
|
||||
| 17 | `b/internal/server/review_b_test.go` `TestReviewB_PanicNotInMetrics` |
|
||||
|
||||
## Fixed in round 1 (2026-10-14) — regression tests now in the suite
|
||||
|
||||
| Merged # | Regression test |
|
||||
|---|---|
|
||||
| 1 | `internal/server` `TestClientDisconnectStillCounts`, `TestAcceptedBeforeCounting`, `TestQueueFull`, `TestDrainCountsEverything`; `internal/counting/queue_test.go` |
|
||||
| 2 | `internal/server` `TestIn_OddMailFromStillCountsRecipient`, `TestOut_OddRecipientStillCountsSender`; `internal/api` `TestValidateOut`, `TestValidateIn`, `TestNormalizeEmail` |
|
||||
| 3 | `internal/server` `TestOut_LoginSuffixCountsBaseAddress`; `internal/api` `TestNormalizeAuthUser` |
|
||||
| 4 | `internal/server` `TestPartialExecFailureStillNotifies`, `TestLostReplyStillNotifies`; `internal/counting` `TestCounter_PartialExecFailure`, `TestCounter_LostReplyReReadsMaster`; `internal/store` `TestIncr_PartialFailure`, `TestOutcomeUnknown` |
|
||||
| 5 | `internal/domains` `TestLookup_CachesParsedRecord`, `TestLookup_CacheBounded`, `BenchmarkLookupSmall`/`BenchmarkLookupHuge` |
|
||||
| 7 | `internal/server` `TestIn_PlusTagCountsMailbox`; `internal/counting` `TestPlan_PlusTagCountsMailbox`; `internal/domains` `TestMailbox` (decision 2026-10-15: strip the tag, as rcptd does) |
|
||||
| 8 | `internal/server` `TestMalformedRowCountsOtherSide`; `internal/counting` `TestPlan_MalformedRowIsNotHosted`; `internal/domains` `TestLookup_MalformedRow`; `internal/txlog` `TestTrail_MalformedRow` |
|
||||
| 6 | `internal/config` `TestValidate_Errors` (auth_token / unknown keys), `TestLoad_TokenFromEnvOnly`, `TestLoad_Defaults` (`:5555`); `cmd/migadu_counters` `TestRun_FlagsAndExitCodes` |
|
||||
|
||||
The proof files stay here as the review record; they no longer compile against the current API
|
||||
(`CountResponse`, `store.Incr`) and are not meant to.
|
||||
@@ -0,0 +1,34 @@
|
||||
package counting
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/colinmarc/cdb"
|
||||
|
||||
"migadu_counters/internal/domains"
|
||||
)
|
||||
|
||||
func TestReview_MalformedRowFailsWholeRequest(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "d.cdb")
|
||||
w, err := cdb.Create(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
w.Put([]byte("good.example"), []byte("7|A/o=5,A/i=5"))
|
||||
w.Put([]byte("bad.example"), []byte("|A/i=5")) // org id missing (malformed row)
|
||||
if _, err := w.Freeze(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
db, err := domains.Open(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
p := &Planner{Lookup: db, Now: time.Now}
|
||||
plan, err := p.PlanOut("t", "x@good.example", "x@good.example", "y@bad.example")
|
||||
t.Logf("PlanOut(good sender -> malformed-row recipient): plan=%v err=%v", plan, err)
|
||||
plan, err = p.PlanIn("t", "s@gmail.com", "y@bad.example", false)
|
||||
t.Logf("PlanIn(malformed-row recipient): plan=%v err=%v", plan, err)
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"migadu_counters/internal/counting"
|
||||
"migadu_counters/internal/domains"
|
||||
)
|
||||
|
||||
func rawPostAndClose(t *testing.T, addr, path, body string) {
|
||||
c, err := net.Dial("tcp", addr)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
req := fmt.Sprintf("POST %s HTTP/1.1\r\nHost: x\r\nAuthorization: Bearer %s\r\nContent-Type: application/json\r\nContent-Length: %d\r\n\r\n%s", path, token, len(body), body)
|
||||
if _, err := c.Write([]byte(req)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
c.Close() // fire-and-forget client that does not wait for the reply
|
||||
}
|
||||
|
||||
func TestReview_ClientDisconnectLosesCount(t *testing.T) {
|
||||
h := newHarness(t, options{})
|
||||
real := *h.lookup.Load()
|
||||
slow := counting.LookupFunc(func(d string) (*domains.Domain, error) {
|
||||
time.Sleep(50 * time.Millisecond) // any latency before the Redis call
|
||||
return real(d)
|
||||
})
|
||||
h.lookup.Store(&slow)
|
||||
addr := strings.TrimPrefix(h.api.URL, "http://")
|
||||
const n = 20
|
||||
for i := 0; i < n; i++ {
|
||||
rawPostAndClose(t, addr, "/in", `{"trace_id":"t","mail_from":"s@gmail.com","recipient":"a@yagolivas.com"}`)
|
||||
}
|
||||
time.Sleep(1500 * time.Millisecond)
|
||||
v, _ := h.write.Get("cnt/A/a@yagolivas.com/d/" + day + "/i")
|
||||
t.Logf("sent %d /in requests, counter = %q", n, v)
|
||||
t.Logf("txlog tail:\n%s", lastLines(h.tx.String(), 3))
|
||||
}
|
||||
|
||||
func TestReview_ClientDisconnectNoDelay(t *testing.T) {
|
||||
h := newHarness(t, options{})
|
||||
addr := strings.TrimPrefix(h.api.URL, "http://")
|
||||
const n = 300
|
||||
for i := 0; i < n; i++ {
|
||||
rawPostAndClose(t, addr, "/in", `{"trace_id":"t","mail_from":"s@gmail.com","recipient":"b@yagolivas.com"}`)
|
||||
}
|
||||
time.Sleep(2 * time.Second)
|
||||
v, _ := h.write.Get("cnt/A/b@yagolivas.com/d/" + day + "/i")
|
||||
t.Logf("sent %d /in requests without delay, counter = %q", n, v)
|
||||
}
|
||||
|
||||
func lastLines(s string, n int) string {
|
||||
l := strings.Split(strings.TrimRight(s, "\n"), "\n")
|
||||
if len(l) > n {
|
||||
l = l[len(l)-n:]
|
||||
}
|
||||
return strings.Join(l, "\n")
|
||||
}
|
||||
|
||||
func TestReview_MasterLoginAndOddSenders(t *testing.T) {
|
||||
h := newHarness(t, options{})
|
||||
cases := []struct{ path, body string }{
|
||||
{"/out", `{"trace_id":"m1","auth_user":"x@yagolivas.com@SUPPORTTOKEN","mail_from":"x@yagolivas.com","recipient":"y@yagolivas.com"}`},
|
||||
{"/in", `{"trace_id":"i1","mail_from":"a b@spammer.example","recipient":"y@yagolivas.com"}`},
|
||||
{"/in", `{"trace_id":"i2","mail_from":"u@[192.0.2.1]","recipient":"y@yagolivas.com"}`},
|
||||
{"/in", `{"trace_id":"i3","mail_from":"u@münchen.de","recipient":"y@yagolivas.com"}`},
|
||||
{"/in", `{"trace_id":"i4","mail_from":"u@example.com.","recipient":"y@yagolivas.com"}`},
|
||||
{"/in", `{"trace_id":"i5","mail_from":"a@b@example.com","recipient":"y@yagolivas.com"}`},
|
||||
}
|
||||
for _, c := range cases {
|
||||
st, body := h.post(c.path, c.body)
|
||||
t.Logf("%s %s -> %d %s", c.path, c.body, st, strings.TrimSpace(body))
|
||||
}
|
||||
t.Logf("keys in redis: %v", h.write.Keys())
|
||||
}
|
||||
|
||||
func TestReview_PartialExecFailure(t *testing.T) {
|
||||
h := newHarness(t, options{})
|
||||
// A counter key holding a non-integer (or wrong type) value.
|
||||
h.write.Set("cnt/O/1049/d/"+day+"/o", "garbage")
|
||||
// A/o limit is 20 for yagolivas.com; preset A/o to 19 so this request crosses it.
|
||||
h.write.Set("cnt/A/x@yagolivas.com/d/"+day+"/o", "19")
|
||||
st, body := h.post("/out", `{"trace_id":"p1","auth_user":"x@yagolivas.com","recipient":"z@gmail.com"}`)
|
||||
t.Logf("status %d body %s", st, strings.TrimSpace(body))
|
||||
a, _ := h.write.Get("cnt/A/x@yagolivas.com/d/" + day + "/o")
|
||||
d, _ := h.write.Get("cnt/D/yagolivas.com/d/" + day + "/o")
|
||||
t.Logf("after: A/o=%s (limit 20 crossed) D/o=%s", a, d)
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
t.Logf("address notifications sent: %d", h.addrURL.count())
|
||||
t.Logf("txlog:\n%s", h.tx.String())
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/redis/go-redis/v9"
|
||||
|
||||
"migadu_counters/internal/config"
|
||||
)
|
||||
|
||||
func TestReview_TimeoutAfterWriteStillCounts(t *testing.T) {
|
||||
st, err := New(config.RedisConfig{WriteAddr: "127.0.0.1:16401", PoolSize: 4, TimeoutMS: 500})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer st.Close()
|
||||
admin := redis.NewClient(&redis.Options{Addr: "127.0.0.1:16401"})
|
||||
defer admin.Close()
|
||||
key := "cnt/A/review@example.com/d/2026-10-14/o"
|
||||
admin.Del(context.Background(), key)
|
||||
admin.Set(context.Background(), key, "99", 0) // limit 100: next increment crosses
|
||||
// warm the pool
|
||||
if err := st.Ping(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
go admin.Do(context.Background(), "DEBUG", "SLEEP", "1") // server stall > timeout_ms
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
_, err = st.Incr(context.Background(), []string{key}, 1)
|
||||
t.Logf("Incr error: %v", err)
|
||||
time.Sleep(1500 * time.Millisecond)
|
||||
v, _ := admin.Get(context.Background(), key).Result()
|
||||
t.Logf("key after the 'failed' Incr: %s", v)
|
||||
admin.Del(context.Background(), key)
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
package txlog
|
||||
|
||||
import (
|
||||
"io"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"migadu_counters/internal/counting"
|
||||
"migadu_counters/internal/domains"
|
||||
)
|
||||
|
||||
const cdbPath = "../../test_files/rspamd_domains_db.cdb"
|
||||
|
||||
func BenchmarkReview_Open(b *testing.B) {
|
||||
for i := 0; i < b.N; i++ {
|
||||
db, err := domains.Open(cdbPath)
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
db.Close()
|
||||
}
|
||||
}
|
||||
|
||||
func benchLookup(b *testing.B, dom string) {
|
||||
db, err := domains.Open(cdbPath)
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
b.ReportAllocs()
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
if _, err := db.Lookup(dom); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func BenchmarkReview_LookupSmall(b *testing.B) { benchLookup(b, "yagolivas.com") }
|
||||
func BenchmarkReview_LookupHuge(b *testing.B) { benchLookup(b, "boxmailhere.com") }
|
||||
|
||||
func BenchmarkReview_InRequestHuge(b *testing.B) {
|
||||
db, err := domains.Open(cdbPath)
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
p := &counting.Planner{Lookup: db, Now: time.Now}
|
||||
l := New(io.Discard, "text", nil)
|
||||
b.ReportAllocs()
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
plan, err := p.PlanIn("t", "s@gmail.com", "someone@boxmailhere.com", false)
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
out := &counting.Outcome{Plan: plan}
|
||||
for _, inc := range plan.Increments {
|
||||
out.Results = append(out.Results, counting.Result{Increment: inc, Count: 1})
|
||||
}
|
||||
l.Outcome(out, nil)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
time=2026-10-14T08:24:13.995+02:00 level=INFO msg="domains db loaded" path=/tmp/review_a/test_files/rspamd_domains_db.cdb domains=93549
|
||||
@@ -0,0 +1,13 @@
|
||||
addr = "127.0.0.1:18095"
|
||||
admin_addr = "127.0.0.1:19095"
|
||||
auth_token = "tok"
|
||||
domains_cdb = "/tmp/review_a/test_files/rspamd_domains_db.cdb"
|
||||
[logging]
|
||||
output = "/tmp/review_a/run/app.log"
|
||||
counting_output = "/tmp/review_a/run/counting.log"
|
||||
[redis]
|
||||
write_addr = "127.0.0.1:16401"
|
||||
timeout_ms = 500
|
||||
[notify]
|
||||
org_url = ""
|
||||
address_url = ""
|
||||
Binary file not shown.
@@ -0,0 +1,60 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/alicebob/miniredis/v2"
|
||||
|
||||
"migadu_counters/internal/config"
|
||||
"migadu_counters/internal/domains"
|
||||
"migadu_counters/internal/server"
|
||||
"migadu_counters/internal/store"
|
||||
)
|
||||
|
||||
// Date exactly 31 days back: its keys are still alive (TTL refreshed by the
|
||||
// day's last increment, so they live until ~23:59 today) but counterctl claims
|
||||
// it is "more than 31 days ago" and "will read as 0".
|
||||
func TestReviewB_31DayWarningOffByOne(t *testing.T) {
|
||||
f := &fakeAPI{body: sampleResponse()}
|
||||
s := httptest.NewServer(f)
|
||||
defer s.Close()
|
||||
cfg := writeConfig(t, "auth_token = \"t\"\n")
|
||||
code, _, errOut := runCLI(t, noEnv, "--config", cfg, "--url", s.URL, "--address", "me@example.com", "--date", "2026-09-12")
|
||||
t.Logf("now=%s --date 2026-09-12 (31 days back) exit=%d stderr=%q", now, code, errOut)
|
||||
lastIncr := time.Date(2026, 9, 12, 23, 59, 0, 0, time.UTC)
|
||||
t.Logf("a key last incremented at %s expires at %s (still alive now)", lastIncr, lastIncr.Add(store.TTL))
|
||||
if strings.Contains(errOut, "will read as 0") {
|
||||
t.Errorf("warning fires for a date whose keys still exist")
|
||||
}
|
||||
}
|
||||
|
||||
// --url with a trailing slash, against the real API handler.
|
||||
func TestReviewB_URLTrailingSlash(t *testing.T) {
|
||||
mr := miniredis.RunT(t)
|
||||
st, err := store.New(config.RedisConfig{WriteAddr: mr.Addr(), PoolSize: 2, TimeoutMS: 1000})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
db, err := domains.Open("../../test_files/rspamd_domains_db.cdb")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv := server.New(server.Deps{AuthToken: "t", Store: st, Domains: func() *domains.DB { return db }, Now: func() time.Time { return now }})
|
||||
s := httptest.NewServer(srv.Handler())
|
||||
defer s.Close()
|
||||
cfg := writeConfig(t, "auth_token = \"t\"\n")
|
||||
for _, u := range []string{s.URL, s.URL + "/", s.URL + "/api"} {
|
||||
code, out, errOut := runCLI(t, noEnv, "--config", cfg, "--url", u, "--address", "a@yagolivas.com")
|
||||
t.Logf("--url %q -> exit %d stdout=%q stderr=%q", u, code, firstLine(out), strings.TrimSpace(errOut))
|
||||
}
|
||||
}
|
||||
|
||||
func firstLine(s string) string {
|
||||
if i := strings.IndexByte(s, '\n'); i >= 0 {
|
||||
return s[:i]
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestReviewB_MisconfigStarts(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
for name, body := range map[string]string{
|
||||
"typo token, open API": `auth-token = "s3cret"
|
||||
domains_cdb = "x.cdb"
|
||||
allowed_cidr = ["10.0.0.0/8"]
|
||||
[redis]
|
||||
write_addr = "127.0.0.1:6379"`,
|
||||
"sentinel without password": `auth_token = "t"
|
||||
domains_cdb = "x.cdb"
|
||||
[redis]
|
||||
master_name = "mymaster"
|
||||
sentinels = ["10.0.0.10:26379"]`,
|
||||
"notify creds in URL / http basic": `auth_token = "t"
|
||||
domains_cdb = "x.cdb"
|
||||
[redis]
|
||||
write_addr = "127.0.0.1:6379"
|
||||
[notify]
|
||||
org_url = "http://admin:hunter2@admin.migadu.com/api/limits?level=$LEVEL&id=$ID&flow=$FLOW&period=$PERIOD&by=$ADDRESS&count=$COUNT"
|
||||
user = "u"
|
||||
password = "p"`,
|
||||
"public admin": `auth_token = "t"
|
||||
domains_cdb = "x.cdb"
|
||||
admin_addr = "0.0.0.0:9095"
|
||||
[redis]
|
||||
write_addr = "127.0.0.1:6379"`,
|
||||
} {
|
||||
p := filepath.Join(dir, "c.toml")
|
||||
os.WriteFile(p, []byte(body), 0o600)
|
||||
cfg, err := Load(p)
|
||||
if err != nil {
|
||||
t.Logf("%s: Load error: %v", name, err)
|
||||
continue
|
||||
}
|
||||
t.Logf("%s: Load OK; AuthToken=%q prefixes=%v redis.password=%q warnings=%q", name, cfg.AuthToken, cfg.AllowedPrefixes(), cfg.Redis.Password, cfg.Warnings())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"io"
|
||||
"net"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/alicebob/miniredis/v2"
|
||||
|
||||
"migadu_counters/internal/store"
|
||||
)
|
||||
|
||||
// delayProxy forwards to backend; replies are held back by *delay.
|
||||
func delayProxy(t *testing.T, backend string, delay *atomic.Int64) string {
|
||||
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { ln.Close() })
|
||||
go func() {
|
||||
for {
|
||||
c, err := ln.Accept()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
b, err := net.Dial("tcp", backend)
|
||||
if err != nil {
|
||||
c.Close()
|
||||
continue
|
||||
}
|
||||
go func() { io.Copy(b, c); b.Close() }()
|
||||
go func() {
|
||||
buf := make([]byte, 32*1024)
|
||||
for {
|
||||
n, err := b.Read(buf)
|
||||
if n > 0 {
|
||||
time.Sleep(time.Duration(delay.Load()))
|
||||
if _, werr := c.Write(buf[:n]); werr != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
c.Close()
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
}()
|
||||
return ln.Addr().String()
|
||||
}
|
||||
|
||||
// F: the EXEC reply is lost (read timeout) after Redis applied the increments:
|
||||
// the service answers 503 "nothing counted", the crossing increment is never
|
||||
// reported, and the web app is never notified for that day.
|
||||
func TestReviewB_LostCrossingOnReplyTimeout(t *testing.T) {
|
||||
backend := miniredis.RunT(t)
|
||||
var delay atomic.Int64
|
||||
proxy := delayProxy(t, backend.Addr(), &delay)
|
||||
h := newHarness(t, options{writeAddr: proxy, handlerTimeout: 10 * time.Second})
|
||||
|
||||
key := store.Key("A", "a@yagolivas.com", day, "o") // A/o = 20
|
||||
backend.Set(key, "19")
|
||||
|
||||
// warm the pool so the handshake is done before replies are delayed
|
||||
if st, body := h.post("/in", `{"recipient":"w@sttml.ru"}`); st != 200 {
|
||||
t.Fatalf("warmup %d %s", st, body)
|
||||
}
|
||||
delay.Store(int64(2500 * time.Millisecond)) // > timeout_ms 2000 of the harness
|
||||
st, body := h.post("/out", `{"trace_id":"t1","auth_user":"a@yagolivas.com","recipient":"x@gmail.com"}`)
|
||||
t.Logf("request 1: %d %s", st, strings.TrimSpace(body))
|
||||
v, _ := backend.Get(key)
|
||||
t.Logf("redis value after 503: %s", v)
|
||||
t.Logf("txlog:\n%s", h.tx.String())
|
||||
|
||||
delay.Store(0)
|
||||
time.Sleep(3 * time.Second)
|
||||
st, body = h.post("/out", `{"trace_id":"t2","auth_user":"a@yagolivas.com","recipient":"x@gmail.com"}`)
|
||||
t.Logf("request 2: %d %s", st, strings.TrimSpace(body))
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
if h.addrURL.count() == 0 {
|
||||
t.Errorf("A/o limit 20 was crossed (value %s -> now >20) but no notification was ever sent", v)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"migadu_counters/internal/notify"
|
||||
"migadu_counters/internal/txlog"
|
||||
)
|
||||
|
||||
func TestReviewB_NotifyURLCredentialsLogged(t *testing.T) {
|
||||
var gotAuth string
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
gotAuth = r.Header.Get("Authorization")
|
||||
}))
|
||||
defer ts.Close()
|
||||
u := strings.Replace(ts.URL, "http://", "http://admin:hunter2@", 1) + "/api/limits?level=$LEVEL&id=$ID&flow=$FLOW&period=$PERIOD&by=$ADDRESS&count=$COUNT"
|
||||
buf := &syncBuffer{}
|
||||
tx := txlog.New(buf, "text", nil)
|
||||
n := notify.New(notify.Config{OrgURL: u, Timeout: time.Second}, NotifyHooks(tx, nil))
|
||||
n.Notify(notify.Event{TraceID: "t", Route: "out", Level: "O", ID: "1049", Flow: "o", Address: "a@yagolivas.com", Count: 20, Limit: 20})
|
||||
n.Shutdown(context.Background())
|
||||
t.Logf("server saw Authorization=%q", gotAuth)
|
||||
t.Logf("counting log: %s", buf.String())
|
||||
if strings.Contains(buf.String(), "hunter2") {
|
||||
t.Errorf("notify credentials written to the counting log")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"migadu_counters/internal/counting"
|
||||
"migadu_counters/internal/domains"
|
||||
)
|
||||
|
||||
func TestReviewB_MalformedOrgIDDropsOtherParty(t *testing.T) {
|
||||
h := newHarness(t, options{})
|
||||
db, err := domains.Open("../../review/_proof_tests/b/bad.cdb")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
lf := counting.LookupFunc(db.Lookup)
|
||||
h.lookup.Store(&lf)
|
||||
st, body := h.post("/out", `{"trace_id":"t","auth_user":"a@good.com","recipient":"b@bad.com"}`)
|
||||
t.Logf("/out good.com -> bad.com: %d %s keys=%v", st, strings.TrimSpace(body), h.write.Keys())
|
||||
st, body = h.post("/in", `{"trace_id":"t","recipient":"b@bad.com"}`)
|
||||
t.Logf("/in bad.com: %d %s", st, strings.TrimSpace(body))
|
||||
if len(h.write.Keys()) == 0 {
|
||||
t.Errorf("sender a@good.com outgoing not counted because the recipient's CDB row is malformed")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"migadu_counters/internal/counting"
|
||||
"migadu_counters/internal/domains"
|
||||
"migadu_counters/internal/store"
|
||||
)
|
||||
|
||||
// F1: an /in request whose (never counted) mail_from is unusual-but-SMTP-valid
|
||||
// is rejected wholesale, so the recipient's incoming counters are not bumped.
|
||||
func TestReviewB_InMailFromDropsRecipientCount(t *testing.T) {
|
||||
h := newHarness(t, options{})
|
||||
for _, mf := range []string{
|
||||
"john smith@example.com", // go-smtp unquotes MAIL FROM:<"john smith"@example.com>
|
||||
"a@b@example.com", // MAIL FROM:<"a@b"@example.com>
|
||||
"info@bücher.de", // SMTPUTF8 U-label sender domain
|
||||
"postmaster@[192.0.2.1]", // address literal
|
||||
"bounce@mail_relay.example.com", // underscore host
|
||||
strings.Repeat("v", 250) + "@example.com", // > 254 chars
|
||||
} {
|
||||
st, body := h.post("/in", `{"trace_id":"t","mail_from":"`+mf+`","recipient":"b@yagolivas.com"}`)
|
||||
t.Logf("mail_from=%.40q -> %d %s", mf, st, strings.TrimSpace(body))
|
||||
}
|
||||
t.Logf("keys written: %v", h.write.Keys())
|
||||
if !h.write.Exists(store.Key("A", "b@yagolivas.com", day, "i")) {
|
||||
t.Errorf("recipient b@yagolivas.com never counted although every message was accepted by mizu")
|
||||
}
|
||||
}
|
||||
|
||||
// F1b: /out where one recipient has a domain the validator dislikes drops the
|
||||
// sender's outgoing increment too.
|
||||
func TestReviewB_OutRecipientDropsSenderCount(t *testing.T) {
|
||||
h := newHarness(t, options{})
|
||||
for _, rc := range []string{"x@[192.0.2.1]", "x@bücher.de", "x@host_name.example.com"} {
|
||||
st, body := h.post("/out", `{"trace_id":"t","auth_user":"a@yagolivas.com","mail_from":"a@yagolivas.com","recipient":"`+rc+`"}`)
|
||||
t.Logf("recipient=%q -> %d %s", rc, st, strings.TrimSpace(body))
|
||||
}
|
||||
if !h.write.Exists(store.Key("A", "a@yagolivas.com", day, "o")) {
|
||||
t.Errorf("sender a@yagolivas.com outgoing never counted")
|
||||
}
|
||||
}
|
||||
|
||||
// F2: plus-addressing creates a separate A counter per tag.
|
||||
func TestReviewB_PlusTagSeparateCounters(t *testing.T) {
|
||||
h := newHarness(t, options{})
|
||||
for _, r := range []string{"b+1@yagolivas.com", "b+2@yagolivas.com", "B+3@yagolivas.com"} {
|
||||
if st, body := h.post("/in", `{"recipient":"`+r+`"}`); st != 200 {
|
||||
t.Fatalf("%d %s", st, body)
|
||||
}
|
||||
}
|
||||
t.Logf("keys: %v", h.write.Keys())
|
||||
if !h.write.Exists(store.Key("A", "b@yagolivas.com", day, "i")) {
|
||||
t.Errorf("mailbox b@yagolivas.com A/i counter (what the web app reads) never incremented")
|
||||
}
|
||||
}
|
||||
|
||||
// F5: a recovered panic is answered 500 but not counted in requests_total.
|
||||
func TestReviewB_PanicNotInMetrics(t *testing.T) {
|
||||
h := newHarness(t, options{})
|
||||
boom := counting.LookupFunc(func(string) (*domains.Domain, error) { panic("boom") })
|
||||
h.lookup.Store(&boom)
|
||||
st, _ := h.post("/in", `{"recipient":"b@yagolivas.com"}`)
|
||||
got := h.metric(h.m.Requests.WithLabelValues("/in", "500"))
|
||||
t.Logf("status %d, requests_total{/in,500}=%v", st, got)
|
||||
if got != 1 {
|
||||
t.Errorf("500 from recovered panic not counted in requests_total")
|
||||
}
|
||||
}
|
||||
|
||||
// F6: rejection log has no trace_id.
|
||||
func TestReviewB_RejectLogHasNoTraceID(t *testing.T) {
|
||||
h := newHarness(t, options{})
|
||||
st, _ := h.post("/in", `{"trace_id":"TRACE-XYZ","recipient":"nobody"}`)
|
||||
if st != 400 {
|
||||
t.Fatal(st)
|
||||
}
|
||||
if !strings.Contains(h.log.String(), "TRACE-XYZ") {
|
||||
t.Errorf("rejection log lacks trace_id:\n%s", h.log.String())
|
||||
}
|
||||
}
|
||||
|
||||
// Leakage: 503 body / health carry raw redis errors with internal addresses.
|
||||
func TestReviewB_ErrorLeak(t *testing.T) {
|
||||
h := newHarness(t, options{})
|
||||
addr := h.write.Addr()
|
||||
h.write.Close()
|
||||
st, body := h.post("/in", `{"recipient":"b@yagolivas.com"}`)
|
||||
t.Logf("api: %d %s", st, strings.TrimSpace(body))
|
||||
resp, err := http.Get(h.admin.URL + "/health")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ := io.ReadAll(resp.Body)
|
||||
resp.Body.Close()
|
||||
t.Logf("health: %d %s", resp.StatusCode, strings.TrimSpace(string(b)))
|
||||
if strings.Contains(body, addr) {
|
||||
t.Logf("API 503 body leaks redis address %s", addr)
|
||||
}
|
||||
}
|
||||
|
||||
// Chunked body over 64 KB is rejected 413.
|
||||
func TestReviewB_ChunkedTooLarge(t *testing.T) {
|
||||
h := newHarness(t, options{})
|
||||
pr, pw := io.Pipe()
|
||||
go func() {
|
||||
pw.Write([]byte(`{"recipient":"b@yagolivas.com","pad":"`))
|
||||
for i := 0; i < 100; i++ {
|
||||
pw.Write([]byte(strings.Repeat("x", 1024)))
|
||||
}
|
||||
pw.Write([]byte(`"}`))
|
||||
pw.Close()
|
||||
}()
|
||||
req, _ := http.NewRequest("POST", h.api.URL+"/in", pr)
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ := io.ReadAll(resp.Body)
|
||||
resp.Body.Close()
|
||||
t.Logf("chunked 100KB: %d %s (cl=%d)", resp.StatusCode, b, req.ContentLength)
|
||||
if resp.StatusCode != 413 {
|
||||
t.Errorf("want 413")
|
||||
}
|
||||
}
|
||||
|
||||
// Empty configured token = no auth at all.
|
||||
func TestReviewB_EmptyTokenOpen(t *testing.T) {
|
||||
h := newHarness(t, options{token: "x"})
|
||||
_ = h
|
||||
}
|
||||
|
||||
var _ = net.Listen
|
||||
var _ = time.Second
|
||||
@@ -0,0 +1,142 @@
|
||||
# Review A — correctness and runtime behaviour
|
||||
|
||||
- Date: 2026-10-13/14 (interrupted by an API usage limit, resumed)
|
||||
- Scope: the whole repository at the state of the first implementation; read-only
|
||||
- Lens: counting rules vs plan/Lua, Redis store, UTC day, concurrency and lifecycle, resource use, error paths
|
||||
- Proof tests: `_proof_tests/a/` (see `_proof_tests/README.md`)
|
||||
- Verification by the coordinating agent and the merged ranking: `README.md`
|
||||
|
||||
The report below is the reviewer's text as delivered. Paths under `/tmp/review_a` refer to its scratch copy, which no longer exists.
|
||||
|
||||
---
|
||||
|
||||
Findings, most severe first. The repo was not modified. All scratch work is in /tmp/review_a: test files `internal/server/review_test.go`, `internal/store/review_test.go`, `internal/counting/review_test.go`, `internal/txlog/review_bench_test.go`, and a daemon build in `run/`. I started Redis on port 16401 and have killed it, along with any daemon processes I started.
|
||||
|
||||
---
|
||||
|
||||
**F1 — high — correctness/lifecycle: a client disconnect cancels the Redis write**
|
||||
- location: `internal/server/server.go:156` (`s.d.Counter.Execute(r.Context(), plan)`)
|
||||
- summary: Counting uses the request context. When mailqueuer closes the connection or its client times out before the reply, Go cancels that context and go-redis aborts before sending the MULTI (pool `waitTurn`, dial and retry backoff all check `ctx.Done()`). The increment is lost.
|
||||
- failure scenario: A fire-and-forget client writes POST /in and closes the socket, or uses a short timeout while Redis or Sentinel is slow. The handler then logs "redis error: context canceled (nothing counted)" and the counter is never incremented. Because mailqueuer never retries, the loss is silent and permanent.
|
||||
- evidence: `TestReview_ClientDisconnectNoDelay` sends 300 raw POST /in requests that close right after writing; the counter ends at "21", so 93% were lost. `TestReview_ClientDisconnectLosesCount`, with 50 ms of latency before Redis, loses 20 of 20. The txlog shows `redis error: redis incr: context canceled (nothing counted)`.
|
||||
- verdict: CONFIRMED. How often it bites depends on how the not-yet-written mailqueuer client behaves.
|
||||
- fix: `ctx, cancel := context.WithTimeout(context.WithoutCancel(r.Context()), budget)` for `Execute`, so the work does not depend on the caller staying connected.
|
||||
|
||||
**F2 — medium — counting rules / contract: master/token logins on /out get a 400, so nothing is counted**
|
||||
- location: `internal/api/validate.go:29,84` (`requiredEmail("auth_user")`, exactly one `@`)
|
||||
- summary: mizu puts the raw SASL login in X-Auth-User (`mizu/pkg/smtp/server.go:2065` passes `s.authenticatedUser`). For master/token logins (Sora support access, remotelookup tokens) that login is `user@dom@SUFFIX`, which NormalizeEmail rejects.
|
||||
- failure scenario: `{"auth_user":"x@yagolivas.com@SUPPORTTOKEN","recipient":"y@yagolivas.com"}` returns 400 `auth_user: invalid e-mail address`. Neither the sender's outgoing A/D/O nor the hosted recipient's incoming A/D/O is counted. The counting log has no line for it, and the warn log omits trace_id, which the plan requires.
|
||||
- evidence: `TestReview_MasterLoginAndOddSenders` prints `-> 400`, and the Redis keys are `[]`. mizu `CLAUDE.md:471-501` documents the `user@domain@SUFFIX` logins.
|
||||
- verdict: CONFIRMED (rejection). How frequent these logins are is PLAUSIBLE, not measured.
|
||||
- fix: Normalise auth_user by stripping a second `@SUFFIX` and counting the base address (`local@domain`). Log rejections to the counting log with the trace_id.
|
||||
|
||||
**F3 — medium — counting rules: a `mail_from` the sender controls can suppress counting of the recipient**
|
||||
- location: `internal/api/validate.go:35,49` (`optionalEmail("mail_from")` fails the whole request)
|
||||
- summary: mail_from is used only for the exempt check, yet a syntactically "invalid" envelope sender returns 400 for the whole request. go-smtp (the migadu fork used by mizu, `parse.go:173-238`) accepts these senders: it unquotes quoted local parts (`"a b"@x` becomes `a b@x`, `"a@b"@x` becomes `a@b@x`), and it accepts address literals, UTF-8 domains and trailing dots. mailqueuer passes X-Mail-From through unchanged.
|
||||
- failure scenario: Inbound spam with `MAIL FROM:<"a b"@spammer.example>` (or `<u@[192.0.2.1]>`, `<u@münchen.de>`, `<u@example.com.>`) gets a 400 on /in, so the hosted recipient's incoming limits are never counted. The same applies to /out for allowed_from regex or wildcard senders.
|
||||
- evidence: the same test: all five /in variants return 400 `mail_from: invalid e-mail address`, and no keys are written.
|
||||
- verdict: CONFIRMED (rejection). That mizu accepts such senders end-to-end is PLAUSIBLE (shown in the parser only).
|
||||
- fix: Treat an unparseable mail_from as unknown: log it, don't match it against the exempt list, and count anyway. Validate strictly only the fields that are counted.
|
||||
|
||||
**F4 — medium — Redis store: ambiguous Redis failures are logged as "nothing counted" and the crossing notification is lost**
|
||||
- location: `internal/store/store.go:128-137`, `internal/server/server.go:156-166`, `internal/txlog/txlog.go:111`
|
||||
- summary: `TxPipelined` returns an error in two cases where the increments may already be committed:
|
||||
- (a) A read timeout after MULTI/EXEC was written. go-redis does not retry (`txPipelineProcessCmds` returns canRetry=false), but Redis executes the transaction.
|
||||
- (b) A per-command error inside EXEC (non-integer or WRONGTYPE key). The other INCRBYs are applied.
|
||||
|
||||
In both cases the service answers 503, logs "nothing counted" and skips crossing detection. The one increment that crossed is consumed, so that level/id/flow is not notified again that day.
|
||||
- failure scenario:
|
||||
- (a) A/o=99 with limit 100 and the master stalls for more than `timeout_ms` (500 ms). Redis applies 100, the service returns 503, and no notification is sent.
|
||||
- (b) The O key holds "garbage" and A/o=19 with limit 20. A becomes 20 and D becomes 1, but the response is 503 "nothing counted" and 0 notifications are sent.
|
||||
- evidence: `TestReview_TimeoutAfterWriteStillCounts` against real Redis 8.6 with `DEBUG SLEEP 1`: `Incr error: ... i/o timeout`, then `key after the 'failed' Incr: 100`. `TestReview_PartialExecFailure` (miniredis): `status 503 ... ERR value is not an integer`, `A/o=20 (limit 20 crossed) D/o=1`, `address notifications sent: 0`.
|
||||
- verdict: CONFIRMED
|
||||
- fix:
|
||||
- Use the per-command results: go-redis sets each cmd's error, so commands that succeeded can still be evaluated for crossing.
|
||||
- On a timeout after the write, re-read the keys from the master and run crossing detection on any key now at or above its limit. Alternatively, raise the read timeout for EXEC.
|
||||
- Log "outcome unknown" instead of "nothing counted".
|
||||
|
||||
**F5 — medium — resource use: huge CDB values are re-parsed (and their keys sorted) on every request**
|
||||
- location: `internal/domains/domains.go:154-167` (`ParseRecord` per lookup), `internal/txlog/txlog.go:215-229` (`summarize` sorts every key per request)
|
||||
- summary: boxmailhere.com (193 KB, 9,798 overrides) is read, split into a map of about 9.8k entries, then key-sorted for every /in or /out request that touches it.
|
||||
- failure scenario: Sustained inbound mail to boxmailhere.com costs about 2.6 ms of CPU and 1.6 MB of garbage per request on an M2, so 100 req/s comes to roughly 160 MB/s of allocations and a quarter of a core. An /out between two such domains doubles it.
|
||||
- evidence: `go test ./internal/txlog -bench Review` gives:
|
||||
|
||||
| benchmark | time/op | memory/op | allocs/op |
|
||||
|---|---|---|---|
|
||||
| LookupSmall | 1.6 µs | 544 B | 9 |
|
||||
| LookupHuge | 908 µs | 1.43 MB | 88 |
|
||||
| InRequestHuge (plan + txlog) | 2.64 ms | 1.60 MB | 266 |
|
||||
| Open (count iteration) | 85 ms | – | – |
|
||||
- verdict: CONFIRMED
|
||||
- fix: Cache parsed records per DB generation, dropping the cache on swap. Or resolve limits lazily: extract only `O/,D/,A/` and `<local>@/<flow>` from the raw value. Pre-sort the base keys once.
|
||||
|
||||
**F6 — low — counting rules / spec deviation: a malformed CDB row fails the whole request instead of being skipped**
|
||||
- location: `internal/domains/domains.go:116-123`, `internal/counting/counting.go:172-179`
|
||||
- summary: The README and plan say "Malformed CDB rows are skipped, not fatal" (the Lua plugin and stats-merger both treat them as not hosted). Here a bad org id surfaces as a lookup error and the request is answered 503.
|
||||
- failure scenario: A recipient row `|A/i=5` (empty org id), with sender x@good.example. PlanOut returns an error, the server answers 503 "domain lookup failed", and the valid sender's outgoing counters are not counted either. Every request touching that domain fails.
|
||||
- evidence: `TestReview_MalformedRowFailsWholeRequest`: `plan=<nil> err=lookup recipient domain bad.example: domain bad.example: invalid org id ""`.
|
||||
- verdict: CONFIRMED. There are no malformed rows in the current fixture.
|
||||
- fix: In `Lookup`, map a ParseRecord error to ErrNotFound (or a distinct "malformed, not counted" result) plus a warning, and continue with the other party.
|
||||
|
||||
**F7 — low — spec deviation: override keys are not lowercased**
|
||||
- location: `internal/domains/domains.go:54,74`
|
||||
- summary: The plan says parsing follows stats-merger, where keys are lowercased (stats-merger `quota.go:103,118`). The code only trims keys, while the local part is lowercased by `normalize`.
|
||||
- failure scenario: A CDB value with `Info@/o=5` and mail to or from `info@domain` gets A/o (for example 100) instead of 5. The per-mailbox limit is silently ignored.
|
||||
- evidence: code trace. ParseLimits stores `k` as given, and Limit looks up `localPart+"@/"+flow` with a lowercased localPart. The fixture has 0 mixed-case overrides out of 22,995.
|
||||
- verdict: CONFIRMED (code path). Impact with today's data is nil.
|
||||
- fix: `strings.ToLower(strings.TrimSpace(k))` in ParseLimits.
|
||||
|
||||
**F8 — low — lifecycle: SIGHUP during startup kills the daemon**
|
||||
- location: `cmd/migadu_counters/main.go:76-101` (`signal.Notify` runs only after newApp, the up-to-2 s Redis ping, Listen and Serve)
|
||||
- summary: Go's default action for SIGHUP is to exit, so a reload or newsyslog HUP during startup terminates the process.
|
||||
- failure scenario: The daemon starts while Redis is slow or unreachable (the ping can take up to 2 s), and `service reload` or newsyslog sends HUP. The process exits with status 129, and requests already being served are dropped.
|
||||
- evidence: Redis on 16401 was held with `DEBUG SLEEP 3`, the daemon started, and HUP was sent after 0.3 s: `daemon exited with status 129`. The app log stops after "domains db loaded".
|
||||
- verdict: CONFIRMED
|
||||
- fix: Call `signal.Notify` (or `signal.Ignore(SIGHUP)`) first thing in `run`, before newApp.
|
||||
|
||||
**F9 — low — notifications: in-memory notifications are dropped on restart or short outages; log is misleading**
|
||||
- location: `internal/notify/notify.go:148-185,217-223`
|
||||
- summary:
|
||||
- `Shutdown` closes `quit`, which aborts every pending retry immediately instead of using the 30 s drain.
|
||||
- Retries hold a semaphore slot during backoff (1+2+4+8 s), so at most 8 events are delivered every 15 s or more while the web app fails.
|
||||
- The total retry window is about 15 s plus attempt timeouts.
|
||||
- When a retry is cut off by shutdown, the Failed hook still reports "after 5 attempts".
|
||||
- failure scenario: A web app deploy lasting about 30 s, or a migadu_counters restart, while limits are crossed. Those crossings never notify, because the counter has already passed the limit and only a reset re-arms it.
|
||||
- evidence: code trace. `case <-n.quit: ... attempt = n.cfg.Attempts`, then `Failed(ev, u, n.cfg.Attempts, ...)`.
|
||||
- verdict: PLAUSIBLE
|
||||
- fix: Let the shutdown deadline, not `quit`, bound retries. Release the semaphore while sleeping. Report the real attempt count. Consider a persistent or replayable record of reached-but-undelivered events, for example a Redis list.
|
||||
|
||||
**F10 — low — TimeoutHandler/shutdown interplay: lingering handlers misreport**
|
||||
- location: `internal/server/server.go:95,181-183`, `cmd/migadu_counters/main.go:322-332`
|
||||
- summary:
|
||||
- After `http.TimeoutHandler` sends its 503, the handler goroutine keeps running, so the request is usually counted and notified despite the 503.
|
||||
- These goroutines are not tracked by `http.Server.Shutdown`. If one finishes after `notifier.Shutdown`, its crossing is dropped and logged as "no address_url configured, not notifying", with `notifications_total{result="disabled"}`.
|
||||
- A panic after the timeout is swallowed with no log, because TimeoutHandler's recover sends it into a channel nobody reads.
|
||||
- verdict: PLAUSIBLE (code trace)
|
||||
- fix: Give `Execute` its own context and budget (see F1). Track in-flight executions with a WaitGroup that Shutdown waits on before stopping the notifier. Distinguish "notifier closed" from "disabled".
|
||||
|
||||
---
|
||||
|
||||
**Checked, no issue found**
|
||||
- **A/D/O planning:** /out counts sender A/D/O outgoing on auth_user's row and hosted recipient A/D/O incoming on the recipient's row. /in counts the recipient only. No duplicate keys in one MULTI, including self-sends and same-org cases.
|
||||
- **Exempt and junk:** auth_user, mail_from and recipient (lowercased) are checked per the plan. Junk is skipped unless `count_junk`. Unknown domains are skipped. The behaviour changes against Lua are the ones the plan documents.
|
||||
- **`<local>@/<flow>` override:** the override wins over `A/<flow>`. A duplicate key resolves last-wins, as in Lua. Empty or non-numeric values are skipped.
|
||||
- **Limit 0 and negative limits:** limit 0 never crosses. This matches the web app, which strips `=0` overrides as "unlimited" (`config.ex` cdb_domain_limits). There are no zero or negative limits in the fixture.
|
||||
- **Crossing detection:** old < limit ≤ new on atomic INCRBY, exactly once under concurrency. After the web app's DEL it re-fires, and the web app's own once-per-day guard (`limit_exceeded_at`) suppresses the duplicate email.
|
||||
- **Key format:** matches `redis_sentinel.ex` (`cnt/<L>/<id>/d/YYYY-MM-DD/<i|o>`, DB 0, lowercased ids). TTL is refreshed in the same MULTI.
|
||||
- **UTC day:** the date is computed once per plan, so all keys in a request share the same day. `/counters`, counterctl and the txlog all use UTC.
|
||||
- **go-redis behaviour:**
|
||||
- TxPipeline is not retried after the write, so there is no double counting.
|
||||
- READONLY, OOM or NOREPLICAS during queueing turn into EXECABORT, which is all-or-nothing.
|
||||
- ReplicaOnly falls back to the master. Sentinel address slices are copied.
|
||||
- MGET parsing treats nil as 0.
|
||||
- **MULTI per-command error semantics:** handled as reported in F4, nothing beyond that.
|
||||
- **CDB swap:** the parsed Domain is a copy. Lookups happen right after `Load`, so the 30 s grace is enough. Reloads are serialised by reloadMu.
|
||||
- **fsnotify on kqueue:** a rename-over produces Remove followed by Create, so the reload fires. The `.tmp` file is ignored.
|
||||
- **File handles on reload:** a failed `cdb.Open` leaks the *os.File only until the GC finalizer closes it. Open with its count iteration takes about 85 ms.
|
||||
- **Watcher shutdown:** Stop and debounce races are harmless; a late reload only affects a process that is already exiting.
|
||||
- **Logging:** LogWriter Reopen and Write are mutex-safe. A SIGHUP during shutdown is ignored. The fatal channel is buffered for both servers.
|
||||
- **Notifier:** wg.Add happens under the closed-check lock, so there is no Add-after-Wait race. URL escaping and `$LEVEL`/`$LIMIT` replacement order are correct, and all six web-app params are present in the example config.
|
||||
- **Panics:** a panic before the timeout is recovered with a 500. Indexing in `/counters` and the address split is guarded by validation.
|
||||
- **Plus-addressing:** counted as a separate address, same as Lua.
|
||||
- **Tooling:** `go vet ./...` is clean, and all existing tests pass in the scratch copy.
|
||||
@@ -0,0 +1,136 @@
|
||||
# Review B — security, contracts and operability
|
||||
|
||||
- Date: 2026-10-13/14 (interrupted by an API usage limit, resumed)
|
||||
- Scope: the whole repository at the state of the first implementation; read-only
|
||||
- Lens: HTTP validation layer, input normalisation and keys, web-app notification contract, config and secrets, counterctl, metrics and /health, docs vs code, test quality
|
||||
- Proof tests: `_proof_tests/b/` (see `_proof_tests/README.md`)
|
||||
- Verification by the coordinating agent and the merged ranking: `README.md`
|
||||
|
||||
The report below is the reviewer's text as delivered. Paths under `/tmp/review_b` refer to its scratch copy, which no longer exists; the fixture `bad.cdb` now lives at `_proof_tests/b/bad.cdb`.
|
||||
|
||||
---
|
||||
|
||||
## Findings (most severe first)
|
||||
|
||||
Scratch copy: `/tmp/review_b`. My proof tests are `internal/server/review_b*_test.go`, `cmd/counterctl/review_b_test.go`, `internal/config/review_b_test.go` and the fixture `/tmp/review_b/bad.cdb`. Nothing under /Users/dice/Work was modified. No Redis server was started; every Redis was miniredis, so there are no processes to kill.
|
||||
|
||||
---
|
||||
|
||||
**F1 — high — contract / security (limit bypass)**
|
||||
- location: `internal/api/validate.go:35,49` (optional `mail_from` goes through `requiredEmail`), `:32`, `:78-103` (`NormalizeEmail`: ASCII-only domain, ≤254, no whitespace); `internal/server/server.go:115-118,133-136`
|
||||
- summary: If any field fails the strict syntax check, the whole request is rejected with 400. That includes `mail_from`, which is never counted. So the sender alone decides whether the recipient's increments happen, and on /out a recipient address can drop the sender's outgoing increments.
|
||||
- failure scenario: An external sender floods `b@yagolivas.com` with `MAIL FROM:<"john smith"@x.com>`. mizu's go-smtp fork (`migadu/go-smtp parse.go:208-224`) unquotes this to `john smith@x.com`. mailqueuer never validates X-Mail-From. Other spellings work the same way: `<"a@b"@x.com>`, `<info@bücher.de>` (mizu sets `EnableSMTPUTF8=true`), `<postmaster@[192.0.2.1]>`, `<x@mail_relay.example.com>`, or any path over 254 chars. Every /in call then returns 400 and the recipient's A/D/O incoming counters never move, so the incoming limit never fires. Legitimate SMTPUTF8 senders on IDN domains are not counted either. On /out, a recipient like `x@[192.0.2.1]`, `x@bücher.de` or `x@host_name.example.com` makes the call 400, so the authenticated sender's A/D/O outgoing increments are lost. A 400 also writes nothing to the counting log (see F9), so the loss is invisible there.
|
||||
- evidence: `TestReviewB_InMailFromDropsRecipientCount`: all 6 mail_from variants return `400 {"error":"mail_from: invalid e-mail address"}` and leave `keys written: []`. `TestReviewB_OutRecipientDropsSenderCount`: all 3 return `400 recipient: invalid e-mail address`, and `cnt/A/a@yagolivas.com/.../o` is never created.
|
||||
- verdict: CONFIRMED for the service behaviour. The end-to-end path is PLAUSIBLE, based on reading the go-smtp parser and confirming that mailqueuer does not validate the from header.
|
||||
- fix: On /in, and on /out for `mail_from`, never reject because of `mail_from`: sanitise it for logging and treat it as non-exempt. Convert U-label domains with `idna.Lookup.ToASCII`. On /out, validate sender and recipient independently and count whichever side is valid. Keep 400 only for a missing or garbage *counted* address.
|
||||
|
||||
**F2 — medium — input normalisation (two spellings, two counters)**
|
||||
- location: `internal/counting/counting.go:164-198` (A id = raw `address`; the override lookup uses the raw local part); `internal/api/validate.go:78`
|
||||
- summary: Plus-addressed recipients get their own A key per tag. The mailbox's own A counter, the one the web app reads and charts, is never incremented, and the per-mailbox `local@/i` override is never applied.
|
||||
- failure scenario: rcptd strips `+tag` when validating (see `rcptd/internal/tables/regex_plustag_test.go`), so mizu accepts `b+1@…`, `b+2@…` and so on. Mail sent to `b+N@yagolivas.com` creates `cnt/A/b+N@yagolivas.com/...` keys, each starting at 0. The mailbox `b@yagolivas.com` never reaches A/i. When a tag key does cross, the web app gets `id=b+N@…`, `get_active_mailbox_by_address` returns nil, and it only sends a "mailbox not found" Telegram canary. An attacker can create unbounded keys, each with a 31-day TTL. The Lua plugin had the same behaviour, but the plan's goal is keys the web app reads.
|
||||
- evidence: `TestReviewB_PlusTagSeparateCounters`: keys = `cnt/A/b+1@…/i, cnt/A/b+2@…/i, cnt/A/b+3@…/i, cnt/D/…, cnt/O/1049/…`; `cnt/A/b@yagolivas.com/.../i` is missing.
|
||||
- verdict: CONFIRMED
|
||||
- fix: Canonicalise the A-level id and the override local part by stripping `+…` from the local part (rcptd's recipient_delimiter), and apply the same rule in `/counters`.
|
||||
|
||||
**F3 — medium — correctness / notification contract**
|
||||
- location: `internal/store/store.go:123-143`, `internal/server/server.go:155-167`, `internal/txlog/txlog.go:111`, `internal/counting/counting.go:204-206`
|
||||
- summary: If Redis applies MULTI/EXEC but the reply is lost (read timeout, connection reset, handler timeout), the service answers 503 and logs "nothing counted". The counters were incremented, though. If that increment was the crossing one, no later increment can cross again, so the web app is never notified for that level/id/flow/day.
|
||||
- failure scenario: A/o limit 20, counter at 19. The master stalls longer than `timeout_ms` (fork, failover). Redis applies the INCR (value 20), go-redis returns `i/o timeout` (txPipeline reads are not retried), and the client gets 503. The counting log says `redis error … (nothing counted)`. The next mail gives count 21 with `reached:false`, and no notification is ever sent.
|
||||
- evidence: `TestReviewB_LostCrossingOnReplyTimeout` uses a delaying TCP proxy in front of miniredis. Request 1 returns `503 redis incr: read tcp …: i/o timeout`, and Redis then holds `20`. Request 2 returns `"count":21,"limit":20,"reached":false`. Zero notifications.
|
||||
- verdict: CONFIRMED
|
||||
- fix: For a failed or ambiguous EXEC, do not claim "nothing counted". Make crossing detection recoverable: on the next successful increment, notify when `old >= limit` and a per-day marker (`SET NX`) is absent; or use a marker key set together with the INCR. Log "outcome unknown" instead of "nothing counted".
|
||||
|
||||
**F4 — medium — config / auth**
|
||||
- location: `internal/server/middleware.go:140-142`; `internal/config/config.go:296-300` (empty token is only a warning), `:151-153` (unknown keys are only warnings)
|
||||
- summary: An empty `auth_token` turns off authentication on /in, /out and /counters, and a typo in the key name produces that state with only a log warning. The plan's step 4 makes the bearer token mandatory.
|
||||
- failure scenario: `auth-token = "s3cret"` and `allowed_cidr = ["10.0.0.0/8"]` (typos), or `MIGADU_COUNTERS_AUTH_TOKEN` missing from the rc environment. The daemon starts, anyone who can reach :8095 can inflate any customer's counters (which triggers the web app's limit actions) or read counters, and the allowlist is silently empty.
|
||||
- evidence: `TestReviewB_MisconfigStarts` "typo token": Load OK with `AuthToken="" prefixes=[]` and three warnings only.
|
||||
- verdict: CONFIRMED
|
||||
- fix: Make an empty `auth_token` a validation error, or require an explicit `insecure_no_auth = true`. Make unknown keys fatal, or at least fatal for security-relevant near-misses.
|
||||
|
||||
**F5 — medium/low — CDB robustness / spec deviation**
|
||||
- location: `internal/domains/domains.go:116-124` (`ParseRecord` errors on a bad org id), `internal/counting/counting.go:172-180`, `internal/server/server.go:145-151`
|
||||
- summary: One CDB row with a non-numeric org id causes 503 for every request that touches that domain. On /out it also drops the other party's valid increments. The plan says malformed rows are skipped, not fatal; Lua treated such a row as not hosted.
|
||||
- failure scenario: The row `bad.com x12|A/i=5` is deployed. `/out a@good.com → b@bad.com` returns 503, and `a@good.com`'s outgoing is not counted. Every `/in` to bad.com returns 503.
|
||||
- evidence: `TestReviewB_MalformedOrgIDDropsOtherParty` (fixture `/tmp/review_b/bad.cdb`): `503 {"error":"domain lookup failed: … invalid org id \"x12\""} keys=[]`.
|
||||
- verdict: CONFIRMED (the real fixture has no such rows today)
|
||||
- fix: Have `Lookup` return `ErrNotFound`, or a dedicated skip error, for unparsable records, log once, and count the other party.
|
||||
|
||||
**F6 — low — counterctl date warning**
|
||||
- location: `cmd/counterctl/main.go:88-90`
|
||||
- summary: The comparison is `now − midnight(D) > 31d`, so the warning fires for a date exactly 31 days back and claims its keys "will read as 0". The plan says "older than 31 days".
|
||||
- failure scenario: The current time is 2026-10-13 12:00 UTC and the operator runs `--date 2026-09-12`. That day's keys were last refreshed around 23:59 and live until about 2026-10-13 23:59, so real non-zero numbers print alongside "more than 31 days ago … will read as 0".
|
||||
- evidence: `TestReviewB_31DayWarningOffByOne`: the warning is printed; a key last incremented at 09-12 23:59 expires 10-13 23:59. The existing test covers only 12 and 42 days back.
|
||||
- verdict: CONFIRMED
|
||||
- fix: Warn only when `today − D ≥ 32 days` (always expired). At exactly 31 days, say the counts may be partial.
|
||||
|
||||
**F7 — low — secrets in logs**
|
||||
- location: `internal/config/config.go:279-287` (userinfo in notify URLs accepted, `http://` with basic auth accepted without warning), `internal/notify/notify.go:147`, `internal/txlog/txlog.go:125,133`, `internal/config/logger.go:110` (mode 0644)
|
||||
- summary: Credentials given as URL userinfo work, because Go sends them as basic auth. The counting log then prints the full URL, password included, on every sent or failed notification, into a world-readable file.
|
||||
- evidence: `TestReviewB_NotifyURLCredentialsLogged`: the server sees `Basic YWRtaW46aHVudGVyMg==`, and the log line is `sent limit exceeded notification to url "http://admin:hunter2@127.0.0.1:…/api/limits?…"`.
|
||||
- verdict: CONFIRMED
|
||||
- fix: Reject userinfo in the notify URLs, or redact with `url.Redacted()` before logging. Warn or fail when `user` is set with an `http://` URL.
|
||||
|
||||
**F8 — low — metrics**
|
||||
- location: `internal/server/middleware.go:73-104`, `server.go:96` (`recoverer(observe(timed))`; `observe` has no defer)
|
||||
- summary: A recovered panic is answered 500 but never counted in `requests_total`. The plan says every rejection or recovery increments `requests_total{status}`.
|
||||
- evidence: `TestReviewB_PanicNotInMetrics`: status 500, `requests_total{/in,500}=0`.
|
||||
- verdict: CONFIRMED
|
||||
- fix: Record in a deferred function in `observe`, or put `observe` outside `recoverer`.
|
||||
|
||||
**F9 — low — operability / spec deviation**
|
||||
- location: `internal/server/middleware.go:46-49`
|
||||
- summary: Rejection logs carry no `trace_id`, although the plan requires it "if parsable". The counting log gets nothing for 4xx rejections, despite README's "Every request writes one block" and `grep '<trace_id>'` returns one transaction.
|
||||
- evidence: `TestReviewB_RejectLogHasNoTraceID`: the log line is `msg="request rejected" route=/in remote=… status=400 reason="recipient: invalid e-mail address"`, without TRACE-XYZ. This combines with F1 to make the dropped counts invisible.
|
||||
- verdict: CONFIRMED
|
||||
- fix: Pass the sanitised trace_id and route into `reject`, and write a `rejected: <reason>` line to the txlog.
|
||||
|
||||
**F10 — low — information exposure**
|
||||
- location: `internal/server/server.go:150,165,236,263,301`; `internal/config/config.go:117` (default `admin_addr=":9095"`, all interfaces, no auth, no allowlist)
|
||||
- summary: 503 bodies, and `/health` (unauthenticated), return raw Redis and CDB errors that include internal addresses and paths. The default admin bind is public, while the code comment says to "bind admin_addr locally".
|
||||
- evidence: `TestReviewB_ErrorLeak`: API returns `{"error":"redis unavailable: redis incr: dial tcp 127.0.0.1:52115: connect: connection refused"}`, and `/health` returns `{"error":"redis: dial tcp 127.0.0.1:52115: …"}`.
|
||||
- verdict: CONFIRMED
|
||||
- fix: Return generic messages and keep the details in the app log. Default `admin_addr` to `127.0.0.1:9095`, or warn when it is not loopback.
|
||||
|
||||
**F11 — low — notification contract (idempotency)**
|
||||
- location: `internal/notify/notify.go:157-213`
|
||||
- summary: Up to 5 attempts, with a retry on client timeout, transport error, 429 or 5xx, against a non-idempotent web action. The A branch (`Core.Email.incoming/outgoing_limit_exceeded!`, `core/email.ex:3572-3584`) emails the customer on every call. The canary Telegram message is also sent before the O-level action can raise.
|
||||
- failure scenario: The web app takes more than 5 s (the per-attempt timeout) but completes. The service retries, and the customer gets 2–5 "limit exceeded" emails plus duplicate canaries. Lua sent once.
|
||||
- verdict: PLAUSIBLE
|
||||
- fix: Retry only on connect errors and 5xx-before-processing, or add an idempotency key (trace_id+level+id+flow+day) that the web app can dedupe on. Consider a longer timeout with fewer retries.
|
||||
|
||||
**F12 — low — config validation**
|
||||
- location: `internal/config/config.go:257-266`
|
||||
- summary: Sentinel mode is accepted without `redis.password`, although the deployment has one Redis password.
|
||||
- failure scenario: The password is missing or the env var is unset. The daemon starts, every INCR gets NOAUTH and returns 503, and mailqueuer ignores 503s, so nothing is counted fleet-wide. Only `/health` and `redis_errors_total` show it.
|
||||
- evidence: `TestReviewB_MisconfigStarts` "sentinel without password": Load OK, `warnings=[]`.
|
||||
- verdict: CONFIRMED
|
||||
- fix: Require the password in sentinel mode, or at least warn. Make a NOAUTH/WRONGPASS reply to the startup PING fatal.
|
||||
|
||||
**F13 — low — test-suite gaps (tests pass for the wrong reasons)**
|
||||
- `cmd/counterctl/main_test.go:180-208` never tests the 31/32-day boundary, which hides F6.
|
||||
- `server_test.go:482-497` checks the panic answer but not the metric (F8).
|
||||
- `:422-454` checks that rejections are logged but not the trace_id (F9).
|
||||
- `:372-395` never covers the empty-token open mode (F4).
|
||||
- Nothing covers an ambiguous Redis failure (F3), plus-tags (F2), or a malformed org id (F5).
|
||||
- `TestValidation_JSONAndFields` asserts as correct the 400 on a `mail_from` the system never counts (F1).
|
||||
- Watcher tests use a 16-slot buffered `done` channel plus sleeps, so a stale signal from an earlier event could satisfy `c.wait` in `TestWatcher_ReloadsOnAtomicReplaceTwice`. This is a mild flakiness and false-pass risk.
|
||||
- verdict: CONFIRMED (by inspection and by the F6/F8/F9 tests above)
|
||||
|
||||
## Checked, no issue found
|
||||
- Middleware order matches the plan's table: method → declared size → token → source IP → content-type → MaxBytesReader. Note that auth runs before the allowlist, so a non-allowed source can tell a valid token (403) from an invalid one (401); this is minor.
|
||||
- Body size: chunked 100 KB body → 413 (`TestReviewB_ChunkedTooLarge`). The declared Content-Length check sits before auth.
|
||||
- Auth: missing, `Basic` and empty `Bearer ` → 401 with `WWW-Authenticate`. ConstantTimeCompare leaks only the length. The `Bearer` scheme is case-sensitive, which is fine for mailqueuer (`routing.go:257`).
|
||||
- Content-Type: parameters are ignored, other types → 415.
|
||||
- JSON: syntax errors, two objects and trailing data → 400. `null` / null fields → "missing" 400. Wrong types → 400 (the message exposes the Go type name). Duplicate and case-insensitive keys: last wins. That is harmless because the caller is trusted.
|
||||
- allowed_cidrs: `Unmap()` handles IPv4-mapped clients. Zoned and IPv6-mapped config entries fail closed.
|
||||
- Unknown paths → 404 JSON. HEAD/PUT/OPTIONS → 405 with `Allow`. `//counters` → 301 → works. Metric labels: route is bounded to 4 values, and status/level/flow/op/result are bounded.
|
||||
- Log injection: the app log uses slog (values quoted). The counting text log can't receive newlines or ESC: addresses reject Cc/whitespace and trace_id strips controls. trace_id still allows spaces, `>` and non-ASCII, which deviates from the plan's "printable ASCII" but has low impact because mizu generates it.
|
||||
- Redis key injection: an address has exactly one `@`, the domain is restricted to `[a-z0-9.-]` and the org id is numeric, so no collision across customers or levels is possible. `/counters?domain=` is unvalidated but read-only and confined to `cnt/D/…`.
|
||||
- Notify URL contract: all six params are in the default templates. `id` is the address for A, the domain for D and the org id for O. Values are QueryEscaped. `/api/limits` routes to `InternalApi.NotificationController` with no route collision. `accepts json` is fine because there is no Accept header. COUNT is now the new value (= limit) where Lua sent limit−1; the plan intends this, and the web app only uses it in canary text.
|
||||
- Exempt via `/out` mail_from: mitigated by mizu `ResolveSender` / rcptd `allowed_from`. On `/in` it behaves the same as Lua.
|
||||
- go-redis txPipeline: read errors are not retried, so there is no double-increment.
|
||||
- Config: db≠0 is rejected, relative `domains_cdb` is resolved against the config dir, env overrides apply only when non-empty, notify URLs are checked for an http(s) scheme.
|
||||
- counterctl: config discovery, env token override, `:8095` → `http://localhost:8095`, trailing slash on `--url` works, `--url …/api` → 404 (expected), `!` uses ≥ (no zero limits exist in the CDB), exit codes are 0/1.
|
||||
- `/health` checks CDB readability and a 200 ms write-side PING. The README's "drain ≤ 30 s" matches the code; the plan said 10+10 s, a documentation-only difference.
|
||||
- CDB fixture: 93,549 rows, every row has all 6 base keys, there are 22,995 lowercase overrides, no non-numeric org ids, no zero or negative limits, and 112 punycode domains.
|
||||
@@ -0,0 +1,85 @@
|
||||
# TODO — migadu_counters
|
||||
|
||||
State as of 2026-10-15. Review findings refer to `review/README.md`; the proof tests live in
|
||||
`review/_proof_tests/` and should become regression tests as each item is fixed (round 1 and items
|
||||
7–8 are mapped at the end of `review/_proof_tests/README.md`).
|
||||
|
||||
## Done
|
||||
|
||||
- Review round 1: #1–#6 (queue + 202, lenient validation + IDN, token logins, INCR with per-key
|
||||
results and master re-read, CDB parse cache, mandatory token / port 5555), plus
|
||||
`contrib/pull-domains.sh` and `counterctl --in/--out`.
|
||||
- #7 plus-addressing counts against the mailbox (`domains.Mailbox`).
|
||||
- #8 malformed CDB row is "not hosted" + warning (`domains.ErrMalformed`).
|
||||
|
||||
## Review round 2 — remaining findings
|
||||
|
||||
Suggested order: #10, #15, #9, then the rest in one sweep.
|
||||
|
||||
### Can stop the daemon or lose counts
|
||||
|
||||
- [ ] **#10 SIGHUP during startup kills the daemon.** The signal handler is installed after the Redis
|
||||
ping (up to 2 s). Install `signal.Notify` before anything slow in `cmd/migadu_counters/main.go`
|
||||
(`run`), and queue a reload that arrives before the app is ready. Evidence: `review/_proof_tests/a/run/`.
|
||||
- [ ] **#15 Sentinel mode starts without a Redis password.** Every write fails; only `/health` and
|
||||
metrics show it. Make an empty `redis.password` fatal in `config.Validate` when `sentinels` is set
|
||||
(allow an explicit opt-out such as `password_optional = true` for dev). Proof:
|
||||
`b/internal/config/review_b_test.go`, case "sentinel without password".
|
||||
- [ ] **Persistent / recoverable counting on Redis outage** (deferred from round 1, #4). A Redis stall
|
||||
longer than a few seconds fills the queue and new requests are dropped (503, `dropped_total`).
|
||||
Options: marker key per transaction for exact lost-reply recovery; a disk-backed queue. Decide
|
||||
whether the drop-and-alert behaviour is acceptable before building anything.
|
||||
|
||||
### Correctness
|
||||
|
||||
- [ ] **#9 Notification retries can send duplicate customer emails.** The web app's mailbox-level
|
||||
handler (`core/email.ex:3572-3584`) emails on every call; the org handler is guarded once per day.
|
||||
In `internal/notify`: treat a slow-but-2xx attempt as success (do not retry after the client timeout
|
||||
if the request was sent — or make the retry budget shorter than the web app's handling time), do
|
||||
not keep the concurrency slot while backing off, let Shutdown finish pending retries up to its
|
||||
deadline, and fix the "after N attempts" count in the failure line.
|
||||
- [ ] **#12 Override keys are not lowercased.** `Info@/o=5` would be ignored and the domain default
|
||||
used instead. Lowercase the key in `domains.ParseLimits`; add a test row. No such keys in today's
|
||||
data.
|
||||
- [ ] **#16 counterctl warns one day too early.** A date exactly 31 days back is reported as expired.
|
||||
Compare against the key TTL in whole days (`now.UTC` day minus date > 31). Proof:
|
||||
`b/cmd/counterctl/review_b_test.go` `TestReviewB_31DayWarningOffByOne`.
|
||||
- [ ] **#18 Work keeps running after a handler timeout.** Not reproduced by the reviewer and largely
|
||||
covered by the queue change (the synchronous part only validates). Re-check that
|
||||
`http.TimeoutHandler` cannot leave a goroutine writing after the 503, then close.
|
||||
|
||||
### Operability / security hygiene
|
||||
|
||||
- [ ] **#11 Rejected requests leave no trace.** 4xx rejections write nothing to the counting log and no
|
||||
`trace_id` to the app log. Decode `trace_id` best-effort before rejecting; add a
|
||||
`rejected: <status> <reason>` line to the counting log and `trace_id` to the warn line. Proof:
|
||||
`b/internal/server/review_b_test.go` `TestReviewB_RejectLogHasNoTraceID`.
|
||||
- [ ] **#13 Credentials in the notify URL are written to the log.** Redact `user:pass@` from URLs in
|
||||
the txlog `sent/failed notification` lines; warn at startup when a notify URL is plain `http://`
|
||||
with basic auth. Proof: `b/internal/server/review_b3_test.go`.
|
||||
- [ ] **#14 Error responses expose internals.** 503 bodies carry Redis addresses and file paths; keep
|
||||
the detail in the log and answer a generic message. Default `admin_addr` to `127.0.0.1:9095`
|
||||
(currently `:9095`, all interfaces, no auth). Proof: `TestReviewB_ErrorLeak`.
|
||||
- [ ] **#17 A recovered panic is not counted in `requests_total`.** The recoverer sits outside
|
||||
`observe`; swap the order or record the 500 explicitly. Proof: `TestReviewB_PanicNotInMetrics`.
|
||||
- [ ] **Test gaps (B F13).** Watcher tests can occasionally pass on a stale signal — make them wait for
|
||||
the specific reload they triggered. Add a test for `handler_timeout` vs queue timeout interplay.
|
||||
|
||||
## Outside this repo (deployment prerequisites)
|
||||
|
||||
- [ ] **mailqueuer client**: `[counters]` section and a fire-and-forget POST `/in`|`/out` per recipient
|
||||
inside the publish loop of `mailqueuer/internal/handler/ingest.go` (after a successful publish;
|
||||
`/out` when `X-Auth-User` is set). Reply is `202 {"accepted":true}`; counted values only in the
|
||||
counting log and `GET /counters`.
|
||||
- [ ] **Ansible role** in `ansible-freebsd3`, cloned from `roles/rcptd`: user/group, `config.toml.j2`
|
||||
(`auth_token` is mandatory, port 5555), rc.d script under `daemon(8)`, newsyslog entries for
|
||||
`app.log`, `counting.log`, `pull.log`, templated `contrib/pull-domains.sh` +
|
||||
`contrib/migadu_counters-pull.cron` (see `contrib/README.md`), agent host key acceptance,
|
||||
firewall: mailqueuer hosts → :5555, service → Sentinel :26379/:6379 and admin.migadu.com:443.
|
||||
- [ ] **migadu_web** `/api/limits` `D` branch (domain level); until then `notify.domain_url` stays empty.
|
||||
- [ ] **Limits re-tuning**: outgoing is counted per recipient (Lua counted per message), so existing
|
||||
`*/o` limits are stricter. Review before relying on `outgoing_limit_exceeded!`.
|
||||
- [ ] **Retire stats-merger's quota module** once this service is the source (only fernrohr reads it).
|
||||
- [ ] **Dependencies**: `golang.org/x/net` is pinned to v0.43.0 because the offline module cache had
|
||||
it; run `go get -u golang.org/x/net && go mod tidy` with network access.
|
||||
- [ ] **git init** the repo (never asked for so far).
|
||||
Reference in New Issue
Block a user