dimitrije/counters
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

First commit Go counter service

Browse Source
This commit is contained in:
Dimitrije Dimitrijevic
2026-10-15 02:12:18 +02:00
commit c43ff17627
59 changed files with 11205 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+213
View File
@@ -0,0 +1,213 @@
# migadu_counters
Real-time mail counting for Migadu. mailqueuer tells it about every ingested
recipient; it increments per-address / per-domain / per-organisation daily
counters in Redis (the keys the web app already charts) and, when an increment
reaches a configured limit, notifies the web app, which decides what happens to
the customer. It replaces the rspamd Lua plugin `migadu_counters.lua` and the
ledger-derived quota in stats-merger. It returns no verdicts and enforces
nothing itself.
Design record: `plans/2026-10-13-migadu-counters.md`.
```
sender ──SMTP──▶ mizu ──POST /ingest──▶ mailqueuer
│ per recipient, after a successful publish:
│ fire-and-forget HTTP call, reply ignored
▼
POST /out {"trace_id","auth_user","mail_from","recipient"} ← X-Auth-User set
POST /in {"trace_id","mail_from","recipient","junk"} ← X-Auth-User empty
Authorization: Bearer <token>
▼
┌─────────────────────────────────────┐
rspamd_domains_db ───▶│ migadu_counters (:5555, admin :9095)│
(domain → org|limits) │ validate → 202 → queue → workers: │
│ lookup domain → MULTI/INCR → notify │──▶ GET admin.migadu.com/api/limits
└──────────────────┬──────────────────┘ (on the increment that crosses a limit)
▼
Redis Sentinel: writes → master, reads → replicas
DB 0: cnt/<A|D|O>/<id>/d/<YYYY-MM-DD>/<i|o> (TTL 31 d)
counterctl --address x@y [--date YYYY-MM-DD] ──GET /counters──▶ migadu_counters ──▶ replicas
```
## API (for mailqueuer)
Every request: `Authorization: Bearer <auth_token>` (same value in mailqueuer's
config; the daemon refuses to start without one), `Content-Type:
application/json`, body ≤ 64 KB, one JSON object. Unknown fields are ignored.
mailqueuer never waits for or acts on the reply.
Counting is **decoupled from the request**: `/in` and `/out` validate the body,
enqueue a job and answer `202` at once; a worker pool does the CDB lookup, the
Redis transaction and the notification under its own time budget, so a client
that hangs up or times out can no longer cancel a count. The counted values
appear in the counting log and in `GET /counters`, not in the reply.
| | |
|---|---|
| `POST /out` | `{"trace_id":"…","auth_user":"x@example.com","mail_from":"x@example.com","recipient":"y@example.com"}` — one call per recipient of an authenticated submission. Sender A/D/O **outgoing** +1 (if the sender's domain is hosted), recipient A/D/O **incoming** +1 (if the recipient's domain is hosted). `auth_user` and `recipient` are validated independently: whichever side is a usable address is counted, the other is logged as `not usable`; `400` only when neither is. A support/token login `user@dom@SUFFIX` counts under `user@dom` (the suffix is logged). |
| `POST /in` | `{"trace_id":"…","mail_from":"s@gmail.com","recipient":"y@example.com","junk":false}` — one call per recipient of inbound mail. Recipient A/D/O **incoming** +1 if hosted. `junk:true` is skipped unless `count_junk = true`. `mail_from` may be `""` (bounce) or anything mizu accepted — it only feeds the exempt check and never rejects the request. |
| `GET /counters?address=…` or `?domain=…` `[&date=YYYY-MM-DD]` | Today's (UTC) or a given day's values and limits, read from the replicas. |
Responses: `202 {"accepted":true}` (validated and queued; skips such as
exempt/junk/unknown domain are decided by the worker and appear only in the
counting log), `4xx {"error":"…"}` from the validation layer (`405` method,
`413` size, `401` token, `403` source IP, `415` content type, `400` JSON /
required fields / e-mail syntax), `503 {"error":"queue full"}` when the
counting queue (`queue_size`) is full — the drop is counted in
`dropped_total{route}` and logged — or `503` on a handler timeout of the
synchronous part.
Rules: addresses are trimmed and lowercased, Unicode domains are converted to
punycode (the CDB key), and a plus-address tag is dropped so `b+tag@…` counts
against the mailbox `b@…` (the counter the web app reads); mail to or from an exempt address (`support@migadu.com`,
`contact@migadu.com`) is never counted; a domain that is not in
`rspamd_domains_db` is not hosted here and not counted; the sender of `/in`
mail is never counted; there is no reject/soft-reject — counting continues past
the limit.
## Limits and keys
Limits come from the rspamd domains CDB (tinycdb, built by the agent):
`example.com → 1049|O/i=200,O/o=20,D/i=200,D/o=20,A/i=200,A/o=20,info@/o=5`.
`<local>@/<flow>` overrides the `A/<flow>` default for one mailbox; a missing
key means unlimited. The file is reloaded on SIGHUP and automatically when it
changes on disk (atomic `mv` is detected); a file that cannot be read or is
empty is rejected and the current table kept. A row that exists but cannot be
parsed (non-numeric org id) is treated as "not hosted" for that domain and
warned about in the app log; the other side of the request is still counted.
Always replace the file with
`mv` (as the agent's `cdb -c` + `mv` do) — rewriting it in place truncates the
inode the daemon holds open and breaks lookups until the next reload.
Redis keys are exactly the web app's (`migadu_web` `core/redis_sentinel.ex`):
`cnt/<A|D|O>/<id>/d/<YYYY-MM-DD>/<i|o>` in **DB 0**, UTC day, TTL 31 days,
`INCR`+`EXPIRE` in one MULTI/EXEC. The web app may `DEL` a day key to reset a
customer; the next crossing notifies again.
Redis failures are handled per outcome: a key whose own `INCR` fails (it holds
a non-number) is logged `redis error on key …` while the other keys are counted
and checked for crossings; when the EXEC reply is lost after the write (timeout,
reset) the keys are re-read once from the master and the block says `redis
outcome unknown … re-read from master`, with a crossing reported when
`count >= limit`; a definite failure (connection refused) logs `redis error: …
(nothing counted)`.
The CDB is pulled from the agent host by `contrib/pull-domains.sh` (see
`contrib/README.md`): rsync, compare, verify, atomic `mv`, every 15 s.
A limit is "reached" by the increment that moves the counter from below the
limit to at or above it (`old < limit ≤ new`). Because INCRBY is atomic exactly
one increment crosses, so the web app is notified once per level/id/flow/day —
`GET <org_url|address_url|domain_url>` with `$LEVEL $ID $FLOW $PERIOD(=d)
$ADDRESS $COUNT $LIMIT` substituted (URL-escaped), basic auth, in the
background, 5 attempts with exponential backoff; 4xx is not retried. An empty
URL disables notifications for that level.
## Counting transaction log
Every request writes one block in the shape of the old Lua plugin's log lines
(rspamd's task id replaced by the trace id) to `logging.counting_output`:
```
2026-10-06 01:12:06.412 <ad8525> out; request auth_user=noah@example.com recipient=y@example.org mail_from=noah@example.com
2026-10-06 01:12:06.412 <ad8525> out; local domain: example.com account_id=116681
2026-10-06 01:12:06.412 <ad8525> out; domain: example.com limits_map={A/i=500, A/o=100, D/i=10000, D/o=2200, O/i=10000, O/o=2200}
2026-10-06 01:12:06.412 <ad8525> out; determined limit (level=[A] flow=[o] key=[cnt/A/noah@example.com/d/2026-10-06/o]) => 100
2026-10-06 01:12:06.412 <ad8525> out; recipient domain not in domains db: example.org (not counted)
2026-10-06 01:12:06.414 <ad8525> out; tracked address counters for noah@example.com => count=12 limit=100 reached=false
2026-10-06 01:12:06.414 <ad8525> out; tracked account counters for 116681 => count=2200 limit=2200 reached=true
2026-10-06 01:12:06.414 <ad8525> out; limit reached (level=[O] id=[116681] flow=[o] by=[noah@example.com]) => notifying org_url
2026-10-06 01:12:06.903 <ad8525> out; sent limit exceeded notification to url "https://admin.migadu.com/api/limits?…"
```
`grep '<trace_id>'` returns one transaction; `grep 'skipped:'` every request
that counted nothing; `grep 'not usable'` every side that failed validation
(`sender address not usable: nobody (invalid e-mail address, not counted)`);
`grep 'dropped:'` every request refused by a full queue. A token login shows as
`… login_suffix=SUPPORTTOKEN` on the request line. `counting_format = "json"` emits the same events as one
JSON object per line. Domains with thousands of per-mailbox overrides print the
base keys plus the address's own overrides and a count of the rest.
## Running
```
migadu_counters -config /usr/local/etc/migadu_counters/config.toml # default paths: /etc/…, /usr/local/etc/…
migadu_counters -version
```
Foreground process; two listeners: `addr` (API, default `:5555`) and
`admin_addr` (`/health`, `/metrics`, no auth). Signals: `SIGHUP` reopens both
log files and reloads the CDB; `SIGTERM`/`SIGINT` stop accepting, drain the
counting queue (every accepted request is counted), wait for in-flight
notifications (≤ 30 s) and exit 0. On FreeBSD run it like rcptd: an rc.d script
with `/usr/sbin/daemon -f -H -p <pidfile> -o <logfile>`; `service
migadu_counters reload` sends SIGHUP. There is deliberately no self-daemonising
or pidfile option (daemon(8) owns both). Startup fails without a readable CDB;
an unreachable Redis is only warned about (`/health` answers 503 until it is
back).
See `config.toml.example` for every option. `auth_token` is mandatory and
unknown keys (typos such as `auth-token`) are fatal at startup. Secrets may
come from the environment: `MIGADU_COUNTERS_AUTH_TOKEN`,
`MIGADU_COUNTERS_REDIS_PASSWORD`, `MIGADU_COUNTERS_NOTIFY_PASSWORD`. `redis.db` must stay 0. Production uses
Sentinel (`sentinels`, `master_name = "mymaster"`): writes go to the master,
reads to a replica (falling back to the master when none is known); without
`sentinels` the static `write_addr`/`read_addrs` are used.
Metrics (`/metrics` on the admin port): `migadu_counters_requests_total{route,status}`,
`_request_duration_seconds{route}`, `_increments_total{level,flow}`,
`_limit_reached_total{level,flow}`, `_notifications_total{level,result}`,
`_redis_errors_total{op}` (`incr`, `incr_unknown`, `mget`, `ping`),
`_dropped_total{route}`, `_queue_depth`, `_queue_wait_seconds`, `_cdb_domains`,
`_cdb_reloads_total{result}`.
## counterctl
```
counterctl --address me@example.com # today, UTC
counterctl --address me@example.com --date 2026-10-12
counterctl --address me@example.com --out # outgoing columns only (--in: incoming)
counterctl --domain example.com --json
```
Reads `addr`/`auth_token` from the daemon's config (`--config`, default paths,
`MIGADU_COUNTERS_AUTH_TOKEN` overrides the token; `--url` overrides the
address) and calls `GET /counters`. Output:
```
me@example.com date 2026-10-13 (UTC) domain example.com org 1234
LEVEL ID IN LIMIT OUT LIMIT
A me@example.com 12 500 200! 200
D example.com 40 1000 9 -
O 1234 120 5000 30 1000
```
`!` marks a value at or above its limit, `-` an unlimited level. `--in` /
`--out` restrict the table to one flow (`LEVEL ID IN LIMIT` or `LEVEL ID OUT
LIMIT`; both or neither show the full table; `--json` ignores them). `--date` takes
an ISO 8601 calendar date (`YYYY-MM-DD`); future dates are refused and dates
older than 31 days are warned about (keys have expired). Exit 0 on success, 1
on any error.
## Development
```
make build # bin/migadu_counters, bin/counterctl
make test # go test -race ./... (tests use test_files/rspamd_domains_db.cdb and miniredis)
make vet # go vet + gofmt
make test-contrib # shell test for contrib/pull-domains.sh (needs tinycdb cdb + rsync)
make freebsd # static FreeBSD amd64 binaries for the Ansible role
```
Manual check with a local Redis: `redis-server --port 6379`, a config in static
mode (`write_addr = "127.0.0.1:6379"`) pointing `domains_cdb` at
`test_files/rspamd_domains_db.cdb`, then
```
curl -s -XPOST localhost:5555/out -H 'Authorization: Bearer <token>' -H 'Content-Type: application/json' \
-d '{"trace_id":"t1","auth_user":"a@yagolivas.com","recipient":"x@gmail.com"}' # -> {"accepted":true}
redis-cli -n 0 KEYS 'cnt/*'
go run ./cmd/counterctl -config config.toml --address a@yagolivas.com --out
```