Alerts across clients¶

Tenantry evaluates alert rules for all clients after each polling round (every 60 s by default) and notifies the configured channels. The Alerts page (/alerts) lists the active alerts and the history (last 200 entries); the dashboard shows the count and the latest entries. Settings › Alerts (/settings/alerts) configures the threshold and the channels (administrators; other users read).

1. Rules¶

Kind	Fires when	Severity	Key (deduplication)
`proxy_unreachable`	A client proxy (gateway, or any node with a `client_code`) has not been seen for more than 20 minutes. Auth/Proxy nodes are ignored.	critical	`proxy:<id>`
`identity_expiring`	Tenantry's Teleport identity file expires in less than 24 h (critical once expired). Only with `TELEPORT_IDENTITY_FILE`. Under Machine ID (1 h certificates by design) only once expired.	warning	`identity`
`identity_not_renewed`	Machine ID (`--auth machine-id`): the identity file was not rewritten by `tbot` within 2 x the renewal interval (40 min by default), or does not exist. `tbot` is probably down; resolves once it writes the file again.	warning	`identity:renewal`
`console_down`	A console failed 2 polls in a row (unreachable, TLS, token…) or reports `health.status = down`.	critical	`console:<id>:down`
`console_audit_broken`	A console reports `audit.chain = broken`.	critical	`console:<id>:audit`
`console_dc_down`	A console reports a domain controller `down` (one alert per DC).	warning	`console:<id>:dc:<fqdn>`
`console_upgrade_overdue`	A planned Steward upgrade (Upgrading consoles) is still not visible in the console version after the configured delay (default 7 days). Only evaluated while the console answers its polls.	warning	`console:<id>:upgrade`
`failed_logins_spike`	A console reports `audit.failedLogins24h` above the threshold (default 20, Settings › Alerts).	warning	`console:<id>:logins`
`certificate_expiring`	The certificate presented by a console (read during its status poll, also when TLS verification fails) or by the Teleport proxy (one TLS handshake per round) expires in less than 14 days; critical when expired or under 3 days. Evaluated only when the certificate was read in this round. See TLS certificates.	warning	`console:<id>:cert`, `teleport:cert`

No decision without data. A rule resolves only when its data was read in this round:

fleet unreadable (tctl error): proxy alerts neither fire nor resolve;
one failed console poll: nothing changes (the second consecutive failure fires console_down);
console unreachable: its audit / DC / logins alerts keep their state until a successful poll;
console or proxy removed: its alerts resolve.

2. Deduplication¶

Each condition fires once (a fired history entry and one notification) and resolves once (resolved entry and one notification), however many rounds it lasts. Values that change while the alert is active (minutes, counts) are refreshed on the page without a new notification. The active set, the history (200 entries) and the last evaluation date are stored in /data/alerts-state.json (atomic writes, see Clients and consoles): a restart neither re-sends the active alerts nor forgets them.

3. Channels¶

Up to 20 channels; each can be disabled without being deleted. All events of one round are batched into one request per channel (a large outage sends one message, not hundreds; text messages list 30 events at most, then "… and N more").

Type	Settings	Request
Generic webhook	HTTPS URL (a query string is allowed), optional HMAC secret (8 to 512 characters)	`POST` JSON (below)
ntfy	Server URL (`https://ntfy.sh` or self-hosted), topic `[A-Za-z0-9_-]{1,64}`, optional access token	`POST <server>/<topic>`, text body, headers `Title`, `Priority` (`urgent` for a critical alert), `Tags`, `Authorization: Bearer <token>`
Microsoft Teams	Incoming webhook URL (Workflows "Post to a channel when a webhook request is received", or a legacy connector)	`POST` message with an Adaptive Card 1.4
Slack	Incoming webhook URL	`POST {"text": …}`

Secrets are write-only. The URL (Teams/Slack/webhook URLs embed their key), the HMAC secret and the ntfy token are stored encrypted (AES-256-GCM, SECRETS_ENCRYPTION_KEY), never returned by the API (only the URL origin is shown, e.g. https://hooks.slack.com, and "secret set"), never logged or audited. On edit, leave the URL empty to keep it, leave the secret empty to keep it, or tick "remove the secret". Changing the type requires the URL again.

3.1 Webhook payload and signature¶

{
  "schema": 1,
  "source": "tenantry",
  "time": "2026-10-18T10:00:00.000Z",
  "events": [
    {
      "event": "fired",
      "text": "[CRITICAL] Steward console down | client acme | console.acme.local | error: timeout",
      "alert": {
        "key": "console:4f6c…:down", "kind": "console_down", "severity": "critical", "clientCode": "acme",
        "subject": "console.acme.local", "params": { "error": "timeout" }, "link": "/clients/acme",
        "firedAt": "2026-10-18T10:00:00.000Z"
      }
    }
  ]
}

event is fired, resolved (the alert then carries resolvedAt) or test (alert: null). With a secret, two headers are added:

X-Tenantry-Timestamp: 1792317600
X-Tenantry-Signature: sha256=<hex HMAC-SHA256(secret, "<timestamp>.<raw body>")>

The receiver recomputes the HMAC over the exact raw body, compares in constant time and refuses a timestamp older than 5 minutes (replay).

3.2 Send test¶

Send test (administrators, 10 per minute) sends one test notification to the channel and shows the result; it is also recorded in the history (test) and the audit (alerts.channel.test).

4. Delivery and errors¶

Same outbound rules as console polling: https only, TLS verified (system store + NODE_EXTRA_CA_CERTS), no redirects, 10 s timeout, response read up to 256 KiB and discarded. A 2xx answer is a success. Each history entry shows the delivery result per channel:

Error	Meaning
`http_error` (+ status)	The receiver answered non-2xx (401/403: revoked webhook or wrong ntfy token; 404: deleted webhook; 429: rate limit).
`redirect_refused`	3xx: use the final URL.
`tls_error`, `dns_error`, `connect_error`, `timeout`	Network path from Tenantry container to the receiver (egress proxy, firewall, DNS).
`secret_unreadable`	The stored URL or secret cannot be decrypted (key changed): edit the channel and enter them again.

A failed delivery is not retried (the next transition is notified normally); watch the history, or add a second channel.

5. Audit¶

alerts.settings (threshold), alerts.channel.create, alerts.channel.update (detail: enabled=…, type=…, url=replaced, secret=replaced|cleared), alerts.channel.delete, alerts.channel.test (outcome error with the error code on failure). Target = type + channel name; the URL origin appears in the creation detail, never the full URL.

6. API¶

GET /api/alerts (active + history), GET /api/alerts/settings, PUT /api/alerts/settings { failedLoginsThreshold } (1 to 100,000), POST /api/alerts/channels, PUT /api/alerts/channels/:id, DELETE /api/alerts/channels/:id, POST /api/alerts/channels/:id/test.