Operating an instance

Day-to-day running of one client console: health, logs, audit, backups, upgrades, recovery and the directory connection.

The examples assume the instance was installed with deploy/install.sh --name acme in the default directory:

Thing	Name
Instance directory	/opt/steward/acme (compose.yml, .env, config/, secrets/)
Compose project	steward-acme
Console container	steward-acme-steward-1
Password service (helpdesk only)	steward-acme-steward-pwd-1
ACME sidecar (ACME certificates only)	steward-acme-acme-1
Cloudflare Tunnel sidecar (tunnel exposure only)	steward-acme-cloudflared-1

Related pages: Notifications (alerts to Teams, Slack, e-mail, webhooks), Configuration baselines (copy a reference setup to another client), Fleet API (status for the MSP control plane), Troubleshooting.

Health and logs

• docker inspect -f '{{.State.Health.Status}}' steward-acme-steward-1 prints healthy when everything is fine. With the helpdesk, check steward-acme-steward-pwd-1 too (it answers GET /health on the internal network only).
• GET /health/live says whether the process is up; GET /health/ready also checks the database. Both are anonymous and expose no detail.
• Configuration › Diagnostics checks everything the console depends on (certificate, TLS and bind per DC, OUs, groups, AD rights, delegation export). Run it after every change.
• Logs go to stdout, one JSON object per line: docker logs steward-acme-steward-1. They never contain passwords, tokens or secrets. The password service logs one line per reset, unlock, disable or enable (docker logs steward-acme-steward-pwd-1): collect it together with the console's logs.
• After an upgrade, read the startup log: a setting that no longer does anything is reported by name as a WARNING.
• The API is described in OpenAPI: see API (OpenAPI).

Audit log

Every action is recorded, including refused ones: sign-ins, authorization decisions, attribute changes with old and new values (except sensitive attributes), role and operator changes, protection-list changes, exports and purges.

• Viewer: the Audit page (permission audit.read), with filters and Show more. Its filters live in the URL (/audit?result=denied&from=…), so a filtered view can be bookmarked or shared; the dashboard's 24-hour counters open it pre-filtered.
• Per user: with audit.read, the History tab of a profile lists what happened to that user. CSV exports are audited too (users.export, users.watch.export, with their row count).
• Deleted operators: deleting an operator removes the console record only; the audit rows stay, and the operators.deleted row records the name, provider, subject and roles it had.
• Tamper evidence: each row includes the hash of the previous one. The Audit page shows whether the chain verifies, and so does GET /api/audit/verify.
• Retention: kept forever by default. Audit:RetentionDays (0 = keep everything, otherwise at least 30) purges older rows daily. The chain keeps verifying from an anchor, a purge is refused if the chain is broken, and each purge is audited.

Forwarding the audit to a SIEM

Every audit row is also written to stdout under the logger category Audit. Ship the container output to your SIEM or syslog with your usual collector (Docker syslog or gelf log driver, Fluent Bit, Vector...). Do it on every production instance: root on the host can rewrite the SQLite file, and the forwarded copy is what makes such tampering detectable and recoverable.

Backup and restore

What a backup must contain:

1. The console volumes (database and /secrets): deploy/backup.sh takes them. The database is useless without /secrets, which holds the master key that encrypts the technical accounts' passwords.
2. The instance directory on the host: /opt/steward/acme/.env, config/ and secrets/ (helpdesk link key and reset password, LAPS password, ACME DNS credentials, Cloudflare Tunnel token). Back it up with your host backup, as root.

Both are as sensitive as the secrets they hold: store them encrypted, with restricted access.

# Online, consistent copy of the database + /secrets + the LDAP trust store. The archive is created with mode 0600.
deploy/backup.sh steward-acme-steward-1 /srv/backups
#   -> /srv/backups/steward-acme-steward-1-<UTC time>.tgz  (+ a .db-path file next to it: keep both together)

# Restore into the same (or a freshly installed) instance:
deploy/restore.sh steward-acme-steward-1 /srv/backups/steward-acme-steward-1-<UTC time>.tgz

backup.sh asks the console for a consistent copy of its database (dotnet Steward.Api.dll backup), then archives it with /secrets through a throwaway alpine container without network access (it pulls alpine:3.22, pinned by digest, once; STEWARD_ALPINE_IMAGE=<mirror>@sha256:<digest> points both scripts at a local mirror). Schedule it with cron or a systemd timer, and copy the archives off the host.

restore.sh first checks the archive: its .db-path companion must name exactly one file data/backups/<name>.db that the archive contains, otherwise it refuses (exit code 2) before stopping anything. It then stops the container, keeps the current state aside (/data/steward.db.before-restore-<time> and /secrets/.before-restore-<time>/), restores, starts the container and waits until it is healthy.

Restoring on a new host: install the instance with the same --name and options (or copy the instance directory back), then run restore.sh. Practise a restore on a test host before you need one.

Lost helpdesk files: if the link key is lost, delete secrets/pwd-link and re-run install.sh --helpdesk: a new key is generated for both containers. The reset password can always be set again (change it in AD, re-run with --reset-password-file). The ACME certificate needs no backup: the sidecar obtains a new one.

Upgrade

deploy/backup.sh steward-acme-steward-1 /srv/backups
sudo deploy/install.sh --name acme --hostname steward.client-example.ch --version 1.1.0

Re-running install.sh with a newer --version (or --image <registry>/steward:<version>) upgrades the instance. Everything else is kept from the previous run (TLS choice, exposure, helpdesk, DC mappings, stored secrets), so the name, the hostname and the new version are enough. The guided install works too: run sudo deploy/install.sh in a terminal, type the same instance name, and press Enter to keep each current value. Add --dry-run to see what would be applied without changing anything.

Pending database migrations run at start, after an automatic copy of the database to /data/backups/. To run them by hand instead, set Database:AutoMigrate=false and run docker exec steward-acme-steward-1 dotnet Steward.Api.dll migrate. Check the image signature of every new release first (Verify the image), and read the startup log afterwards.

Upgrade notes

• Teleport step-up. Teleport operators are not asked to confirm sensitive changes: their Teleport session is the step-up (Teleport enforced its second factor at sign-in; each such change is audited stepup:teleport). The former keys Authentication:Teleport:TrustSessionMfa and Authentication:Teleport:StepUpMaxAge are ignored and logged as obsolete at start. To forbid Teleport operators any step-up-gated change, set Authentication:Teleport:StepUp: Deny (environment STEWARD_Authentication__Teleport__StepUp=Deny; default Session), then remove the obsolete keys. See Teleport sign-in.
• Super administrator. Every instance has a built-in Super administrator role, held by nobody until you assign it (see Super administrator and step-up mode). A custom role already named "Super administrator" was renamed "... (custom)".
• New delegable actions (moves, renames, groups, LAPS...) need the AD rights of the write account: after an upgrade, run Configuration › Diagnostics; when the delegation export is reported out of date, export it again and have a domain administrator apply it.
• Authenticator required at sign-in for operators with a role (step-up mode authenticator, the default). After this upgrade, the session of every AD operator who holds a role ends at its next request unless it was opened with an authenticator code; at their next sign-in, operators without an authenticator enrol one inside the sign-in (password, then they scan a key and type a first code; no session before that). Tell operators beforehand, and check Configuration › Diagnostics (signin_enrolment_pending: how many still have to enrol). In modes password / none nothing is forced, but diagnostics warn (signin_password_only). An administrator can now reset another operator's authenticator only when they hold all of that operator's console rights (a Super administrator's only by another Super administrator).
• Teleport operators' AD accounts are resolved from the JWT at every sign-in (Authentication:Teleport:AdAccountClaims, default username,trait:windows_logins) and protected like an AD operator's own account. Trait mappings whose value is an AD group DN or SID make that group a protected group (saved again after the upgrade: existing mappings are resolved only when the mapping set is saved, so open Delegations › Teleport and save once).
• Password service pins: the domain. steward-pwd now also pins the AD domain (STEWARD_PWD_DOMAIN) and only works over LDAPS port 636. Until you re-pin it refuses every operation (trust_not_pinned): run deploy/install.sh --name <instance> --pin-password-service after the upgrade.
• Delegation script. Grant-StewardDelegation.ps1 now requires -ManagedOu and -Identity (they must match the exported model), prints the plan and asks for confirmation, and refuses OUs holding privileged objects; see Delegation.

Recovery

All commands run inside the console container: docker exec <container> dotnet Steward.Api.dll <command>.

Situation	What to do
"Teleport only" is on and Teleport is unusable (nobody can sign in)	allow-password-login --confirm switches the setting off: AD password sign-in works again within 5 seconds, if the host configuration allows it (Authentication:ActiveDirectory:Enabled). Audited (settings.teleport_only, off:recovery). See Teleport sign-in.
An AD operator lost their authenticator app and no administrator can reset it	reset-totp <operator> (operator id, AD objectGUID, or a unique display name) clears it, audited operators.totp_reset (reason recovery). The operator enrols a new app on the Security page, confirming their AD password once. An administrator can also do it in Delegations › Operators (with their own step-up).
The first administrator must confirm a sensitive change right after setup	Expected: nothing is enrolled yet. The step-up dialog links to Security: type the AD password once, scan the QR code, activate with a first code (it also counts as the step-up), then retry.
Setup wizard locked (five wrong tokens), or no administrator left	reset-setup --confirm re-opens the wizard and prints a new token. Data is kept; the action is audited.
Define many roles at once (automation)	docker exec -i <container> sh -c 'cat > /tmp/r.json' < roles.json, then import-roles /tmp/r.json [--replace]. Same validation as the UI.
Consistent copy of the database	backup (prints the path of the copy under /data/backups/; deploy/backup.sh uses it).
Run pending migrations by hand	migrate (with Database:AutoMigrate=false).
Rotate the password-reset account's password	Change it in AD, then install.sh --name <name> --hostname <host> --reset-password-file <file>. The password service is recreated with it. See Helpdesk.
Rotate the helpdesk link key	Delete <instance>/secrets/pwd-link, then re-run install.sh ... --helpdesk. Both containers are recreated with the new key.
Rotate the DNS API token or the tunnel token	Re-run install.sh with the new --dns-credentials-file or --tunnel-token-file, then revoke the old one.

Directory connection after setup

Configuration › Directory edits what the setup wizard configured. Every change needs settings.manage and a step-up, is tested against the directory with the NEW settings before anything is stored, is saved in one transaction, and is audited (settings.directory.*, old and new values; fingerprints and UPNs only, never a password or a key). The console reloads its connection at once; the password service follows within 3 seconds. The domain and NetBIOS names cannot change: another domain is another instance.

Change	What is checked before saving
Add, remove or reorder Domain controllers	Each added DC: TLS handshake with the configured CA and host name, then a bind of the read account on that DC. A configured DC that does not answer now is kept and reported (maintenance); at least one DC of the new list must work. The order is the failover order.
LDAPS port	Always 636. Plain LDAP (389, 3268) is refused, and so is 3269 (Global Catalog over TLS: read-only, partial attribute set).
Replace the Certificate authority	Every configured DC must present a certificate that chains to the NEW CA and accept a bind through it. A PEM containing a private key is refused. During a CA migration, use a PEM containing both CAs, then remove the old one once every DC certificate was re-issued.
UPN suffixes	Syntax only. Removing a suffix used by onboarding templates needs an explicit confirmation (the list is shown).
Technical accounts (password rotation)	The new password is bound on every DC (refused if any reachable DC rejects it), stored encrypted, never shown again. Accounts provided as files (Docker secrets) are changed on the host. The password-reset account is always a file (--reset-password-file).
Managed OUs	Each OU (DN or GUID) is resolved in AD; it must be an organizational unit strictly below the domain root. Removing an OU still used by delegations or onboarding templates needs a confirmation after the impact list is shown.

Rotating a technical account password (read or write account): reset the password in AD; wait for replication to every DC (otherwise the DC that still has the old one refuses the save); then Configuration › Directory › Technical accounts, type the new password, Test and save. Until then the console keeps using the old password: sign-ins fail ("Active Directory cannot be reached right now") and diagnostics show the technical account's sign-in as refused.

High availability across domain controllers

• Configure at least two DCs. Otherwise diagnostics warn ("Only one domain controller configured") and the dashboard's Directory redundancy card says so.
• DCs are tried in the configured order. A DC whose connection fails (unreachable, TLS refused, busy, timeout) is tried last for 60 seconds (Directory:FailedDcCooldown), so a dead DC costs one connect timeout (Directory:ConnectTimeout) per minute, not one per request. When the cooldown ends, one request retries it; success puts it back in its place.
• A background probe (Directory:HealthProbeInterval, 30 seconds, 0 = off) tests every DC nothing has contacted for that long (TLS handshake and read-account bind), so a recovered DC comes back without a user request paying for the retry. Without it, such a DC is shown "Not tested recently".
• A wrong password never fails over: every DC would say the same. Without a read account, only TLS is tested and the health view says so.
• Failover happens only when a connection is opened. A change that was sent and lost is never replayed on another DC: it is checked on the DC it was sent to, or reported as an unknown outcome.
• Per-DC health: Configuration › Directory › Controller status (last success and failure, latency, the DC tried first, Test each controller), the dashboard health card, and diagnostics.
• Discover via DNS asks _ldap._tcp.dc._msdcs.<domain> (host resolvers, or Directory:DiscoveryDnsServer) and proposes the DCs it finds; nothing is added until an administrator adds and saves them.
• Each DC name must resolve to that DC only. If the containers cannot resolve the DC names, map them with install.sh --add-host <fqdn>:<ip> (kept across re-runs).

Adding a second DC

1. On the new DC: LDAPS with a certificate whose SAN contains its FQDN, issued by the CA configured in the console (Configuration › Directory › Certificate authority). The AD delegation is stored on domain objects: nothing to do per DC.
2. From the console host, the name must resolve and TCP 636 must be reachable.
3. Configuration › Directory › Domain controllers: type the FQDN (or Discover via DNS), order the list (first = tried first), Test and save, confirm your identity.
4. The result shows TLS and bind per DC; Test each controller can be run at any time.
5. Run Configuration › Diagnostics: the redundancy line must be green.

Super administrator and step-up mode

Super administrator (Delegations): every right on every managed OU and its sub-OUs, computed by the server (new OUs, attributes and actions are covered automatically); read-only, cannot be deleted. Privileged / Tier 0 accounts and groups stay refused. To give it to someone:

1. Delegations › Operators › the operator › Roles › tick "Super administrator" › Save, and confirm your identity. Audited super_administrator:granted (and super_administrator:revoked). The setup wizard's first-administrator step offers it as an opt-in (unchecked by default).
2. Configuration › Diagnostics › Export AD delegation, and have a domain administrator apply the file (see the delegation guide). Until it is applied, AD refuses the writes the service accounts lack.

Risk: the holder (or anyone at their unlocked session) can change, reset, disable and create every standard account and group of the managed OUs, VIP included. Keep it for one or two named administrators; use narrow roles for daily helpdesk work.

Step-up mode (Configuration › Sign-in › Step-up mode, settings.manage): what AD operators give before a sensitive change. authenticator (default, recommended: a code from their authenticator app), password (their AD password again; weaker) or none (no confirmation; not recommended: shown in red, diagnostics warn). Teleport operators are not affected. Changing it needs the current mode's step-up and is audited (settings.step_up_mode). See Security model.

Moves, renames and groups

Delegable actions that need no password service (they work with the helpdesk off). The Super administrator covers them.

Action	What it allows	The delegation must also hold
users.move	Move a user to another OU. The source and destination OUs must both be in a users.move scope; a VIP account needs vip.manage with users.move at the destination too	users.read
users.rename	Change the relative name (cn), display name, first and last name	users.read
users.rename.login	Also change the sign-in names (sAMAccountName, UPN) during a rename; asks for a step-up	users.rename
groups.create	Create a group in an OU of the scope	groups.read
groups.attributes.write	Edit a group's description, manager (managedBy), mail and notes (info)	groups.read
groups.delete	Delete a standard group (with members: only after seeing and confirming them); step-up	groups.read
groups.owned.members.write	Change the members of the groups whose manager is the operator's own AD account ("the groups I own")	groups.read

Never touched, whatever the role: privileged, protected (adminCount=1), unclassifiable, built-in and well-known accounts and groups, critical system objects, and anything in or moved to a Tier 0/1 OU or a container that is not an OU (such as CN=Users). Nobody moves or renames their own account.

In the UI: the profile header has Move (only allowed destinations are listed; a name already used there gets a suggestion such as "Jane Doe (2)") and Rename (live preview, conflicts checked domain-wide); the Groups page has New group; a group page has Edit and Delete.

AD rights for the write account, only where a role needs them (the delegation export generates them):

• users.move: Create user objects on the destination OUs, Delete inherited by user objects (source), and Write property on cn and name (Windows refuses a cross-OU move without them).
• users.rename: Write property on cn, name, displayName, givenName, sn; users.rename.login adds sAMAccountName and userPrincipalName.
• groups.create: Create group objects; groups.delete: Delete inherited by group objects; groups.attributes.write: Write property on description, managedBy, mail, info; groups.owned.members.write: Write property on member.

Delete is always granted on the objects, never as Delete child on the OU: AdminSDHolder-protected objects do not inherit it, so a stolen write account cannot delete or move them.

Instance title

Configuration › Instance identification (settings.manage) sets a title (plain text, 1 to 60 characters) and an accent colour. Operators see it in the top bar and as a prefix of the browser tab title, so a technician working for several clients knows which console this is. Anonymous visitors see it on the sign-in page only if Show on the sign-in page is ticked (off by default: the client name is not disclosed before sign-in). Every change is audited (settings.branding).

User search: paging and limits

The Users page shows 50 users per page; Show more loads the next 50, and the header shows how many users the operator may see in total.

• The first page enumerates the matching accounts with LDAP paged results (Directory:PageSize entries per LDAP page, default 500), up to Directory:SearchMaxResults (default 5000). Beyond that the list is marked incomplete ("refine your search").
• Rows are ordered by the console on what the operator may see (chosen column, then display name, then objectGUID), so pages never overlap.
• The enumeration is held in memory for 10 minutes per operator; the next page uses an opaque cursor encrypted and signed by the console, valid for that operator only. No LDAP state reaches the browser.
• Scope is decided per row, on every page: rows the operator may not read are dropped before ordering and counting, so neither the rows nor their number leak.
• Directory:MaxResults (default 200) bounds the other lists (group members, watch lists, pickers).

Lab

The Samba lab has a second DC: lab/ha-up.sh starts dc2.lab.steward.test (lab/ha-up.sh --status shows replication), and tests/ha/dc-failover.sh proves the failover end to end. Lab tools never target a production domain controller.