HTTPS certificate¶

Goal: no client ever sees a certificate error, and nobody has to push a CA by GPO. The default path is a publicly trusted certificate (Let's Encrypt by default), obtained and renewed automatically, also for consoles that are only reachable on the client's internal network.

Steward serves the certificate itself (no reverse proxy needed) and reloads it without a restart: when the files change, the new certificate is validated and served to new connections within seconds. A file that is not valid (key that does not match, expired, half-written) is refused and the previous certificate keeps being served.

Choose a method¶

The guided install (sudo deploy/install.sh, step 5/6) asks these questions and builds the options below for you.

flowchart TD
  q0{"Self-signed is fine?<br/>(lab, demo only)"} -->|yes| ss["--tls selfsigned"]
  q0 -->|no| q1{"Certificate files<br/>already issued?"}
  q1 -->|yes| files["--tls files"]
  q1 -->|no| q2{"Own ACME CA?"}
  q2 -->|yes| own["--acme-server &lt;URL&gt;<br/>+ HTTP-01 or DNS-01"]
  q2 -->|"no: public CA"| q3{"Port 80 of the name<br/>reachable from the Internet?"}
  q3 -->|yes| http["--tls acme-http"]
  q3 -->|"no (usual)"| dns["--tls acme-dns<br/>--dns-provider ..."]

Situation	Method	`install.sh`
Console reachable only internally (the usual case)	ACME DNS-01: the CA checks a TXT record created through your DNS provider's API. Nothing has to be reachable from the Internet.	`--tls acme-dns --dns-provider <name> --dns-credentials-file <file>`
Console reachable from the Internet on port 80	ACME HTTP-01: the CA fetches a token from Steward on port 80.	`--tls acme-http`
Client has its own ACME CA (step-ca, AD CS with an ACME front end, ...)	ACME against that CA (DNS-01 or HTTP-01), with its root. Clients must already trust that root (they usually do: it is their PKI).	`--acme-server https://<directory URL> --acme-ca-bundle <root.pem>`
A certificate you already have (bought, issued by the client's PKI)	Files. Replace them before they expire; the console picks the new ones up.	`--tls files --cert <fullchain.pem> --key <key.pem>`
Lab, demo	Self-signed (every browser shows an error).	`--tls selfsigned`

Behind Cloudflare Tunnel no port is published, so HTTP-01 is impossible: use DNS-01 or your own files (Exposure and access). None of the providers below is a default: use the one that hosts the client's public zone.

Requirements for a public CA: the name users open (--hostname) must be under a public domain the client (or you) controls, for example steward.client-example.ch. It may resolve only on the internal DNS: DNS-01 only needs the public zone to hold the temporary _acme-challenge TXT record. Names such as steward.corp.local cannot get a public certificate: add a name under a real domain (an internal DNS record pointing to the console is enough).

Certificate Transparency

Every publicly trusted certificate is published in the public CT logs, so the host name becomes public (not its address). If that matters, use --wildcard: the certificate is issued for *.client-example.ch and the console's own name is not published. A wildcard needs DNS-01.

How it works¶

  acme container (acme.sh, pinned)                       steward container
  ─────────────────────────────────                      ─────────────────────────────
  issue / renew (twice a day: acme.sh --cron)            reads /certs (read-only volume)
     │ DNS-01: provider API (token = Docker secret)      watches the files (debounced),
     │ HTTP-01: token in the shared webroot  ───────────▶ serves /.well-known/acme-challenge/ on :80
     ▼                                                    │
  checks key ↔ certificate, then atomic rename into  ───▶ validates, then serves the new certificate
  steward-certs volume: key.pem, fullchain.pem (0440)     (no restart, no Docker socket, no reload command)

Image: neilpang/acme.sh, pinned by version and digest in deploy/compose.acme.yml; its self-update is turned off.
Key: ECDSA P-256 (ACME_KEY_TYPE=ec-256). Renewal 30 days before expiry (60 days into a 90-day certificate).
The sidecar runs read-only, without capabilities except CHOWN (to hand key.pem to Steward's user), on its own network.
The DNS API token lives in <instance>/secrets/acme-credentials (root only, mode 0400), mounted as a Docker secret in the acme container only. It never appears in the compose files, in the environment of the steward container, or in the output of install.sh. acme.sh keeps a copy in its own volume (steward-acme, root only) to renew.

Install¶

# DNS-01, Cloudflare (credentials file: NAME=value lines, see "Providers")
sudo install -m 600 /dev/null /root/cf.env && sudo vi /root/cf.env
sudo deploy/install.sh --name acme --hostname steward.acme-client.ch --version 1.0.0 \
     --dns-provider cloudflare --dns-credentials-file /root/cf.env --acme-email ops@msp.example
sudo shred -u /root/cf.env      # a copy is in /opt/steward/acme/secrets/acme-credentials (0400)

--dns-provider implies --tls acme-dns. The script then:

checks the options and that the credentials file holds the variables the provider needs (names only; values are never read or printed);
obtains the certificate before Steward starts, checks it (key matches, chain verifies against the public roots, not about to expire) and prints its subject, issuer and expiry;
starts Steward with it, and the acme container for renewals;
connects to https://<hostname>:<port>/ the way a browser would (public roots, real name) and fails if that does not verify.

When the first issuance fails, Steward is not started and the script says why (see Troubleshooting).

Other options (all kept across re-runs):

Option	Meaning
`--acme-email <e-mail>`	ACME account contact (expiry warnings from the CA). Recommended.
`--acme-server <name or URL>`	`letsencrypt` (default), `letsencrypt_test` (staging: untrusted certificates, generous limits, for rehearsals), `zerossl`, `google`, `buypass`, `sslcom`, or an ACME directory URL (internal CA).
`--acme-eab-kid <kid> --acme-eab-hmac-file <file>`	External Account Binding (ZeroSSL, Google, some internal CAs). The HMAC key goes into the secret file.
`--acme-ca-bundle <pem>`	Root(s) of an internal ACME CA: used to talk to it and to check the issued chain.
`--wildcard`	Certificate for `*.<parent domain>` (DNS-01 only).
`--acme-dns-sleep <seconds>`	Wait this long after creating the TXT record instead of acme.sh's own propagation check (slow providers, split DNS).
`--acme-challenge-alias <zone>`	`_acme-challenge.<host>` is a CNAME to `_acme-challenge.<host>.<zone>`: the token only needs rights on that zone (see below).

HTTP-01: sudo deploy/install.sh --name acme --hostname steward.acme-client.ch --tls acme-http --acme-email ops@msp.example. Port 80 of the host must be reachable from the Internet under that name; Steward answers the challenge itself (port 80 stays Steward's, it redirects everything else to HTTPS).

Your own files: --tls files --cert fullchain.pem --key key.pem. fullchain.pem holds the server certificate first, then the intermediate certificate(s); the key is an unencrypted PEM (PKCS#8, PKCS#1 or EC). They are copied to <instance>/tls/ (readable only by the container user). To renew, re-run the script with the new files, or replace them in place atomically (write next to them, then mv): the console serves them within seconds.

Without docker compose's help from install.sh, the same thing is deploy/compose.yml plus the override deploy/compose.acme.yml (or compose.tls-files.yml), and Steward's options Tls:CertificatePemPath/Tls:KeyPemPath (PEM) or Tls:CertificatePath/Tls:CertificatePasswordFile (PFX), which are hot-reloaded the same way.

Providers¶

Put the variables in the credentials file, one NAME=value per line (# comments allowed, nothing else). Always create a dedicated token, limited to the zone that holds the console's name, with DNS rights only: whoever holds it can obtain certificates for that zone.

Provider	`--dns-provider`	Variables	Least-privilege token
Infomaniak	`infomaniak`	`INFOMANIAK_API_TOKEN`	Manager › API tokens: a token with the `domain` scope only, created by a user who manages only this domain if possible (Infomaniak tokens cannot be limited to one zone).
OVHcloud	`ovh`	`OVH_END_POINT` (`ovh-eu`, `ovh-ca`, `ovh-us`), `OVH_AK`, `OVH_AS`, `OVH_CK`	Create the application key and its consumer key at `https://eu.api.ovh.com/createToken/` (`ca.`/`api.us.` for other regions) with only: `GET /domain/zone/<zone>`, `GET /domain/zone/<zone>/`, `POST /domain/zone/<zone>/`, `DELETE /domain/zone/<zone>/*`, and no expiry or a long one you track.
Cloudflare	`cloudflare` (`cf`)	`CF_Token`, `CF_Zone_ID`	My Profile › API Tokens › Create custom token: permission Zone › DNS › Edit, zone resources Include › Specific zone › . With `CF_Zone_ID` set, no account-wide read is needed. Optional: client IP filtering to the host's egress address. Never the Global API Key (`CF_Key`/`CF_Email`).
Gandi	`gandi_livedns` (`gandi`)	`GANDI_LIVEDNS_TOKEN`	Personal Access Token (Organizations › Sharing › Personal access tokens): permission "Manage domain name technical configurations", restricted to this domain, with an expiry you track. The old `GANDI_LIVEDNS_KEY` API key is deprecated.
Hetzner	`hetznercloud` (`hetzner`)	`HETZNER_TOKEN`	Hetzner Console (DNS now in Cloud projects): an API token Read & Write of a project that holds only this zone (tokens are scoped per project).
AWS Route 53	`aws` (`route53`)	`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`	Dedicated IAM user with the policy below (only TXT records named `_acme-challenge.*` of one hosted zone).
Azure DNS	`azure`	`AZUREDNS_SUBSCRIPTIONID`, `AZUREDNS_TENANTID`, `AZUREDNS_APPID`, `AZUREDNS_CLIENTSECRET` (or `AZUREDNS_MANAGEDIDENTITY=true`)	App registration (service principal) with the role DNS Zone Contributor assigned on the DNS zone resource only (not the resource group or subscription). Secret with an expiry you track.
deSEC	`desec`	`DEDYN_TOKEN`	Token with a scoping policy: domain = this domain, subname `_acme-challenge…`, type `TXT`, write allowed; default policy denies.
Others	`dns_<name>`	see the acme.sh DNS API list	Same principle.

Route 53 policy (replace ZONEID):

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": ["route53:ListHostedZones", "route53:ListHostedZonesByName", "route53:GetChange"], "Resource": "*" },
    { "Effect": "Allow", "Action": "route53:ListResourceRecordSets", "Resource": "arn:aws:route53:::hostedzone/ZONEID" },
    { "Effect": "Allow", "Action": "route53:ChangeResourceRecordSets", "Resource": "arn:aws:route53:::hostedzone/ZONEID",
      "Condition": {
        "ForAllValues:StringLike": { "route53:ChangeResourceRecordSetsNormalizedRecordNames": ["_acme-challenge.*"] },
        "ForAllValues:StringEquals": { "route53:ChangeResourceRecordSetsRecordTypes": ["TXT"] } } }
  ]
}

Narrowest option: CNAME delegation. Create once, in the client's zone, _acme-challenge.steward.client.ch CNAME _acme-challenge.steward.client.ch.acme-validation.msp.example. and give the sidecar a token for the small zone acme-validation.msp.example only (--acme-challenge-alias acme-validation.msp.example). The token then cannot change any record of the client's own zone.

Example credentials file (Cloudflare):

# Cloudflare token: Zone.DNS:Edit on client.ch only
CF_Token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
CF_Zone_ID=0123456789abcdef0123456789abcdef

Example: a client zone hosted at Infomaniak

This is one worked example, not a recommendation of a provider: the steps are the same for the others, with the variables of the table above.

In the Infomaniak Manager, as a user who manages the client's domain (ideally only that one): API tokens > Create a token, scope domain only, a name such as steward-acme-dns01, and an expiry you track.

Guided install: step 5/6, answer 1 (public CA), n (port 80 not reachable), provider 1) Infomaniak, paste the token (hidden). Scripted:

sudo install -m 600 /dev/null /root/infomaniak.env
sudo vi /root/infomaniak.env      # one line: INFOMANIAK_API_TOKEN=<token> (an editor keeps it out of the shell history)
sudo deploy/install.sh --name acme --hostname steward.client-example.ch --version 1.0.0 \
     --dns-provider infomaniak --dns-credentials-file /root/infomaniak.env --acme-email ops@msp.example
sudo shred -u /root/infomaniak.env

The installer creates _acme-challenge.steward.client-example.ch through the API, the CA checks it, the certificate is published, and the record is removed. Renewals repeat this every 60 days, unattended.

To rotate a token: re-run install.sh with the new --dns-credentials-file (the secret is rewritten and acme.sh re-saves it), then revoke the old token.

Monitoring renewals¶

Diagnostics (Configuration › Diagnostics) show the served certificate: subject, names, issuer, expiry; a warning under 21 days (Tls:ExpiryWarningDays), an error once expired, an error for a self-signed certificate in production, an incomplete chain (missing intermediate), a CA this host does not trust, every --hostname the certificate does not cover, and a refused renewal (with the reason).
Notifications (Configuration › Notifications): HTTPS certificate expiring (warning, critical under 7 days) and HTTPS certificate renewal failed (a renewed file was refused, or the configured certificate could not be loaded at start), at most once a day each. Subscribe the MSP channel to both.
Since an ACME certificate is renewed 30 days before expiry, the 21-day warning always means renewal stopped working, with three weeks left to fix it.
Logs: docker logs steward-<name>-acme-1 (every renewal run), and the console logs HTTPS certificate served (...) for each certificate it starts serving.

Manual check from any machine: openssl s_client -connect steward.client.ch:443 -servername steward.client.ch </dev/null | openssl x509 -noout -subject -issuer -enddate.

Force a renewal (e.g. after changing the name): re-run install.sh (it re-issues when the names changed), or docker exec steward-<name>-acme-1 acme.sh --home /acmebin --config-home /acme.sh --renew --force --ecc -d <hostname>.

Troubleshooting¶

install.sh prints the last lines of acme.sh and the likely cause:

Message	Cause and fix
DNS API refused	Wrong or revoked token, wrong variable name, token not allowed on the zone holding the name, or the zone is not at that provider. Check the table above; test the token with the provider's API.
DNS propagation	The CA did not see the TXT record. The zone edited must be the public one (split DNS: the internal zone does not count); secondary name servers can be slow (`--acme-dns-sleep 300`); a CNAME on `_acme-challenge` needs `--acme-challenge-alias`.
Rate limit	Let's Encrypt allows 5 identical certificates per week and 5 failed validations per hour and name. Wait for the time given, and rehearse with `--acme-server letsencrypt_test`.
CAA	A CAA record of the domain does not allow this CA: add `0 issue "letsencrypt.org"` (or the CA you use).
External Account Binding	ZeroSSL/Google need `--acme-eab-kid` and `--acme-eab-hmac-file` from their console.
HTTP-01	The CA could not fetch the token on port 80: DNS name, firewall, NAT. For internal hosts use DNS-01.
cannot reach the ACME server	Outbound HTTPS from the host (proxy, firewall) to the CA and to the provider's API.
Diagnostics: self-signed, configured certificate could not be loaded	`/certs` is empty or unreadable: `docker logs steward-<name>-acme-1`. The console serves its lab certificate until the file appears, and switches by itself.
Diagnostics: renewed certificate refused	The new file was invalid (reason shown: `key_mismatch`, `expired`, `unreadable`...). The previous certificate is still served.
Diagnostics: incomplete chain	The file has the server certificate only: use the full chain (`fullchain.pem`).

Updating the ACME image¶

The tag and digest are in deploy/compose.acme.yml. To move to a newer acme.sh release, take a tag at least two weeks old, read its digest (docker buildx imagetools inspect neilpang/acme.sh:<tag>), update both, and re-run install.sh. To use a mirror, set STEWARD_ACME_IMAGE=<reference> when running install.sh.