NATS certificates: generation, TLS, layout, and container deployment

All NATS material lives under build/keys/nats/ (gitignored; carries private keys):

build/keys/nats/
├── ca.key  ca.crt            # internal CA: EC P-256, self-signed, 365 days
├── nats-server.key  nats-server.crt   # broker leaf: EC P-256, 90 days, SANs
└── <service>.key  <service>.crt       # one mTLS client leaf per service
     # ores.iam.service, ores.refdata.service, ..., ores.shell,
     # ores.compute.wrapper, ores.analytics.service  (19 services)

The service list is the _SERVICES constant in projects/ores.compass/src/nats_certs.py. It is still hand-maintained (it mirrors the service registry but is not generated from it); adding a service means adding its name there and re-running compass nats certs. The Anatomy of a Service doc flags driving certs from the registry as a known automation gap.

The generator is nats_certs.py, exposed standalone as compass nats certs and called automatically by compass env configure (compass env configure → env_init.py invokes nats_certs.generate() before writing .env). Requires openssl in PATH.

Settings in effect (all from nats_certs.py):

• Curve choice: openssl ecparam -name prime256v1 -genkey -noout — EC P-256 everywhere; no RSA certs (the only RSA material in the build tree is the IAM JWT signing key, see below).
• Server SANs: --hostname (default localhost) is added to the server certificate. The extra entry is typed DNS: normally, but IP: when the argument parses as an IP address — TLS clients verifying a connection to an IP target check the SAN's IP entries (RFC 6125/2818), and a DNS-typed entry containing dotted-decimal text fails that check with a generic SSL error.
• Serial file: ca.srl is created by openssl x509 -CAcreateserial during leaf signing and deleted when generation finishes.
• Intermediates: there are none — leaves are signed directly by the CA.

Idempotency and rotation: generation skips any existing *.key (presence of the key is the gate), so re-running is a no-op. --force regenerates everything (use in CI with ephemeral keys, or to rotate); --hostname <name> adds a SAN to the server cert.

./compass.sh nats certs                       # idempotent — no-op when keys exist
./compass.sh nats certs --force               # rotate: regenerate CA + all leaves
./compass.sh nats certs --hostname devbox.local   # add a SAN to the server cert

Because leaves are 90-day certificates, rotation is a periodic concern; the 365-day CA is the real clock everything chains to. After regeneration, the running NATS server and all services must be restarted to pick up new certs (compass env deploy <host> re-runs idempotently and recreates the containers; locally, compass nats ensure restarts the systemd unit — see below).

Separate from the NATS PKI, compass env configure also generates build/keys/iam-rsa-private.pem (openssl genrsa 2048, chmod 600, preset-independent). It signs the JWTs services mint for service-to-service authentication, and is consumed in containers only by ores.iam.service (passed via --env ORES_IAM_SERVICE_JWT_PRIVATE_KEY=$(cat <pem>) in docker/remote-run.sh; the value is a single-line PEM string). See Identity and Access Management for the JWT side of the model.

Nothing in ORE Studio pins a TLS version — not in the NATS server config, not in the C++ client, not in the container scripts. The negotiated range comes entirely from the two stacks' defaults:

• Server: the NATS broker is the Go nats-server v2.14.3 binary (docker/nats.Dockerfile, statically linked, runs in a scratch image). Go's crypto/tls server default is a minimum of TLS 1.2 (since Go 1.18) and negotiates TLS 1.3 when the client supports it.
• Client: services connect through cnats (the NATS C client) over OpenSSL. client.cpp only loads the CA and the certificate chain (natsOptions_LoadCATrustedCertificates / natsOptions_LoadCertificatesChain) — no version options are set. The effective floor is the system OpenSSL configuration (Debian bookworm's openssl.cnf sets MinProtocol = TLSv1.2).

The practical outcome is that all connections negotiate TLS 1.2 or 1.3. To verify against a live broker:

# TLS 1.1 must be refused, TLS 1.2 accepted:
openssl s_client -connect localhost:20405 -tls1_1 </dev/null   # fails
openssl s_client -connect localhost:20405 -tls1_2 </dev/null   # succeeds

(The port is per-environment, from the port: line of build/config/nats-<label>.conf.)

compass nats init renders build/config/nats.conf.template into build/config/nats-<label>.conf with per-environment values:

./compass.sh nats init

The TLS block of the rendered config is:

tls {
  cert_file: "<checkout>/build/keys/nats/nats-server.crt"
  key_file:  "<checkout>/build/keys/nats/nats-server.key"
  ca_file:   "<checkout>/build/keys/nats/ca.crt"
  verify:    true   # require client certificates (mTLS)
  timeout:   5
}

verify: true is what makes the setup mutual: the server presents nats-server.crt and requires every connecting client to present a certificate signed by ca.crt. The server's config and certs are consumed on the host (or from the container's bind mounts) — the Go binary reads them as plain files.

• Domain services (the C++ ores.<x>.service binaries): the TLS trio is a command-line option group defined in ores.nats/src/config/nats_configuration.cpp: --nats-tls-ca, --nats-tls-cert, --nats-tls-key (env names ORES_NATS_TLS_CA, ORES_NATS_TLS_CERT, ORES_NATS_TLS_KEY). At connect time, client.cpp first verifies all three files exist (a clear diagnostic instead of a cryptic handshake error), then loads CA + chain via cnats and logs mTLS enabled; a missing client cert trips the mTLS gate and the connection is refused.
• Deliberate fallback exclusion: the trio is not in the shared-domain env fallback (NATS Wire Format documents this decision and its regression tests register_shared_domain_does_not_resolve_tls_ca): the cert/key paths are genuinely per-service, and letting only the CA resolve from a shared ORES_NATS_CA would leave cert/key empty and trip the mTLS gate. Per-app ORES_<APP>_NATS_TLS_* mirrors win where present; the container runners pass the trio explicitly on the command line.
• Shared client identity: reads ORES_NATS_TLS_CA/CERT/KEY from .env, where env_init.py points them at ca.crt and the ores.shell.crt/.key pair.
• ores.shell: keeps its own ORES_SHELL_NATS_TLS_* mirror block in .env (same trio, ores.shell certs), read via its app-prefix mapper.

Two transport paths exist, both driven by compass env deploy <host> (deploy recipe): local deployment (docker/run-pod.sh) and remote deployment (docker/remote-run.sh). The mechanism is identical in both.

Runtime role:

1. env_deploy.py scp's build/keys/nats/ to $REMOTE_ROOT/build/keys/ on the remote (removing the previous remote copy first so scp -r doesn't nest), plus the IAM PEM and a path-rewritten copy of the NATS config.
2. remote-run.sh stages the client certs into a podman-managed volume: podman volume create ores-nats-client-certs-<label>, cp -a the keys into the volume's mountpoint, then chmod -R u+rwX,go-rwx (owner-only: the services container reads the volume as the same uid, --user $(id -u):$(id -g)).
3. Each service container mounts the volume read-only (= -v "$certs_volume:$keys_dir:ro"=) and receives the CLI trio pointing into that path. The NATS sidecar instead gets plain bind mounts of the config and the keys directory (see below).

Compute role: the wrapper's client certs come from the serving environment's checkout (the ORES_COMPUTE_NATS_TLS_CA/CERT/KEY profile keys, local paths in that checkout); env_deploy.py copies them into the staging tree, scp's them to $REMOTE_ROOT/compute/keys/, and remote-run.sh volume-stages them exactly like the runtime role. The wrapper resolves them via the app-prefixed ORES_COMPUTE_WRAPPER_NATS_TLS_* block in compute.env, with paths rewritten to $REMOTE_ROOT/compute/keys/<basename>.

The bind-mount quirk (why the volume exists): documented in the header of docker/run-pod.sh — cnats/OpenSSL silently fails to present a client certificate over a bind mount, even though the bytes, path, and permissions are correct. Staging through a podman volume works, and so does baking the cert into the image; a bind mount does not. The Go nats-server binary in the sidecar has no such problem, which is why the server's config + keys are bind-mounted directly while every C++ service gets the volume.

./compass.sh nats certs --force                 # regenerate everything
./compass.sh nats certs --hostname devbox.local # new SAN, rest untouched
./compass.sh nats ensure                        # local: restart the server with the new certs
./compass.sh env deploy <host>                  # remote: idempotent re-deploy
./compass.sh env deploy <host> --stop           # tear down (--purge drops the certs volume too)

compass env deploy <host> re-runs idempotently — the remote keys directory is removed and re-scp'd, files are overwritten, the certs volume is recreated, and the containers are rm -f before recreate. No manual cleanup required.

compass nats ensure composes the pieces above so a session never starts against stale infrastructure: certificates (idempotent, --force to rotate), the rendered config, and — the step nothing else performs — a restart of the per-environment systemd user unit nats-server-<label>.service. That restart is the whole point: the Go server loads certs at startup and does not pick up a regenerated CA/leaf by itself, so every regeneration must be followed by a restart or every mTLS client fails with SSL Error / bad record MAC (see Idempotency and rotation above). The command finishes by probing the client port for the NATS INFO handshake, so success means the server is actually up and speaking TLS.

./compass.sh nats ensure            # certs + config + restart + probe
./compass.sh nats ensure --force    # force-rotate certs, then restart

After a rebase the local schema can drift from the code it now sits next to (the DB was seeded from an older commit), surfacing as repository errors such as column "workspace_id" does not exist or Invalid concurrency_policy: skip. Check with compass db status and, only when it reports drift, recreate:

./compass.sh db status              # reports schema drift, if any
./compass.sh db recreate -y -k      # drop + recreate + reseed (-y: skip prompt, -k: kill connections)