Task: Design the per-service container architecture and pilot it with one service

Read from source (process_supervisor.hpp~/.cpp~):

1. Desired-state data: service_definition (binary, replicas, restart policy, max restarts, args template, enabled) and service_dependency (edges: A must start before B) – both DB tables, bitemporal, changeable without a redeploy.
2. Startup ordering: topological sort over the dependency graph (boost::graph), launched in dependency-first order.
3. Readiness gating: for any service with dependents, block launching those dependents until the string "Service ready." appears in the service's own log file (wait_for_log_ready, polling every 500ms, 60s timeout). This string is logged by every service via the shared ores.service runner (domain_service_runner_impl.hpp, signing_service_runner_impl.hpp, wt_service_runner.hpp) right after it registers its NATS handlers – i.e. "ready" already means "connected to NATS and serving", not just "process started".
4. Process spawn/monitor: boost::process::v2 execs the binary in-process, builds its CLI args from the template, monitors exit via async_wait, and applies restart policy (always~/~on-failure~/ ~never, capped by max_restart_count).
5. DB-backed audit trail: service_instance (current phase/pid) and service_event (started/stopped/exited history) written by the controller itself on every launch/exit – this is what ores_controller_service_instances_tbl (checked in the Newton task) reflects.
6. Dynamic control: request_launch~/~request_stop~/~request_restart triggered by NATS requests, so a service can be added/scaled/ restarted at runtime without restarting the controller.

• Process spawn/monitor/restart-policy (#4): moves fully to podman. podman generate systemd (or a compose/pod spec's own restart: policy) replaces boost::process::v2 + the restart-policy branch in monitor_process entirely – this is exactly the class of bug that has caused two debugging sessions so far (the ~co_spawn~/executor-hop race in PR #1724, and the Newton corruption finding in PR #1747), both inside this in-process child-bookkeeping code. Deleting it, not fixing it again, is the actual point of this story.
• Startup ordering (#2): moves to the orchestration layer's own dependency expression – systemd unit After=~/~Requires= (if podman generate systemd), or depends_on (compose/podman-compose). The DB-driven dependency graph (#1) doesn't disappear: a small generator step reads service_dependency and emits the ordering directives, so the graph stays data-driven rather than hand-maintained in a static file. This generator is new work, not yet built.
• Readiness gating (#3): moves to a podman HEALTHCHECK per service image. Since there is no HTTP/NATS health-probe surface today (checked – no /health endpoint, no ping subject), the pragmatic first cut is a healthcheck script that greps the container's own stdout (via podman logs or, if using the log-file convention, the mounted log file) for "Service ready.", i.e. automate today's wait_for_log_ready polling loop as a container-native probe instead of inventing a new readiness protocol in this task. A real health-probe endpoint (NATS ping/pong, since every service is already a NATS client) is a better long-term answer and worth its own follow-up, not blocking this pilot.
• DB-backed audit trail (#5): does not have a natural orchestrator equivalent (podman doesn't know about our schema). Two options: (a) each service self-reports its own phase via a NATS event on startup/shutdown instead of being reported on its behalf, or (b) a thin process watches podman events --format json and mirrors state transitions into the DB, keeping today's schema and audit trail intact without any per-service code change. (b) is less invasive and is the pilot's choice – see below.
• Dynamic control (#6): request_launch~/~stop~/~restart become podman start/stop <container> (or systemd systemctl --user start/stop <unit>) calls instead of boost::process spawns. Still triggered by the same NATS requests; the controller keeps this responsibility, it just shells out instead of forking directly.

IAM is the one dependency every other service already waits on (has_dependents in start_all()), so it exercises readiness gating (#3) even alone, and is the natural first real container:

1. Build a per-service image parameterised by entrypoint args (reusing docker/service-runtime.Dockerfile's binary/lib layout, just ENTRYPOINT ["./ores.iam.service"] instead of the controller) – not 18 separate Dockerfiles; one shared image, selected at podman run time.
2. Add a HEALTHCHECK that tails IAM's own log for "Service ready."
3. Run IAM standalone under podman (no controller exec'ing it), talking to the same NATS/Postgres this environment already uses.
4. Verify: IAM reaches healthy per podman ps, podman logs iam shows its own output directly (no --log-to-console plumbing needed – podman's own log driver captures container stdout regardless), and a dependent service can be pointed at it successfully.
5. Explicitly check whether the Newton corruption finding (PR #1747) reproduces here – it shouldn't, since there is no in-process multi-child bookkeeping left to race, but this needs confirming, not assuming.

Not attempted in this task: the dependency-graph-to-orchestration-spec generator (#2), the podman events DB-mirroring watcher (#5), or rolling out to the remaining 17 services – those are follow-up tasks once the pilot proves the model on IAM.

First cut of this task tried reusing the existing "kitchen sink" docker/service-runtime.Dockerfile (all 18 binaries, ENTRYPOINT fixed to the controller) with --entrypoint overridden at podman run time. Correctly rejected in review as not idiomatic: every service's container would ship all 17 other services' dead code and a larger attack surface, defeating the point of "one container, one service".

Fixed properly:

• docker/stage-runtime.sh gained --service <binary-name>: stages just that one binary plus its own ldd-computed dependency closure (a single ldd call already resolves the full transitive closure, not just direct links, so no recursion is needed even though our libraries depend on one another). Default behaviour (all 18) is unchanged and still used for the combined controller image.
• docker/service-runtime.Dockerfile gained ARG SERVICE_NAME and now builds an entrypoint symlink (ln -s "./${SERVICE_NAME}" /app/bin/entrypoint) in the debian strip stage rather than hard- coding ENTRYPOINT ["./ores.controller.service"]. Tried creating that symlink in its own separate minimal debian stage first, pointing at an absolute path (/app/bin/${SERVICE_NAME}) that only exists in the final stage: buildah's COPY --from= validates/dereferences symlinks against the source stage's own filesystem and fails on a target that doesn't exist there yet. Fixed by creating the symlink in the strip stage (where the real binary already exists) with a relative target (./${SERVICE_NAME}), which resolves correctly in both stages since binary and symlink always end up siblings in the same directory.
• Result: an IAM-only image is 410MB (40 libs) vs. 2.1GB (130 libs) for the all-18 image – real, working, and independent of the corruption/ cert-loading findings below.

The alarming symptom from the first pilot attempt (IAM's NATS mTLS connect closing with the server logging "client didn't provide a certificate", despite the client log claiming "mTLS enabled") is not a bug in our code, and not related to the Newton memory-corruption finding (PR #1747) – it reproduced with a single container (no in-process multi-child bookkeeping at all to race), which itself was the first sign these are different issues.

Root cause, found by instrumenting client.cpp::connect() to actually check natsOptions_LoadCATrustedCertificates~/ ~natsOptions_LoadCertificatesChain's return status (previously ignored – a real bug in its own right, fixed alongside this finding): chain_status=SSL Error. Rootless podman, without --userns=keep-id, maps container UID 1000 to an unrelated host subuid (from /etc/subuid), not to the real host account that owns the NATS client private key files (chmod 600, owner-read-only). --user 1000:1000 therefore does not mean "the same UID 1000 as the host" – the container process's real, kernel-visible UID differs, so reading a 600-permission file it doesn't own fails with a plain EACCES that OpenSSL surfaces as a generic SSL error, previously silently ignored. The world-readable CA cert (664) never showed this, since any UID can read it regardless of ownership – only the owner-only private key depended on UID identity matching.

This is the exact same root cause as the Newton cert-permission finding in PR #1747 (which is why docker/run-pod.sh already uses --userns=keep-id) – it just hadn't been connected to this pilot until reproducing and diagnosing it directly here. First confirmed on IAM; on retesting the controller with the same instrumented build, it turned out the earlier "successful" controller run in this session was never real success – head -N truncated the log before it reached the NATS connect call at all (the controller's own log order has start_all()'s "Starting all services…" line appear before nats.connect() is even attempted), so every container run this session, controller and IAM alike, has consistently needed --userns=keep-id.

Fixed permanently in ores.nats/service/client.cpp: both LoadCATrustedCertificates and LoadCertificatesChain now check their return status and throw a nats_connect_error with an actionable message (naming the exact file and pointing at the --userns=keep-id cause) instead of silently continuing to a connection attempt that fails much later with an opaque "Connection Closed".

With --userns=keep-id, IAM connects to NATS successfully standalone (no controller exec'ing it) and progresses into real application logic (reading effective permissions, attempting its own service-account JWT signing) – direct evidence the per-service container model is viable for a real domain service, not just the controller. Ran out of time in this session to chase the next error surfaced there (JWT token creation failed: failed to load key: bio read failed – looks like a similar file-loading issue, quite possibly the same uid-remap class of bug applied to a different file, i.e. the IAM JWT private key rather than the NATS client key) and to build/verify the HEALTHCHECK and full end-to-end acceptance criteria.