Story: Split ORE Studio services into one container per service
Table of Contents
This page documents a story in Product backlog — inbox, carried unfinished from Sprint 24 at close. It captures the goal, current status, acceptance criteria, and the tasks that compose it.
Goal
Deploy each ORE Studio backend service (currently 17, plus the controller
itself) as its own container, launched and supervised by the container
orchestrator (podman pod, compose, or eventually k8s) rather than by
ores.controller.service spawning them as child processes inside a single
container. This is the idiomatic Podman/container
pattern – one process per container – and gets us standard per-service
health checks, restart policies, and log routing "for free" from the
orchestrator instead of reimplementing them in process_supervisor.
The controller currently does real control-plane work beyond process
babysitting – dependency-ordered startup, DB-backed instance/phase
tracking, readiness gating (see
ores.controller.core/service/process_supervisor.cpp) – so this is not a
simple lift-and-shift: that logic needs to either move into the
orchestration layer (init containers / dependency ordering in the pod spec)
or stay as a thin coordination service that no longer itself execs the
other binaries.
Confirmed (see the dependency-graph-generator task): the controller has
no responsibilities beyond process_supervisor – no other NATS
handlers exist on it – so it is decommissioned entirely, not kept as a
thin coordinator. Systemd becomes the single orchestration substrate
for both supported deployment modes: concrete, per-environment
systemd units (native binaries, --user scope, EnvironmentFile=
pointing at each environment's own .env) for local dev machines,
each running possibly several environments/checkouts side by side; and
Quadlet .container units (still systemd under the hood) for
podman/remote hosts like Newton. (Local units are deliberately
concrete, not systemd template units via %i – an earlier
template design hit a real systemd specifier-expansion bug; see the
dependency-graph-generator task's Notes.) A per-environment .target
unit (ores-<env>.target) aggregates NATS + every service for that
environment in both modes, so systemctl --user start/stop
ores-<env>.target starts or stops a whole environment (e.g.
"swift_curie") as one command.
Status
| Field | Value |
|---|---|
| State | DONE |
| Carried from | Sprint 24 (unfinished at close) |
| Now | Superseded by Remote WSL offload and compute nodes — tasks moved to sprint 25. |
| Waiting on | Nothing. |
| Next | Nothing — superseded; follow-up work lives in the sprint-25 story. |
| Last touched | 2026-08-10 |
Acceptance
- Each of the 18 service binaries runs as PID 1 of its own container image (or a small number of shared images parameterised by entrypoint args), not spawned as a child process by another container.
- Startup ordering (e.g. IAM before its dependents) and readiness gating are expressed at the orchestration layer, not inside a C++ process supervisor.
- Per-service logs are retrievable via
podman logs <service>– requires each service's own container to pass--log-to-consolealongside--log-enabled(confirmed empirically in the IAM pilot: podman's log driver only captures stdout/stderr, and none of our services write there by default). - The existing DB-backed service_instance/phase tracking either moves to the orchestrator's own state (preferred) or is demonstrably still useful and is kept deliberately, not by default.
- End-to-end smoke test: the full fleet starts, reaches steady state, and survives killing and restarting one service container without taking down the others.
Tasks
| Task | State | Start | End | Description |
|---|---|---|---|---|
Decisions
- Per-service images: one parameterised
docker/service-runtime.Dockerfile(ARG SERVICE_NAME) plusdocker/stage-runtime.sh --service <name>, not 18 separate Dockerfiles and not one shared "kitchen sink" image switched via--entrypoint(the latter was tried first, rejected in review as not idiomatic – ships every service's dead code and a larger attack surface per container). - Rootless podman containers reading host-owned NATS client private
keys (mode 600) must use
--userns=keep-id– without it, the container's mapped uid does not match the host uid that owns the key, causing a silentEACCESinside OpenSSL that surfaced only as a generic "Connection Closed" (fixed at the source inores.nats/service/client.cpp, which now checks and reports this explicitly). This is the same root cause as the Newton cert- permission fix in PR #1747/~docker/run-pod.sh~ – confirmed to apply uniformly to every container in this architecture, not just Newton. - The Newton PR #1747 memory-corruption finding does not reproduce in the per-service pilot (as expected – no in-process multi-child bookkeeping exists there to race). What initially looked like a second instance of it was actually the uid-remapping issue above – a distinct, now-understood, unrelated bug.
- The IAM pilot's "bio read failed" JWT-signing failure was not a
distinct bug either: it was the exact same escaped-
\nenv-file corruption already root-caused and fixed generically in PR #1779 (docker/generate-env.sh's quote-stripping plus a--envpassthrough for that one variable), just predating that fix in this pilot's own session. No source change was needed to close this task once verified. The Newton storageEIOfinding from that same PR also reproduces locally under--userns=keep-id, confirming it was never Newton-specific either – both are uid-remap-dependent, not host-dependent. HEALTHCHECKneeds a real binary, not a shell script –cgr.dev/chainguard/glibc-dynamichas no shell at all. Added a small statically-linked C program (docker/healthcheck.c), built in its own debian stage, that scans/app/log/*.logfor "Service ready.". Also: podman silently dropsHEALTHCHECKunless built withpodman build --format docker(the default OCI format has no health-check concept) – easy to miss, only a build-time warning.- The Newton storage
EIOfinding's actual fix (not just its uid- remap-dependent nature, established above):--userns=keep-idplus a matching--user~/–group~ is not enough on its own – the image's own baked-in mode-777/app/log~/~/app/rundirs still returnEIO(notEACCES) when written to by a uid other than the one that built them, an overlay-layer quirk independent of uid-remap settings. The fix is bind-mounting real host directories over/app/log~/~/app/runinstead of relying on the image's own copies (confirmed in the Quadlet generator task; same fix PR #1779 originally applied). Any future per-service container tooling in this architecture needs these bind mounts, not just--userns=keep-id.
Out of scope
- Kubernetes manifests – podman pod/compose is the target for this story; k8s is a possible later step, not required here.
- Redesigning
ores.controller.service's domain responsibilities (service definitions/dependencies as data) beyond what's needed to stop it exec'ing other services directly. - Production hardening (secrets management, TLS cert rotation, HA) – tracked separately.