Story: Split ORE Studio services into one container per service

Table of Contents

This page documents a story in Product backlog — inbox, carried unfinished from Sprint 24 at close. It captures the goal, current status, acceptance criteria, and the tasks that compose it.

Goal

Deploy each ORE Studio backend service (currently 17, plus the controller itself) as its own container, launched and supervised by the container orchestrator (podman pod, compose, or eventually k8s) rather than by ores.controller.service spawning them as child processes inside a single container. This is the idiomatic Podman/container pattern – one process per container – and gets us standard per-service health checks, restart policies, and log routing "for free" from the orchestrator instead of reimplementing them in process_supervisor.

The controller currently does real control-plane work beyond process babysitting – dependency-ordered startup, DB-backed instance/phase tracking, readiness gating (see ores.controller.core/service/process_supervisor.cpp) – so this is not a simple lift-and-shift: that logic needs to either move into the orchestration layer (init containers / dependency ordering in the pod spec) or stay as a thin coordination service that no longer itself execs the other binaries.

Confirmed (see the dependency-graph-generator task): the controller has no responsibilities beyond process_supervisor – no other NATS handlers exist on it – so it is decommissioned entirely, not kept as a thin coordinator. Systemd becomes the single orchestration substrate for both supported deployment modes: concrete, per-environment systemd units (native binaries, --user scope, EnvironmentFile= pointing at each environment's own .env) for local dev machines, each running possibly several environments/checkouts side by side; and Quadlet .container units (still systemd under the hood) for podman/remote hosts like Newton. (Local units are deliberately concrete, not systemd template units via %i – an earlier template design hit a real systemd specifier-expansion bug; see the dependency-graph-generator task's Notes.) A per-environment .target unit (ores-<env>.target) aggregates NATS + every service for that environment in both modes, so systemctl --user start/stop ores-<env>.target starts or stops a whole environment (e.g. "swift_curie") as one command.

Status

Field Value
State DONE
Carried from Sprint 24 (unfinished at close)
Now Superseded by Remote WSL offload and compute nodes — tasks moved to sprint 25.
Waiting on Nothing.
Next Nothing — superseded; follow-up work lives in the sprint-25 story.
Last touched 2026-08-10

Acceptance

  • Each of the 18 service binaries runs as PID 1 of its own container image (or a small number of shared images parameterised by entrypoint args), not spawned as a child process by another container.
  • Startup ordering (e.g. IAM before its dependents) and readiness gating are expressed at the orchestration layer, not inside a C++ process supervisor.
  • Per-service logs are retrievable via podman logs <service> – requires each service's own container to pass --log-to-console alongside --log-enabled (confirmed empirically in the IAM pilot: podman's log driver only captures stdout/stderr, and none of our services write there by default).
  • The existing DB-backed service_instance/phase tracking either moves to the orchestrator's own state (preferred) or is demonstrably still useful and is kept deliberately, not by default.
  • End-to-end smoke test: the full fleet starts, reaches steady state, and survives killing and restarting one service container without taking down the others.

Tasks

Task State Start End Description
         

Decisions

  • Per-service images: one parameterised docker/service-runtime.Dockerfile (ARG SERVICE_NAME) plus docker/stage-runtime.sh --service <name>, not 18 separate Dockerfiles and not one shared "kitchen sink" image switched via --entrypoint (the latter was tried first, rejected in review as not idiomatic – ships every service's dead code and a larger attack surface per container).
  • Rootless podman containers reading host-owned NATS client private keys (mode 600) must use --userns=keep-id – without it, the container's mapped uid does not match the host uid that owns the key, causing a silent EACCES inside OpenSSL that surfaced only as a generic "Connection Closed" (fixed at the source in ores.nats/service/client.cpp, which now checks and reports this explicitly). This is the same root cause as the Newton cert- permission fix in PR #1747/~docker/run-pod.sh~ – confirmed to apply uniformly to every container in this architecture, not just Newton.
  • The Newton PR #1747 memory-corruption finding does not reproduce in the per-service pilot (as expected – no in-process multi-child bookkeeping exists there to race). What initially looked like a second instance of it was actually the uid-remapping issue above – a distinct, now-understood, unrelated bug.
  • The IAM pilot's "bio read failed" JWT-signing failure was not a distinct bug either: it was the exact same escaped-\n env-file corruption already root-caused and fixed generically in PR #1779 (docker/generate-env.sh's quote-stripping plus a --env passthrough for that one variable), just predating that fix in this pilot's own session. No source change was needed to close this task once verified. The Newton storage EIO finding from that same PR also reproduces locally under --userns=keep-id, confirming it was never Newton-specific either – both are uid-remap-dependent, not host-dependent.
  • HEALTHCHECK needs a real binary, not a shell script – cgr.dev/chainguard/glibc-dynamic has no shell at all. Added a small statically-linked C program (docker/healthcheck.c), built in its own debian stage, that scans /app/log/*.log for "Service ready.". Also: podman silently drops HEALTHCHECK unless built with podman build --format docker (the default OCI format has no health-check concept) – easy to miss, only a build-time warning.
  • The Newton storage EIO finding's actual fix (not just its uid- remap-dependent nature, established above): --userns=keep-id plus a matching --user~/–group~ is not enough on its own – the image's own baked-in mode-777 /app/log~/~/app/run dirs still return EIO (not EACCES) when written to by a uid other than the one that built them, an overlay-layer quirk independent of uid-remap settings. The fix is bind-mounting real host directories over /app/log~/~/app/run instead of relying on the image's own copies (confirmed in the Quadlet generator task; same fix PR #1779 originally applied). Any future per-service container tooling in this architecture needs these bind mounts, not just --userns=keep-id.

Out of scope

  • Kubernetes manifests – podman pod/compose is the target for this story; k8s is a possible later step, not required here.
  • Redesigning ores.controller.service's domain responsibilities (service definitions/dependencies as data) beyond what's needed to stop it exec'ing other services directly.
  • Production hardening (secrets management, TLS cert rotation, HA) – tracked separately.

Emacs 29.3 (Org mode 9.6.15)