Story: Containerize the ORE Studio service runtime and verify it on a remote WSL host

Table of Contents

This page documents a story in Sprint 24. It captures the goal, current status, acceptance criteria, and the tasks that compose it.

Goal

Work out and prove, on a real remote WSL host (Newton), the mechanics of running ORE Studio's service runtime somewhere other than the local dev machine: containerize the service binaries (glibc mismatch ruled out a plain scp-binaries approach), solve the networking model for a host with its own native Postgres, and get all 18 services running reproducibly with a full Acme scenario passing end-to-end. This is groundwork for actually offloading the day-to-day dev environment to WSL, split out as its own story – Offload service and DB runtime to a WSL host over SSH – once first-class tooling exists to make deploys repeatable rather than a manual SSH/podman sequence.

Status

Field Value
State DONE
Parent sprint Sprint 24
Now All 18 services running reproducibly on Newton, full Acme scenario passed end-to-end. Closed at sprint 24 health review: the remaining compass remote-deploy tooling task is the actual WSL-offload cutover work, not groundwork, so it was split into Offload service and DB runtime to a WSL host over SSH in the product backlog.
Waiting on Nothing.
Next Nothing – see the split-off story for the WSL cutover itself.
Last touched 2026-08-10

Acceptance

  • A WSL-based host is provisioned and reachable via SSH from the dev machine.
  • The service runtime is containerized and deployed to that host, with all 18 services running reproducibly across restarts.
  • A full end-to-end scenario (Acme) passes against the remote deployment.
  • The networking/deployment model (native-Postgres host, --network=host, etc.) is documented in the story's Decisions.

Tasks

Task State Start End Description
Analyse approach for offloading services/DB to a WSL host DONE 2026-07-23 2026-07-24 Outline the approach for provisioning a WSL host, running services/DB there (directly and/or via Docker), and connecting a local client over SSH/network.
Set up Docker for the service runtime deployment DONE 2026-07-24 2026-07-27 Build a Dockerfile and image for ORE Studio's service runtime, targeting a base whose glibc is compatible with the WSL/Debian 11 deploy host, replacing the plain scp-binaries approach that a glibc mismatch ruled out.
Debug: start_all() coroutine never resumes inside podman container DONE 2026-07-27 2026-07-29 process_supervisor::start_all()'s coroutine, spawned via co_spawn(io_ctx, …, use_awaitable) in application.cpp, never begins executing inside the podman container – its unconditional first log line never appears despite the same io_ctx demonstrably running other coroutines concurrently (NATS connect succeeds, JWKS-fetch retries fire on schedule). Natively, the identical code reaches that line and does its DB reads within ~200ms. Consequence: no supervised services (IAM etc.) ever launch in-container, so the controller's own JWKS fetch from IAM loops forever.
Deploy and verify service runtime on the Newton WSL host DONE 2026-07-29 2026-07-29 Deploy the containerized service runtime (docker/run-pod.sh) to the actual Newton WSL host (192.168.1.22, WSL2 Debian 11 bullseye) over SSH. Reached partial success (image/uid/log/DB-password fixes, –network=host model); all-18-running and Qt-client-connects deferred, blocked on an apparent memory-corruption issue in process_supervisor under real load, carried to the split-per-service follow-on story rather than fixed here.
Get all 18 services running end-to-end on Newton DONE 2026-07-30 2026-07-30 Reached full success: all 18 services (22 rows with compute.wrapper replicas) running reproducibly across 3+ restarts, no distinct corruption bug remained once several real bugs (Newton storage EIO, DB connection exhaustion, stale schema, missing http.server/compute.wrapper staging, wrong http-base-url, podman env-file quote handling, NATS cert SAN type, compute dispatch tenant mismatch) were fixed. Acme end-to-end scenario passed in full, including a compute job run to completion. Two deeper issues (compute-domain tenant scoping, a vendored ORE package missing its own shared libs) filed as captures rather than fixed blind.
Decommission ores.controller.service and process_supervisor DONE 2026-08-01 2026-08-02 Removed ores.controller.service/process_supervisor entirely. Bigger than a pure deletion: the controller had a real NATS API consumed by a 950-line Qt Service Dashboard panel (removed too), and compass services start/stop/status – the local dev workflow every worktree uses – worked by launching the controller and letting it cascade-spawn everything; rewrote it to drive systemd generate/deploy + systemctl –user instead, same CLI surface. Verified end-to-end: full build clean, ctest green, 23-unit fleet up/down correctly, Qt client launches cleanly.
Design the per-service container architecture and pilot it with one service DONE 2026-07-29 2026-07-29 Work out how startup ordering, readiness gating, and the DB-backed service_instance/phase tracking currently owned by process_supervisor should move to the orchestration layer (podman pod/compose) once each service is its own container's PID 1 rather than a child process the controller execs. Prove the design on one real service end-to-end (IAM is the natural pilot: it is the dependency every other service waits on) before rolling out to the remaining 17.
Finish IAM per-service pilot: JWT signing failure and HEALTHCHECK DONE 2026-07-30 2026-07-31 The per-service container pilot for IAM got past a rootless-podman uid-remapping issue (fixed, needs –userns=keep-id) and connects to NATS successfully standalone, but does not yet reach a fully healthy end-to-end state: a distinct, not-yet-root-caused JWT token signing failure (in-memory PEM parsing, bio read failed) surfaces afterward. Root-cause and fix that, then add the HEALTHCHECK probe (tailing IAM's own log for Service ready.) and confirm IAM reaches podman ps healthy end-to-end.
Generate per-environment plain-systemd units from service_dependency DONE 2026-07-31 2026-07-31 Scope narrowed to local/plain-systemd only. Generator reads service_definition/service_dependency and renders concrete systemd units (native, –user, EnvironmentFile per environment) for local dev machines, wrapped in a per-environment ores-<env>.target for one-command start/stop/status, plus sd_notify(READY=1) for real readiness-gated ordering and a compass systemd deploy step. Quadlet/podman and controller decommission now tracked as separate follow-up tasks.
Generate Quadlet (podman) units for remote hosts from the same dependency graph DONE 2026-08-01 2026-08-01 Second renderer over the same service_definition/service_dependency graph, emitting Quadlet .container units (compass systemd quadlet/quadlet-deploy) with the same After=/Requires= ordering and a per-environment .target as the plain-systemd path. Verified end-to-end locally (IAM pilot reaches podman ps healthy / systemctl active). Scoped to rendering + local/SSH install only – image build/transfer tooling and the full Newton rollout stay with the sibling offload-to-WSL-host story's own tasks, discovered mid-task.
Re-scope podman-fleet tasks after the pivot to systemd DONE 2026-07-31 2026-07-31 Roll out per-service containers to the remaining 17 services and Mirror podman events both predate the story's pivot to systemd (native + Quadlet) as the single orchestration substrate; abandon the rollout task (superseded by the Quadlet generator task) and narrow the podman-events watcher to remote/Quadlet hosts only, blocked on the Quadlet generator landing.
Roll out per-service containers to the remaining 17 services ABANDONED   2026-07-31 Superseded: its podman pod/compose plan predates the pivot to systemd as the orchestration substrate. Rolling out per-service containers to all 18 services is now the Quadlet generator task's job – same DB-driven graph, already-parameterised image, no separate manual rollout.

Decisions

  • Newton (192.168.1.22, WSL2 Debian 11 "bullseye") is the target host: systemd as PID 1, mirrored networking (no portproxy needed), PostgreSQL 17 + TimescaleDB 2.22 on port 5433 (5432 is blocked by Windows' Hyper-V port-exclusion range, not a real conflict).
  • compass env configure now supports a non-default PGPORT in .env — first-class, not a per-host hack.
  • Packaging: build/cpack/CMakeLists.txt gained a TGZ generator alongside DEB so compass build package produces a plain install-tree tarball. A CPack services/qt component split (to drop the desktop client from the deploy artifact) was scoped and found non-trivial — ~97 untagged install(TARGETS ...) sites across the codebase — and captured separately rather than done inline.
  • Reversed the story's initial no-Docker lean: a live deploy test showed our own C++ service binaries (built on this box's Debian sid/forky) cannot run on Newton's older glibc (Debian 11) when simply copied over — GLIBC_2.38=/=GLIBCXX_3.4.32 etc. not found. This is unrelated to the DB (separate Postgres install) and to nats-server (vendored as a static per-OS binary under external/nats/, unaffected since Go binaries carry no libc dependency) — it's specific to binaries we compile. Docker is now the plan for those, via the new Set up Docker for the service runtime deployment task.
  • Newton reaches its own native Postgres from inside a podman container via --network=host, not the pod/sidecar pattern used locally – run-pod.sh's pod netns assumes Postgres is remote from both containers, which isn't true when Postgres is native on the same host the pod runs on (confirmed slirp4netns can't route to it; WSL2's mirrored networking doesn't extend into podman's own virtual network).
  • Resolved, not a distinct bug after all: deploying to Newton had surfaced an apparent memory-safety/corruption issue in process_supervisor's child-process bookkeeping under real load (garbled bytes in a DB query string, a cert re-read failing after succeeding earlier in the same run), originally left un-chased and attributed to the single-container architecture Split ORE Studio services into one container per service was about to replace. It wasn't a distinct bug: it was the --userns=keep-id uid-remapping issue (root-caused and fixed in that same split-per-service story's own pilot, PR #1752), plus several further real but unrelated bugs (Newton storage EIO, DB connection exhaustion, stale schema, missing staged binaries, wrong http-base-url, podman env-file quote handling, NATS cert SAN type, compute dispatch tenant mismatch) found and fixed getting all 18 services running reproducibly. No corruption remained once these were fixed – see Get all 18 services running end-to-end on Newton.
  • 2026-08-10 (sprint-25 consolidation): the seven closed per-service-container tasks of the split-services backlog story (container pilot, IAM healthcheck, orchestration spec, Quadlet generator, controller decommission, re-scope, rollout) were moved into this story, because their work completed during sprint 24 (2026-07-29..08-02). Their backlog home was closed as superseded; the open podman-events watcher and the compass remote-deploy tooling task live in the sprint-25 story Remote WSL offload and compute nodes.

Out of scope

Emacs 29.3 (Org mode 9.6.15)