Task: Deploy and verify service runtime on the Newton WSL host

Newton had no podman at all (Debian 11 bullseye ships podman 3.0.1 in its main repo, no extra apt sources needed). As root:

apt update
apt install -y podman
loginctl enable-linger marco

Rootless remapping was already fine (/etc/subuid~/~/etc/subgid already had a marco:100000:65536 entry from base provisioning).

After enabling lingering, podman info still reported cgroupManager: cgroupfs (fallback) instead of systemd – rootless podman on this bullseye build needs a D-Bus session bus (/run/user/<uid>/bus) to talk to the user's systemd --user instance for transient-scope cgroup delegation. dbus-user-session was already installed, but its dbus.socket user unit hadn't been activated yet (first boot after enabling lingering). Fixed once, as marco:

export XDG_RUNTIME_DIR=/run/user/1000
systemctl --user start dbus.socket

systemctl --user is-enabled dbus.socket reports static – it is socket-activated as part of sockets.target, which the user manager pulls in by default, so this should self-heal on future reboots without needing this manual start again. Worth re-checking after Newton's next reboot to confirm that assumption.

The existing docker/run-pod.sh pattern (services + NATS sidecar sharing a pod network namespace, Postgres reached over the real network) assumes Postgres is remote from both containers. On Newton, Postgres is a native install on the same host the pod runs on, so the services container needs to reach the host's own localhost:5433 – unreachable through the pod's isolated netns/slirp4netns (tried the slirp4netns gateway IP, got "Network unreachable"; WSL2's mirrored networking mode does not extend into slirp4netns's own separate virtual network). Ran both containers with --network=host instead (bypassing the pod object entirely – no -p port mappings needed either, host ports are used directly): confirmed via a throwaway container that psql -h localhost -p 5433 then reaches Newton's own Postgres correctly.

Newton's Postgres already had the ores_dev_swift_curie database, schema (270 tables) and all ores_swift_curie_* roles provisioned from earlier work, but with different (stale) passwords than this environment's current .env – services failed with db_connection_failed (exit code 3). Fixed by running compass db recreate -y -k against Newton: since the command reads ORES_DB_HOST~/~PGPORT only from the repo's own .env with no override flag, this required temporarily editing ORES_DB_HOST to Newton's IP and adding PGPORT=5433 to this environment's .env, running the recreate, then restoring the original .env from a backup copy. Filed as its own follow-up (compass should support named env files instead of this).

Derived docker/.env.newton from docker/.env (the compass env configure-managed local one) via sed: all _DB_PORT=5432 -> 5433 (host stays localhost, correct under --network=host), and every absolute path (NATS certs, JetStream store dir) rewritten from this checkout's /mnt/development/... prefix to /home/marco/ores-deploy (Newton has no permission to write under /mnt, and no checkout of this repo). Certs, the rendered build/config/nats-swift_curie.conf (also needed the same path rewrite, plus compass nats init re-run first since its port had drifted from a mid-session .env port reassignment), and this env file were all scp'd to /home/marco/ores-deploy/ on Newton. Added docker/.env.* to .gitignore since this file carries the same secrets as docker/.env.

process_supervisor always launches every child service with --log-enabled and a relative --log-directory ../log (see default_args_template), and writes each child's PID file under ../run – both resolve to /app/log~/~/app/run given WORKDIR /app/bin. Neither directory existed in the chainguard glibc-dynamic final image at all, so the controller's writes there failed outright (permission denied / no such directory) regardless of which uid it ran as. Fixed in docker/service-runtime.Dockerfile: the directories are created (world-writable, since the runtime uid is only known at podman run --user time) in the debian strip build stage (which has a shell) and COPY'd into the final stage (which doesn't).

Running the services container without an explicit --user runs it as whatever uid the image defaults to (chainguard's 65532), which does not match the host uid (marco, 1000) that owns the staged NATS certs (chmod'd owner-only, mirroring run-pod.sh's existing rationale for --userns=keep-id + --user "$(id -u):$(id -g)" locally). Fixed by passing --user 1000:1000 explicitly (no pod object here to carry --userns=keep-id, so the container's own --user has to match the host uid directly instead).

After all the above fixes, start_all() progressed noticeably further (from 0 services ever reaching running to a handful reaching it) but never all 18, and was unstable across runs (different subsets succeeded each time). One run produced a genuinely alarming symptom: a DB query for ores.assets.service's exit-cleanup contained corrupted binary bytes where the service name string should have been (SELECT set_config('app.current_service', '<garbage bytes>'...)), immediately followed by a permission-denied re-reading ca.crt that had already been read successfully earlier in the same run. This looks like a genuine memory-safety issue (a std::string or similar getting corrupted/reused across threads) in process_supervisor's real 18-child-process orchestration under load on real hardware – qualitatively the same class of cross-thread hazard already flagged (and partially fixed) in PR #1724's review, just not fully chased down here.

Deliberately not investigated further in this task: the corruption lives specifically in the single-container process-supervisor's in-process child-process bookkeeping (spawning, monitoring, and DB-recording 18 children from one controller process), which is exactly the architecture the sibling follow-on story (Split ORE Studio services into one container per service) replaces. Revisit there first; if the corruption reproduces in the split architecture too, it is a separate, real bug worth its own task.