Task: Debug: start_all() coroutine never resumes inside podman container

Table of Contents

This page documents a task in the Offload service and DB runtime to a WSL host over SSH story. It captures the goal, current status, acceptance, and any notes or results.

Goal

Find and fix why process_supervisor::start_all()'s coroutine never gets resumed when ores.controller.service runs inside a podman container, even though the same io_context is demonstrably alive and running other coroutines concurrently (NATS connect succeeds, JWKS-fetch retries fire on their exponential backoff schedule). The likely next step is attaching a debugger inside the container's own PID namespace (nsenter + gdbserver, since host-side gdb -p <pid> is blocked by podman's rootless PID/user namespace isolation) or adding targeted tracing around the co_spawn call and process_supervisor's internal db_pool_ (boost::asio::thread_pool{1}) to see whether that dedicated thread is even starting.

Once fixed, re-verify end-to-end via docker/run-pod.sh: all 18 services should reach the running phase in public.ores_controller_service_instances_tbl and the controller's own JWKS fetch from IAM should succeed on the first attempt, not loop forever.

Status

Field Value
State DONE
Parent story Offload service and DB runtime to a WSL host over SSH
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-07-27

Acceptance

  • Root cause identified and documented (thread/cgroup limit, stack-size difference, executor-binding bug, or something else).
  • docker/run-pod.sh end-to-end: ores.controller.service.log (or console log) shows "Starting all services…" within a second of the NATS connect log line, matching native timing.
  • All 18 rows in public.ores_controller_service_instances_tbl reach phase running (or a legitimately expected terminal phase) within the normal startup window, with fresh created_at timestamps for that run.
  • The controller's own JWKS public-key fetch from IAM succeeds without entering its retry-with-backoff path.

Plan

(Implementation strategy. Written when work starts; key decisions are distilled into the parent story's * Decisions at close, but the plan itself stays — it is the historical record of what we did.)

Notes

TODO before closing: mop up temporary diagnostics + fix logging gaps

While debugging, added raw fprintf(stderr, ...) diagnostics directly in process_supervisor.cpp~/~application.cpp instead of BOOST_LOG_SEV, specifically to avoid a separate, real bug found along the way: lifecycle_manager::make_console_sink() never calls backend->auto_flush(true) on the console sink's text_ostream_backend (unlike make_file_sink(), which does). Without it, console output sits in std::cout's buffer and only appears once it fills or the process exits/flushes explicitly – misleading under podman logs, where output can appear to "hang" for a long time before a burst arrives. This cost real debugging time before being identified.

Before closing this task:

  • Fix make_console_sink() to auto_flush(true) like the file sink already does.
  • Remove the raw fprintf diagnostics added during this investigation (co_spawn entry points, start_all()~/~do_launch() entry, DB read returns, topo-sort/loop-iteration markers).
  • Promote the genuinely useful ones to permanent BOOST_LOG_SEV(..., trace) (or debug) log lines instead of deleting outright – several pin down exactly where startup is in a way the existing log statements don't (e.g. per-iteration launch-loop progress, DB read counts).

Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario, run through the QA Validation Runner panel) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

PRs

PR Title
#1724 [ores.controller] Fix start_all() never resuming (co_spawn use_awaitable hazard)

Review

# Comment summary File Decision Notes
1 Cross-thread race: completion signal (start_all_done/start_all_failure) can run on db_pool_'s thread instead of io_ctx, racing the async_wait/read on io_ctx process_supervisor.cpp, application.cpp Accepted do_launch() now posts back onto ioc_ before returning, mirroring monitor_process(); keeps the whole start_all() coroutine chain on io_ctx
2 catch (const std::exception&) too narrow inside a detached coroutine; non-std::exception would call std::terminate() application.cpp Accepted Added catch (…) alongside, recording a generic failure
3 catch (const boost::system::system_error&) swallows any error, not just operation_aborted application.cpp Accepted Now rethrows unless the code is operation_aborted
4 Root-cause narrative comment's claim about co_spawn scheduling worth double-checking application.cpp Declined Comment reflects extensive empirical verification described in the PR; left as-is

Result

Root cause: process_supervisor::start_all() was spawned via co_spawn(io_ctx, ..., use_awaitable) in application.cpp without an immediate co_await on the returned awaitable. Confirmed by direct, repeated experiment that this pattern never gets its first resume scheduled in some environments (reproducible over hundreds of retries / hours of container runtime), even though the same io_context was demonstrably running other coroutines throughout.

Fix: spawn start_all() detached instead (detached's first resume runs immediately and reliably), and signal completion via a one-shot steady_timer (start_all_done, moved into the past when start_all() finishes) that application::run() awaits before touching supervisor again – the standard Asio idiom for a join point without holding the coroutine's own completion token.

Review (4 passes) surfaced a related cross-thread race in that new completion signal: process_supervisor::do_launch() hops onto a separate db_pool_ thread pool for post-launch DB writes and never hopped back, so once start_all()'s loop crossed onto that thread the whole coroutine chain – including the completion signal – could run there instead of on io_ctx, racing the async_wait() on the same timer object. Fixed by having do_launch() post back onto ioc_'s executor before returning, mirroring the existing pattern in monitor_process(). Also widened exception handling in the detached coroutine (catch (...) alongside catch (const std::exception&)) and narrowed the timer's exception handling to only swallow operation_aborted, rethrowing anything else.

Along the way, fixed a second real bug found during debugging: lifecycle_manager::make_console_sink() never called auto_flush(true) on the console sink, unlike the file sink – this made podman logs output appear to "hang" in bursts, which cost real debugging time. Also fixed a docker/run-pod.sh podman rootless uid-remap issue (--userns=keep-id on the pod).

Acceptance met: all 18 services now reach running phase end-to-end via docker/run-pod.sh, matching native timing; verified natively and by extensive manual runs described in PR #1724. No new automated test coverage was added for the scheduling/threading fix itself – it is a timing/environment-dependent race, hard to assert on deterministically, consistent with all four review passes' own conclusion.

PR #1724 merged into main.

Emacs 29.3 (Org mode 9.6.15)