Story: Hotfix: the services die on shutdown
Table of Contents
This page documents a story in Sprint 28. It captures the goal, current status, acceptance criteria, and the tasks that compose it.
1. Goal
A service survives stopping. Every service that owns a NATS subscription can reach a handler that is already running while the state that handler writes to is being destroyed, and dies on a core dump instead of exiting. The fix goes at the subscription handle: destroying it must wait for the dispatch thread to return from the handler, because the handler reaches the owner's state.
The recorded failure is ores.marketdata.service on 2026-10-18 at 09:55:24. The
shutdown had already logged "Shutdown complete"; the ingest loop was then
destroyed while a tick was still being handled, and the dispatch thread reached
the freed rate engine:
#2 ores::marketdata::service::app::crm_ingest_bridge::update #1 ores::analytics::quant::service::rate_engine::update #0 std::unordered_map::_M_locate_tr ores.marketdata.service-...: Main process exited, code=dumped, status=11/SEGV
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent sprint | Sprint 28 |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-10-18 |
3. Acceptance
- Destroying a subscription returns only after the dispatch thread has left any handler it was running.
client::drain()returns only after the connection has closed, as its documentation already promised.- A full fleet start and stop leaves every unit stopped and none failed.
- A regression test fails if the handle stops waiting: it holds a handler inside a delivery and asserts destruction does not return until release.
- The suite still passes.
4. Tasks
| Task | State | Start | End | Description |
|---|---|---|---|---|
| Scaffold story: Hotfix: the services die on shutdown | DONE | 2026-10-18 | 2026-10-18 | Story scaffolding rides this task: documents, sprint wiring, and the scaffold PR. Close it before merging that PR. |
| Wait for in-flight handlers when a subscription is destroyed | DONE | 2026-10-18 | 2026-10-18 | Initial task for: Hotfix: the services die on shutdown |
5. Decisions
- The wait belongs at the subscription handle, not at each service. Every service that owns a subscription with a handler reaching its own state has the same race, so one fix at the seam covers them instead of a per-service teardown order.
- A drain that does not finish leaks rather than frees. A leaked subscription at shutdown is cheaper than a handler writing into freed memory.
- A closed connection counts as drained, because closing it joins the dispatch threads.
6. Out of scope
- The heartbeat coroutine that
on_startedspawns is never cancelled and keeps publishing until the process exits. This story does not join it; it is a separate leak of the same shutdown order. - Rebuilding every dependent of
libores.natsand running the whole suite here. The change is in one shared library, its binaries are relinked by the build on push, and the affected suites and a fleet cycle were run instead.
7. Result
PR #3550. Destroying a subscription now waits for the handler it carries, so a service can destroy its state as soon as the handle goes out of scope. The recorded crash, one more tick arriving after "Shutdown complete" and reaching a freed rate engine, can no longer happen: the handler finishes first, or the subscription is leaked and the failure logged at error.
Review raised fourteen findings across the two axes; all are answered in the implement task's Review table. The most valuable was that the first version still freed the closure on the timeout path, which would have reintroduced the same use-after-free, silently.