Service Bootstrap Phases: Wiring/Readiness vs. Post-Readiness Activities

A NATS subscription's callback runs on the NATS client library's own dispatch thread (see the IMPORTANT comment in ores.nats/src/service/client.cpp's on_msg), not on the application's io_ctx. nats.subscribe(...)=/=queue_subscribe(...) makes a subject live the moment it is called — independent of whether the surrounding register_handlers() function has returned, and independent of whether the calling coroutine has yielded control back to io_ctx. Likewise, request_sync is a genuinely blocking call into the underlying cnats library (natsConnection_RequestMsg), not an io_ctx-scheduled operation — it does not starve io_ctx of the chance to process anything, because nothing about serving that request needs io_ctx in the first place.

That means: the only real constraint is subscribe before you call. There is no deeper thread/executor deadlock lurking here — but getting the order backwards still produces avoidable retries (the client-credentials provider's authenticate() already backs off and retries "the startup race where [the target] hasn't subscribed its NATS subjects yet") and, for a same-service self-call, reads in the logs exactly like a hang until you know this. Put the two kinds of work in separate, ordered phases and the question never comes up.

Everything that makes the service able to answer a request:

• Connect the NATS client (nats.connect()).
• Build the per-service JWT verifier/signer.
• Construct every handler and call subscribe=/=queue_subscribe for every subject the service owns — via the per-entity register_<entity>_handlers() registrars (Entity-composed registrars), or directly for hand-written handlers. This includes the service's own subjects it might later call itself (e.g. IAM's iam.v1.auth.service-login).

At the end of Phase 1, every subject is subscribed and will accept a request — from another service, or from itself. That is not quite the same as every subject giving its fully warmed answer: a handler that depends on a Phase-2 activity (e.g. a cache not yet armed/warmed) can already be receiving traffic before that activity completes. See "Known limitation" below — this is a narrow, self-healing window, not a correctness problem, but it is real and worth naming rather than implying Phase 1 alone means "fully ready."

Everything that needs to call a subject, including the service's own:

• Mint a service token via ores::iam::client::make_service_token_provider — whether calling another service (the common case; see Service-to-Service Auth: Client Credentials vs. On-Behalf-Of Delegation) or, for IAM itself, calling its own service-login subject.
• Warm any nats-event-cache-generated cache (or a hand-written equivalent) — a bulk read against a subject that, for IAM's party_cache, now requires that same freshly-minted token.
• Any other one-shot startup query against another (or the same) service.
• Finally, the on_started hook (heartbeat publisher, etc.) and the "Service ready" log line — see run_signing=/=domain_service_runner in ores.service.

IAM is the sharpest case because it is self-referential: its party_cache (see Generic entity-mirror cache primitive + codegen facet) calls refdata's read_parties_for_cache, which — since that subject started requiring a valid signed JWT — needs a token. The obvious-looking implementation, "mint the token and construct party_cache right where party_cache is needed," breaks because that point in registrar::register_handlers comes before IAM's own auth_handler (owner of service-login) is constructed later in the same function. Reordering — registering auth_handler (Phase 1) before minting the token and warming the cache (Phase 2) — is the whole fix:

Concretely, inside ores.iam.core's registrar::register_handlers:

1. Phase 1: construct party_cache (token-less — it's needed by auth_handler=/=account_handler=/=bootstrap_handler's constructors), then construct and subscribe every IAM handler (auth_handler included) — in whatever order, as long as all of them precede Phase 2.
2. Phase 2: make_service_token_provider(nats, ctx.service_account(), service_password) (safe now — service-login is live); call party_cache::set_token_provider to arm it; call warm_and_subscribe_party_cache(tenant_ids) to do the initial load and subscribe to ores.refdata.party_changed for future reloads.

A non-IAM service calling another service (e.g. ores.reporting minting a token to call into IAM, or ores.synthetic the same) has no self-reference to worry about — the target service is a separate, already-running process, so there is no ordering constraint relative to its own Phase 1. The two-phase discipline still applies for the same reason it always does (don't call out before you're ready to be called back, e.g. for a reply-and-retry loop that itself needs a subject you haven't subscribed yet), but the sharp, self-referential failure mode is IAM-specific among today's services.

Phase 1 subscribes auth_handler (so real login traffic can already be arriving) before Phase 2 arms party_cache with a token and warms it. A login landing in that exact window that triggers a cache miss (auth_ensure_parties_cached) calls party_cache::load() with no token provider set yet; since read_for_cache now requires a valid JWT, that particular load is rejected server-side and party_cache::load() just logs a warning and returns — leaving that one tenant's partition empty for that request. It is self-healing (the next cache miss, after Phase 2 completes, succeeds normally) and the window is only open for the few subscribe calls between auth_handler and the end of Phase 2 — but it is a real, if narrow, behavioural change from before this pattern existed (previously read_for_cache had no auth check, so the same race was silently harmless). Accepted as-is rather than engineered away, because doing so would mean delaying real traffic until Phase 2 completes — a bigger change with its own tradeoffs — for a window that is already this narrow and self-correcting. If it ever needs to be closed, the options are: gate auth_handler's subscriptions on Phase 2 completing too (defeats the whole point of separating the phases), or have party_cache::load() distinguish "no token provider yet" from a real auth failure so it is diagnosable rather than a generic warning.

• [ ] Every subject the service owns is subscribed in Phase 1, before any Phase 2 activity — including subjects the service might call on itself.
• [ ] Any make_service_token_provider call — self-referential or not — happens in Phase 2, after all of Phase 1's subscribe calls.
• [ ] Any cache warm-up (nats-event-cache-generated or hand-written) happens in Phase 2, after its producer's subjects (if the same service) are subscribed, and with whatever auth token its subjects now require.
• [ ] on_started (heartbeat, etc.) still runs last, as today.
• [ ] If a Phase-1-subscribed handler depends on a Phase-2 activity (e.g. a cache), confirm the failure mode during that narrow window is self-healing and non-silent-in-a-harmful-way — see "Known limitation" above — not a hard error users would notice.