Story: Streams are not provisioned for the subjects services durabl…
Table of Contents
This page documents a story in Sprint 28. It captures the goal, current status, acceptance criteria, and the tasks that compose it.
1. Goal
Streams are not provisioned for the subjects services durably consume, and two defects share that cause. First, nothing in the codebase creates a JetStream stream: a search for stream creation across every service and core finds none. The workflow engine publishes step commands to a stream, times out after five seconds at workflow_engine.cpp:283, and reports that no service was asked to do the work, so provisioning aborts and reads as flakiness rather than as a missing stream. Second, stale and overlapping streams stop a service starting: iam fails with NATS error 10065 JSStreamSubjectOverlapErr when an old store predates main's stream-splitting change. The pattern is already written down in doc/knowledge/nats_patterns/pattern_stream_provisioning.org (added 2026-10-14): the service that durably reads a subject ensures its stream in its startup path, before it registers handlers, using ensure_stream over covering_subjects, and fails to start if it cannot. It exists, is documented, and is not implemented where it is needed. The work is to apply it to the owning services, resolve the overlap, and add a check that every durably consumed subject is covered by a stream its owner ensures and that streams do not overlap. Without the check the next service repeats it, because the only symptom is a timeout. Found while atomising the tenor basis swap convention, where it blocked ACME tenant provisioning.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent sprint | Sprint 28 |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | None. |
| Last touched | 2026-10-18 |
3. Acceptance
- Every durably consumed subject has exactly one stream covering it, and no two streams overlap. Enforced by a check that fails on each defect.
- A command published while its service is down is held and delivered when the service returns, proven by a test that fails if the consumer is created implicitly.
- The delivery bound is stated rather than inherited, and a consumer that keeps failing spans the stated window instead of spending its attempts in a burst.
- Every from-DQ trigger has a generated constant, with the subject set proven unchanged by comparing it rather than by two passing runs.
4. Tasks
| Task | State | Start | End | Description |
|---|---|---|---|---|
| Ensure the streams a service reads | DONE | 2026-10-17 | Apply the documented stream provisioning pattern to the services that durably consume a subject. Each such service ensures its stream in its startup path, before it registers handlers, over a filter covering the subjects it reads, and fails to start when it cannot. This is what makes a step command survive being published before its consumer is running, and it removes the five second publish timeout that makes provisioning look flaky. | |
| Settle which service owns which stream | DONE | 2026-10-17 | Two streams claim overlapping subjects, which NATS refuses with JSStreamSubjectOverlapErr and which stops the IAM service starting. Decide which service owns each subject, make the streams disjoint, and make the ownership legible, so the next service that adds a subject does not collide by accident. | |
| Check that every durable subject has its stream | DONE | 2026-10-18 | Add the check that keeps the next service from repeating the two defects: every durably consumed subject is covered by a stream its owner ensures, and no two streams overlap. Without it the only symptom is a publish timeout, which is why nobody noticed. | |
| Make the durable consumers durable | DONE | 2026-10-18 | The durable push consumers are not durable. Stopping a service removes its consumer immediately, so a command published while it is down is dropped rather than held, which contradicts both the durable subscription pattern and the workflow engine's assumption that a step waits on a stream. Create the consumer with js_AddConsumer and a stable deliver subject and group, bind it with the jsSubOptions so the server keeps it across a service restart, and prove a command published while the service is down is delivered when it returns. | |
| Give the from-DQ step subjects constants | DONE | 2026-10-18 | The from-DQ step subjects for refdata, dq, assets and marketdata are hand-written string literals in their registrars, because no protocol constant exists for them. A renamed subject therefore leaves a stale stream filter and a consumer that silently receives nothing. Emit a constant for a subject the protocol declares, the way the other services already get theirs, and use it. | |
| Test the down-window delivery | DONE | 2026-10-18 | The durable consumer fix is proven by a manual procedure: stop a service, confirm its consumer survives, publish a command, restart, and see it delivered and acked. That procedure decays. Add an automated test that does the same thing deterministically, so a consumer that again disappears on drain, or a message dropped while the service is down, fails a test rather than being rediscovered by hand. | |
| Settle the redelivery bound | DONE | 2026-10-18 | The streams keep a message for a day, but a push consumer redelivers at most MaxDeliver times, currently ten, with the default AckWait of thirty seconds, so a service down for more than a few minutes exhausts its attempts and the command is lost anyway. That is a narrower promise than the durable subscription pattern now implies. Decide whether to widen the bound, make it explicit per stream, or state the real window, and make the documents say what the system actually does. |
5. Decisions
- Stream provisioning was already implemented on main when this story started, so the story changed shape to the consumer lifecycle and its consequences rather than adding what was already there.
- MaxDeliver stays at ten. It is the claim-takeover budget against a 120 second lease, not a tolerance for downtime, and downtime spends no delivery.
- Subject constants come from operation models. The protocol tables in each component are documentation the generator never reads, which is why writing a subject there produced no constant.