Stream Provisioning at Start
Table of Contents
This page defines the NATS pattern Stream Provisioning at Start in the nats pattern language. A service that reads durably creates its stream during startup, before it subscribes, and a failure to do so stops the service.
1. Also known as
- Idempotent stream bootstrap, declarative stream creation.
2. Context
A durable subscription needs a stream to exist. The publisher is the workflow engine or another service, which cannot create the consumer's stream without taking over a resource it does not own, and the publisher may run first.
3. Problem
Which process creates the stream a durable subscription needs, and when?
4. Forces
- A message published while no subscriber exists is lost unless a stream holds it.
- The service that reads the subject owns the data in it, so it owns the stream.
- Several replicas start at once, so creation must be idempotent.
- `ensure_stream()` updates a stream whose filter differs, so the call also repairs drift.
- A stream that cannot be created means no message is ever delivered.
5. Solution
The owning service ensures the stream in its startup path, before it registers handlers, from the subject constants that name the subjects it reads:
try { const auto stream = nats.make_stream_name("iam_steps"); nats.make_admin().ensure_stream( stream, nats.covering_subjects({provision_tenant_step_subject})); } catch (const std::exception& e) { // No stream means no step is delivered, so fail rather than run // looking healthy and dropping every step addressed to it. BOOST_LOG_SEV(lg(), error) << "Failed to ensure stream: " << e.what(); throw; }
Three rules make it correct:
- Ensure, do not create. `ensure_stream()` is idempotent, so every replica calls it and the second finds rather than fails.
- Fail the start. The stream is a precondition for the service working at all, so a failure to create it is fatal, not a warning.
- Cover the subject's children. `covering_subjects({…})` returns the bare subject and its `>` wildcard, prefixed. A filter on the bare subject alone matches none of its children, so the stream would hold nothing.
6. What the age does not say
The age set here says how long the stream holds a message that no consumer has acknowledged. It does not say how long a consumer keeps retrying. Those are two limits, and the consumer's `MaxDeliver` and `AckWait` set the second one. A consumer that is bound and keeps failing gives up after its delivery bound. The message then stays in the stream, unacknowledged, until this age expires, but that consumer never reads it again. State the age and the delivery bound together, or the reader will take one for the other. See Durable Subscription.
The age is also not a promise that the work is still wanted. A consumer that returns after a long absence reads the message, and the handler may then find that the work has expired or been superseded.
7. Failure handling
| Concern | What the pattern does |
|---|---|
| Errors | A failed ensure stops the service, so the absence is a failed unit rather than a silent service. |
| Timeouts | Not applicable: the call is synchronous at startup. |
| Retries | The process supervisor's restart is the retry; `ensure_stream()` is safe to repeat. |
| Idempotency | Guaranteed by the server: create-or-update, keyed by stream name. |
| Observability | The service logs the stream name it ensured. |
8. Pitfalls
- Creating the stream on the publishing side. The publisher owns no part of the reader's data and cannot know its retention.
- A filter on the bare subject. The stream is created, the service starts, and it receives nothing.
- Logging the failure and continuing. The service looks healthy and drops every message addressed to it.
- Renaming a subject without the stream name passed to the subscribe. A durable consumer is created by name on its stream, so the subscribe cannot run without it and refuses the call; see Durable Subscription.
9. Resulting context
- Benefits
- The reader owns its persistence, and a start is either complete or refused.
- Liabilities
- A service now depends on JetStream being enabled on the server.
10. In ORE Studio
`ores.iam.service` ensures `iam_steps`; `ores.workflow.service` ensures `workflow`. A step-command stream is a queue of work, so it keeps a message for one day under `interest` retention; that day is the backstop for a command no consumer can take. The consumer on it gives up after ten unanswered deliveries at thirty seconds each, which is about five minutes of a bound, failing service, not five minutes of downtime. The catalogue is JetStream streams and durable consumers.
`build/scripts/check_stream_coverage.py` reads every stream and every durable consumer in the tree. It fails when a consumed subject has no stream that covers it, when two streams claim the same subject, and when a declaration it cannot read exists. The server refuses both defects at creation, so a running broker cannot show them and the check reads the declarations instead.
11. Related patterns
- Durable Subscription — what needs the stream.
- Commit Through the Stream — the publisher on the other side.
- At Least Once Prerequisites — how the retention policy you set here decides what removes a message.
12. Sources
- NATS documentation, Your first stream.
- NATS documentation, Shaping the stream — retention and filters.
13. See also
- NATS service patterns — the structure note that orders this pattern language, and where to read this page in it.