JetStream streams and durable consumers

JetStream is the persistence and delivery layer of NATS. Plain NATS delivers a message to whoever is subscribed at that instant and forgets it; JetStream writes messages into a named stream and lets a consumer read them later, from where it left off. ORE Studio uses it for two things and no more: the market data a service must not lose while its consumer is restarting, and the workflow engine's messages, both the engine's own control commands and the step commands it dispatches to the services that serve them. A durable consumer is the one kind of subscription that survives a service restart. Two failures to know about. A subject rename leaves a consumer whose stored filter differs from the one the code now asks for, and a service that declares a consumer per subject leaves server-side objects that nothing declares and nothing reads.

Three consequences decide where we use it. A message published while no one is listening is lost without a stream and kept with one. A durable consumer is identified by name, so a service that comes back attaches to the same position it left. And it is the only part of the bus with state outside a process — a stream, and the consumers on it, live in the server's file store and outlive both the publisher and the subscriber.

• Not market and synthetic ticks. A tick is perishable. A tick that arrives late, while a newer tick waits, is worse than a tick that is lost, so ticks travel on core NATS and no stream stores them. A stream would replay old ticks first after a restart, and it would make each publish wait for an acknowledgement. See The tick ingest pipeline.
• The workflow engine. A workflow run is a sequence of persisted steps. The engine subscribes durably, so a run that is mid-flight when the engine restarts resumes rather than stalls.

Compute work does not use JetStream. A queue of jobs that wait for hours does not suit a push consumer, so the queue is a table in the database and a free node claims a job from it. See How the compute grid queues and claims work. A deployment that ran an earlier wrapper may still hold a stream named compute_assignments and a durable consumer on it. Nothing reads or writes them now, and they may be deleted.

Nothing else uses JetStream. An entity operation has its answer in the reply and needs no history, so it stays on plain request/reply.

Two calls in ores.nats, and one in ores.service, are the whole surface. Everything else is naming.

// The publisher side: create the stream if it is absent, then keep it honest.
admin.ensure_stream(stream_name, {subject_filter}, max_age_days, retention);

// The reader side: name the stream so the durable is created on it explicitly,
// then bind a queue subscription to it; both outlive a restart.
nats.js_queue_subscribe(subject, durable_name, queue_group, handler, stream_name,
                        max_deliver, ack_wait);

// The service side: declare a family of step commands once, and serve the whole
// family on one durable consumer.
ores::service::messaging::step_routes routes;
routes.on(subject, handler);
serve_step_commands(nats, service_name, routes);

Three rules make the names line up:

• make_subject prepends the deployment's prefix to a relative subject, so the same code addresses a different bus per environment. _INBOX subjects are exempt: the server creates them and a prefix would break request/reply.
• make_stream_name turns the prefix's dots into underscores, because a JetStream stream name may not contain one. ores.dev.local2 plus workflow is ores_dev_local2_workflow.
• A stream's subject filter is a wildcard over the subjects it holds. A broadcast subject that carries a source name or a key below its prefix — a tick, an assignment — is covered with a trailing .>.

Every stream is file-backed. The retention policy follows the kind of message the stream carries. A data stream keeps every message for seven days (ensure_stream's max_age_days default) under the limits policy. A step-command stream is a queue of work, so it uses interest retention with a one-day backstop: the server removes a command as soon as the consumer that filters it has acknowledged it, and the day only bounds a command no consumer can take. A stream's subject filter is the contract that decides which publishes it holds, so ensure_stream compares it, the retention policy and the age limit against the ones asked for and updates them when any differs, in place. A durable consumer stores its own filter and delivery settings, so js_queue_subscribe reconciles those the same way, also in place. Both reconciles are what make a renamed subject heal without losing a position, and what makes a changed delivery bound reach a consumer that is already stored.

The consumer is not left to the subscribe to create. js_queue_subscribe creates it explicitly, with a delivery subject derived from the stream and the consumer name and a delivery group named for the service, and only then binds the subscription to it. A consumer that the subscribe creates itself is removed by the server when the subscription drains, even when it is marked durable, and a service drains on shutdown; that is the shape that loses a command published during a restart. Creating it explicitly keeps the consumer and its position on the server, and keeps the delivery subject stable, so the next start finds both.

Each stream is created by the service that owns the data or the work in it.

The tick subjects have no stream. Both services delete the three tick streams marketdata_ticks, synthetic_ticks and synthetic_sandbox_ticks when they start. The delete is idempotent, so a server that still holds them from an older deployment loses them.

A service that serves two independent families of step commands declares two step_routes tables, and the group names the second one. The groups in use are apply for refdata's book and counterparty apply steps, from_dq for IAM's data-quality steps, and execution for reporting's report-execution steps.

The two engine consumers are a single position each, because the engine is a single logical reader.

A step-command stream has exactly one consumer, named after the stream, and it filters every subject the service declared. One consumer per subject is what this replaces: a service with forty-five data-quality steps owned forty-five server-side objects, and every renamed subject left one behind. On start, serve_step_commands subscribes the one consumer and then deletes every other consumer on the stream, so a service that still carries the older shape cleans it up on its first start. The prune runs after the subscribe, never before: between the two the stream would have no consumer at all, and under interest retention a command published in that window would be dropped rather than held.

The stream's age and the consumer's delivery bound answer different questions, and the code sets both.

MaxDeliver and AckWait are passed to every js_queue_subscribe call and written onto the consumer by ensure_durable_consumer. They are not inherited from a server default, because the claim protocol is written against them: step_claim_lease sizes the takeover from ten deliveries at thirty-second spacing. Do not lower either number without revisiting that invariant.

The delivery bound is not a downtime window. A push consumer with no bound subscriber receives nothing, so it spends no delivery while its service is down. A command published in that window stays unprocessed on the consumer, and the stream's age is the only limit. Measured on server 2.14.6: a push consumer with MaxDeliver 3 and AckWait 2 s was published to with no subscriber, and after seven seconds it showed NumPending 1, NumAckPending 0 and NumRedelivered 0. It delivered the command when a subscriber bound.

The bound applies once a member is bound and keeps failing. The server redelivers after AckWait, up to MaxDeliver deliveries. Ten at thirty seconds is about five minutes. Then the server stops delivering that message to that consumer. It does not acknowledge the message and it does not park it: the message is removed from the consumer's pending set and stays in the stream, unacknowledged, until the stream's age expires. The consumer that gave up never receives it again, even after a restart. A new consumer reading from DeliverAll would still see it. Treat the message as lost to the stored durable.

A consumer whose bound changes is updated in place by ensure_durable_consumer, which keeps its position and its creation time. The update reaches future deliveries only. A message the consumer already gave up on is no longer pending, so a raised bound does not bring it back.

A step command is work, and the engine fails a step at its deadline. A command delivered after that deadline is answered and discarded by the claim. The stream kept it for a day, but the work was no longer wanted.

This is the one failure that costs a service its start, and it fails closed.

A rename rewrites the subject a service subscribes to. ensure_stream then reconciles the stream, so the stream carries the new filter and everything looks right. But a durable consumer's filter is stored in the consumer, not the stream, and nothing reconciles a consumer unless the subscribe does.

js_queue_subscribe closes that hole. The stream name is required — pass it as make_stream_name returns it — and the call reads the durable's stored configuration first, updates the filter, the ack policy, the delivery bound, the ack wait, the delivery subject and the delivery group in place where any differs, and only then binds:

const std::string engine_stream(nats.make_stream_name("workflow"));

nats.js_queue_subscribe(
    messaging::step_completed_event::nats_subject,
    "workflow-engine-step-completed",
    qg,
    handler,
    engine_stream,
    step_max_deliver,
    step_ack_wait);

The update is in place because the alternative costs the consumer its position. The server accepts a changed filter set, and the consumer keeps its cursor, so widening a filter set does not replay a backlog the consumer had already worked through. The filter set is compared as a set, and a stored singular filter is read as a set of one, which is how the server stores a single filter. A consumer whose configuration already matches is reused as it stands, which is the normal restart path.

The stream name is not optional. The consumer is created by name on that stream, so a call without it is refused before anything is subscribed.

To read a stored filter by hand:

nats --server "$ORES_NATS_URL" --tlsca build/keys/nats/ca.crt \
     --tlscert build/keys/nats/<service>.crt --tlskey build/keys/nats/<service>.key \
     consumer info <stream> <durable>

Verified on 2026-10-15 against server 2.14.6: a push consumer with a queue group and one filter was widened to three filters, and its delivered sequence stayed where it was. Its stored configuration was updated, not replaced.

• Message Queue — the structure note that orders this cluster, and where to read this page in it.
• NATS — the substrate: subjects and wildcards, pub/sub against request/reply, queue groups, and what a stream is in NATS's own terms.
• ores.nats — the library that wraps it, where ensure_stream and js_queue_subscribe live.
• NATS Entity Protocol Specification — the grammar a stream's subjects have to obey.
• A subject rename strands every durable consumer — the memory: name the stream so the rename heals.
• At Least Once Prerequisites — the pattern a stream and a consumer have to satisfy before their delivery is safe to rely on.