Durable Subscription

• NATS: durable consumer, push consumer.
• The pattern language: Queue-Based Load Leveling.

A message arrives that is the only record of a fact: a workflow step command, a market tick. The consumer may be restarting, redeploying or briefly overloaded, and the publisher has already moved on.

How does a message survive the moment when no member of the queue group is listening?

• Core NATS delivers to whoever is subscribed at that instant and forgets the message otherwise.
• A restart is routine, so losing work to one is not acceptable.
• The consumer's position must survive the restart too, not only the message.
• The publisher must not have to know whether the consumer is up.

Publish to a subject a stream covers, and read it from a durable consumer:

const std::string engine_stream(nats.make_stream_name("workflow"));

// The durable name is the reader's position, so it must not change between
// runs. The stream reconciles the durable before the subscription attaches.
// The last two arguments say how many deliveries the server makes before it
// stops, and how long each delivery may stay unanswered.
nats.js_queue_subscribe(
    messaging::step_completed_event::nats_subject,
    "workflow-engine-step-completed",
    queue_group,
    handler,
    engine_stream,
    step_max_deliver,
    step_ack_wait);

Three properties define the pattern:

• The durable name is the reader's identity. Restarting attaches to the same position, so nothing between the last ack and now is skipped.
• The queue group still applies: a durable consumer scales across replicas exactly as Queue-Group Subscription does.
• The stream name is passed. The subscribe then reconciles a durable whose stored filter differs from the subject asked for, which is what a renamed subject leaves behind; see Stream Provisioning at Start.

The consumer is created explicitly, with a name, a delivery subject derived from the stream and that name, and a delivery group named for the service, and the subscription is bound to it afterwards. A consumer that the subscribe creates itself is removed by the server when that subscription drains, even when it is marked durable, and a service drains on shutdown; the consumer would then be gone while the service restarts. Creating it explicitly keeps the consumer, the delivery subject it publishes to, and the position it holds on the server, so a message published during the restart waits for the next start to read it.

A durable consumer reads at-least-once. The handler must therefore be safe to run twice; see Idempotent Message Identity.

Two different limits govern a message, and they apply to different failures.

• Stream age. The stream keeps a message for its max_age. A command stream in ORE Studio uses one day; the workflow stream uses seven. This is the only limit while no member is bound.
• Delivery bound. MaxDeliver and AckWait are set on the consumer. ORE Studio sets ten deliveries and thirty seconds.

The delivery bound does not measure downtime. A push consumer with no bound subscriber receives nothing. The server leaves the message unprocessed on the consumer and spends no delivery on it. So a service that is down for an hour still reads its command when it returns, for as long as the stream keeps it. The delivery bound applies only while a member is bound and keeps failing. Each attempt waits AckWait for an answer, then the server redelivers. With ten deliveries at thirty seconds, that is about five minutes. A handler that asks for an immediate retry instead of waiting spends the budget faster, so a delayed retry is what makes the bound a time window.

When the last delivery also goes unanswered, the server stops delivering that message to that consumer. It does not acknowledge the message and it does not move it to a dead-letter queue. The message stays in the stream until max_age. The consumer that gave up never receives it again, even after a restart. A new consumer that reads from DeliverAll would still see it, but the service's stored durable is not new. Treat the message as lost to that consumer.

This is why the delivery bound must be explicit. A bound inherited from a server default is not the bound the code is written against, and the window it produces cannot be stated.

• Omitting the stream name. The consumer cannot be created by name on a stream, so the call refuses to run: a durable that the subscribe creates is removed on drain, and the next restart would miss every message published while the service was down.
• One durable name for two readers. They share a position and split the work between them.
• A durable for everything. A request whose answer is in the reply needs no history; durability is for messages that outlive the publisher's interest.
• An unbounded redelivery of a poison message. Without an attempt bound it is an endless loop.
• Reading the stream's age as the delivery promise. The age says how long the stream holds an unacknowledged message. It does not say how long a consumer keeps retrying. The delivery bound does, and it stops first.

Benefits

A restart costs no work, and the reader's position is durable state held by the server rather than the process.

Liabilities

The subscription now depends on server-side state, which a subject rename can strand. A message that exhausts its delivery bound stays in the stream but is no longer readable by the consumer that gave up.

The workflow engine reads `start_workflow` and `step_completed` from the `workflow` stream; IAM reads the tenant-provisioning step from `iam_steps`; the marketdata ingest loop reads ticks. The streams and consumers are catalogued in JetStream streams and durable consumers.

A step command is work, and the stream keeps it for a day. The engine may fail the step before that day ends, because a step has its own deadline. A command delivered after the deadline is answered and discarded by the claim; the stream kept it, but the work was no longer wanted. The day is a delivery backstop, not a promise that the work is still useful.

The subject a durable consumer reads needs a stream that covers it. `build/scripts/check_stream_coverage.py` fails when that stream is missing, and when two streams claim one subject.

• Queue-Group Subscription — the load-sharing half.
• Stream Provisioning at Start — the stream that has to exist first.
• Acknowledge What Was Handled — what a durable consumer must do.
• At Least Once Prerequisites — what has to be true around this consumer before its delivery is safe.

• NATS documentation, Delivery and acknowledgment — the delivery lifecycle and at-least-once.
• NATS documentation, Pull consumers in depth.

• NATS service patterns — the structure note that orders this pattern language, and where to read this page in it.