Task: Settle the redelivery bound
Table of Contents
This page documents a task in the Streams are not provisioned for the subjects services durabl… story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
Make the delivery bound of the durable consumers explicit, and make the three documents state the window the system actually gives: the stream holds a command while a service is down, and the consumer's ten ack-waited deliveries bound a service that is up and failing. The two are not the same window.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Streams are not provisioned for the subjects services durabl… |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-10-18 |
3. Acceptance
- The ack wait is a value the code passes to the consumer, not one the server chooses on the code's behalf.
- The documents state both limits, what the delivery bound does, and what happens to a message when it is spent.
- A live broker shows a command surviving a wait longer than the old bound.
- A test shows a stored consumer taking a changed bound in place, with its position intact.
4. Plan
- Read the settings from the running broker and find where the code sets each.
- Measure, on a live broker, whether downtime spends deliveries, and what the server does when the bound is spent.
- Decide from the measurements. Keep ten deliveries at thirty seconds, because the claim takeover is sized from those two numbers and a step command is work with a deadline. Make the ack wait explicit. Space the unreachable-engine retry so the bound is a real time window. State the real window in the documents.
- Prove it on a live broker and in the suite, and record the evidence.
5. Measurements
| What | Value | Command | Commit |
|---|---|---|---|
Step consumer AckWait / MaxDeliver on the fleet |
30 s / 10 | nats consumer info <step stream> <stream>_consumer |
ad5ccd8b9b |
Workflow engine consumers AckWait / MaxDeliver |
30 s / 10 | nats consumer info ores_dev_festive_dijkstra_workflow workflow-engine-start |
ad5ccd8b9b |
Server default AckWait / MaxDeliver when not set |
30 s / -1 (unlimited) | scratch nats-server 2.14.6, nats consumer add without the flags |
ad5ccd8b9b |
| Deliveries spent while no subscriber is bound | 0 | scratch push consumer, MaxDeliver 3, AckWait 2 s, wait 7 s |
ad5ccd8b9b |
| Fleet command after 330 s unbound | NumPending 1, NumAckPending 0, NumRedelivered 0 |
live fleet probe, MaxDeliver 10, AckWait 30 s |
ad5ccd8b9b |
| Consumer after its bound is spent | NumAckPending 0, stream message still present, rebind delivers nothing |
scratch probe, MaxDeliver 3, AckWait 2 s |
ad5ccd8b9b |
The raw command output is in redelivery_bound_live_broker.org.
6. Notes
MaxDeliverisstep_max_deliverinprojects/ores.service/include/ores.service/messaging/step_command_stream.hpp. The workflow engine's two consumers pass the same constant.AckWaitwas not set anywhere in the tree before this task. Every consumer inherited the server's 30-second default, whilestep_claim_leaseis written against that value.- When the bound is spent the server stops delivering the message. It does not
ack it and does not park it. The message stays in the stream until the
stream's age expires. The consumer that gave up never receives it again, and
a new consumer reading from
DeliverAllwould. - A push consumer with no subscriber spends no delivery. The stream's age is the only bound on downtime.
- The number ten is also the claim-takeover budget: nine hold-offs of 30 s must exceed the 120 s lease. Do not lower it.
- The unreachable-engine path used to NAK immediately, which could spend all ten deliveries in a burst. It now waits the claim hold-off, so the bound is about five minutes.
7. Test Scenarios
| Scenario | State | Notes |
|---|---|---|
8. PRs
| PR | Title |
|---|---|
9. Review
| Comment summary | File | Decision | Notes |
|---|---|---|---|
10. Result
The bound is kept and made explicit. Ten deliveries at a thirty-second ack wait is about five minutes for a bound consumer that keeps failing. It is not a downtime window: a push consumer with no subscriber spends no delivery, so a service that is down for hours still reads its command for as long as the stream keeps it (one day for a step stream).
The decision rejects widening the bound to the stream's age. A day of retries would be spent on work the engine has already failed at its deadline, and the immediate retry paths make a high bound a spin rather than a wait. It also rejects leaving the ack wait to the server default: the claim protocol is written against thirty seconds, so the code must pass it.
js_queue_subscribe and js_queue_subscribe_multi now take an ack wait and
write it to the consumer. ensure_durable_consumer reconciles it in place
beside the filter, the delivery bound and the delivery subject, so a stored
consumer keeps its position and its creation time. A live consumer whose stored
ack wait already matches is left as it is. A message the consumer already gave
up on is not pending, so an update does not revive it.
serve_step_commands and the workflow engine's two consumers pass
step_ack_wait (30 s). The unreachable-engine retry now uses the claim
hold-off rather than an immediate NAK, so ten deliveries really do span
minutes.
The test that exercised the update path for the first time found a latent
double free there: the code edited the server-owned jsConsumerConfig in
place, and then destroyed the consumer info whose string fields it had
overwritten. The update now copies the configuration, replaces only the fields
it sets, and destroys the stored record unchanged. This path had never run in
the suite before, because no test had changed a stored consumer's settings.
The documents now say all of this:
pattern_durable_subscription.org, pattern_stream_provisioning.org and
jetstream_streams_and_consumers.org.