Failure, compensation and retry

A step command is published to a subject, and the engine waits for the service that owns that subject to report the outcome. Nothing in that path notices a service that has died, a command the bus dropped, or a handler that wedged: the step stays in_progress, the run stays in_progress, and every waiter eventually gives up on a run that never failed. That is the failure the deadline exists for, and it is the only one the engine cannot observe by itself.

Every step therefore states how long it may run, and the engine states the rule when a definition does not: a built step with no deadline is refused, and no run exists to wait on. There is no default, because the number depends on what the step does and one value would be wrong for both a write that answers in milliseconds and an import that publishes tens of thousands of rows. The definitions read their budgets from two names in the definition contract — data_step_timeout for work that publishes or imports, write_step_timeout for a write or a handshake — and a step whose work is neither states its own.

A run states more than the steps it waits for. A provisioning step hands its work to another run and waits for the answer, so its budget is larger than the budget of the steps inside it: the inner deadline is what fails, and the outer wait reports the reason instead of running out of its own patience first.

The budget travels with the run, in the snapshot the engine materialises at start, rather than being read from the definition when the deadline is checked. A run started under a longer budget keeps it, however the definition changes afterwards.

The engine reads the clock on a pass every thirty seconds, and fails any step whose command was published more than its budget ago. The deadline is measured from the dispatch, so what it asks is "how long has this dispatch been silent" and not "how old is this run". When the step was dispatched is the row's to state, because a restart, a retry and a recovery all rewrite it there: a retry that re-dispatches a step starts its silence again, and so does the recovery pass that re-dispatches whatever a restart interrupted.

The pass walks the steps the engine dispatched rather than every run the store holds. What it is looking for is work in flight, which is what the engine itself handed out, and a pass that read every run of every tenant would grow with the deployment's history rather than with the work in flight. The list decides nothing: the pass reads each entry's step row, which is the authority on whether the step is still running, so the worst a lost entry costs is a deadline that fires late.

One pass fails one step of one run, because only one step of a run is ever in_progress. A step the pass has failed is invisible to the next pass, so a pass that runs twice cannot fail the same step twice, and a completion that arrives while the pass is running cannot interleave with it: the pass and the step-completed handler hold the same lock.

The engine stops the run on a deadline, whatever failure policy its definition declares. A rollback is the right answer for a step that reported that it failed, because such a step did nothing; a step that went silent is a step whose work is unknown, since it may have died before writing anything or may be slow and about to finish. Compensating on that guess would undo a run whose work was in fact done, which is the worse of the two mistakes. So the run stops in failed with every completed step kept, and the decision belongs to a person: retrying is safe because a step is idempotent, and discarding is a separate action.

The engine cannot tell those two apart, and does not pretend to. The bus drops a command whose subject has no subscriber and says nothing about it, so from the engine's side both are silence; the deadline is what decides, and presence is what explains.

Every service publishes a heartbeat, and the engine keeps the last time it heard from each, keyed by the canonical service name. A step's command subject names its component as its first token — marketdata.v1.… belongs to ores.marketdata.service — so a deadline failure states what the engine saw: that nothing has reported as that service since the engine started, that the service was not reporting when the command was published and when it last reported, or that it was still reporting, in which case the command reached a service that did not finish the step. That is evidence and not proof: a service that is running but wedged reports for duty as well.

The failure contract starts with the step's own report. A handler that cannot do its work publishes a failed completion, and the run records that outcome on the step together with its error and its log. A successful step advances the run to its next step, while a failed step follows the failure policy its definition declares.

Compensation is the rollback of the run, and it belongs to the definitions whose completed work is a liability. The engine walks the completed steps in reverse and compensates only those the contract marks completed; a step that finished completed_with_warnings is not compensated. When the rollback finishes, the run reaches compensated and both its result and its error are cleared, which is the outcome the definition asked for when it declared compensate.

The engine keeps the run state failed for the paths that cannot proceed without a step report at all: an unknown workflow type, a step position out of range, a startup recovery that finds no steps, a definition that states no deadline for a step, and a failure while starting. A step that outlives its deadline is failed with the reason the engine recorded, and the run stops: see the deadline sections above.

The failure policy belongs to the definition, and it differs with what a completed step is worth. In a saga of writes — provisioning one party, say — a partial run is a liability, so undoing it is the safe answer, and such a definition declares compensate. In a tenant provisioning run the completed steps have published bundles of real reference data; undoing them costs minutes to hours and helps nobody, while retrying the failed step is cheap because every step is idempotent, so provisioning declares stop-and-retry.

Provisioning declares stop-and-retry: its steps are idempotent, a failed step stops the instance, the tenant stays provisioning and can be retried from that step, and Discard tenant is a separate system-admin action. A failed line therefore offers Retry from failed step and Discard tenant, because the completed steps are kept and retrying is safe.

The policy therefore stands on the definition, not on the engine. A definition declares it, and the engine reads it: a definition that declares stop leaves the run in failed with its error intact until a person acts, and every completed step untouched, while a definition that declares compensate keeps the rollback above.

A retry is a message, and the message resumes a failed run at its failed step. The verb carries the instance to resume and optionally names the step to resume from. It re-dispatches that step under the same step identity, restores the run's current step, and clears the run's error so that the failure no longer stands against the run.

A resume reads only what the run persists. The failed step keeps its position, its error and its log; the completed steps keep their results, which a later step's command builder reads by name; the run records how many steps it declared and which steps it declared at start; and each step carries its stable identity and its command. A retry clears the run's error explicitly, because clearing acts only on a value that is present.

The step identity must stay stable for a retry to be safe, and stability is the property the retry reuses. The engine ignores a completion for a step that is not in_progress, and a handler can ask the engine to replay the step's last outcome instead of repeating the work. A retry that re-dispatches the step under its own identity therefore runs the step once, however the messages interleave — see idempotency in a workflow step.

A deadline that fires on a service that was merely slow leaves the work done and the run failed, and the engine is explicit about both. The late completion is ignored, because the record the engine keeps is the one a person was already shown and a later report does not un-say it; the log says the answer came too late, which is a different thing from a duplicate that changes nothing. The run recovers in one retry, and the retry carries the step's own id, which is the idempotency key: the service that already finished replays the outcome it holds rather than doing the work a second time.

The window a person should know about is the one between the deadline firing and the retry: a retry dispatched while the original is still running gives one step two executions in flight, and the idempotency key deduplicates them only once the first has recorded its outcome. A step that can be interrupted must therefore be safe to repeat, and a step whose work cannot be repeated safely must finish well inside its budget rather than relying on the deadline.

A definition declares its failure policy. Provisioning declares stop; a saga of writes keeps compensate. The engine rests a stopped run in failed with the failed step left failed and every completed step untouched, which is what the progress read exposes.

The retry message re-dispatches the step under its own identity and leaves the completed steps alone, and that same identity is what makes the retry safe: the engine ignores a late completion for a step that is not in_progress, and a handler that replays instead of repeating runs the work once.

Discard is a separate action, and it must also stop the run: a delete of a tenant must leave no workflow instance of that tenant running behind it.