Task: Hold a workflow as a dependency graph

The chain is a std::vector<workflow_step_def> built in code, and the instance stores each step's ordinal, so the only edge in the graph is the ordinal. That is enough while every step produces exactly one artifact and the chain is a line, and it stops being enough the moment a step gathers per trade set: an ordinal says which step is next, not what a step is waiting for.

Persist the graph with the run — steps as vertices, artifact dependencies as edges — and reconstruct it as a boost::adjacency_list to decide what is dispatchable.

• A step's dependencies are persisted with the run and read back at dispatch; the ordinal alone no longer decides what runs.
• The engine dispatches a step only when its declared inputs exist, whatever position it holds.
• A consumer of an artifact that N producers write runs only once all N have finished, and runs once, not N times.
• A run in progress can be resumed after a restart by reconstructing the graph from the database, not from the code that built the chain.

Vertices are steps; an edge runs from the producer of an artifact to each of its consumers. The ready set is every vertex whose incoming edges are all satisfied, which is what the engine dispatches from in place of the ordinal.

The precedent is in this tree: ores.dq's publish_service.cpp builds a boost::adjacency_list over datasets and their declared dependencies, orders it with boost::topological_sort, and publishes in that order. The same shape serves here, and a reviewer already knows how to read it.

Two things must survive the change. Compensation is a saga walk of the satisfied prefix in reverse, so it needs a defined order from a structure that is no longer a line; and recover_in_progress must rebuild the graph from the database, which is what makes persistence the point rather than a convenience.

Where the edges are persisted is a decision to take with the schema in hand: an edge table keeps a step's dependencies open-ended, while a depends_on column on the step keeps the read simple and would have to become a list. The edge table is the one that survives a step with two producers without a second migration.

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. A number without both is not a measurement, because a clean-up record describes a tree and trunk moves under a branch: a consumer count, a gate's component count and a suite's size all change with no code changing, and the reader of a bare number cannot tell which tree it came from. Where the raw output matters, put it in an evidence/ file beside the task and name that file in the Command column.

Landed so far. The graph itself, and the engine deferring to it.

workflow_graph holds a run's steps as vertices and an edge from a step to every step that reads its result, built with boost::adjacency_list and queried through in_edges. It answers two questions: whether the chain can be run at all, and which steps may run given what has answered.

It is rebuilt from the run's own materialised_steps_json, not from the definition as it stands now, so a definition edited after a run started cannot change what that run was waiting for. That is the persistence the task asked for: the graph travels with the run and a restart reconstructs it from the database.

Three chains are refused before a run exists, and each is a definition that cannot be executed rather than a state a run can reach: a consumer of a name no step produces; two steps with one name, because a consumer cannot say which it reads; and a consumer of a step the engine reaches only later.

That last refusal is deliberate and worth naming. The graph derives an order and would run a chain declared the other way round; the engine advances along the declaration, so it would dispatch the consumer first and find its input had not run. A chain the engine cannot walk is refused up front rather than left to fail at the step that discovers it. Dispatching from the graph's own order is what lifts the restriction.

At dispatch the engine asks the graph whether the step it is about to take is ready, and fails the run naming the inputs it is waiting on when it is not.

What the task still owes. Two things, and both are the same change.

The advance loop still walks the chain by index and dispatches one step per completion. Until it dispatches every ready step instead, a consumer of many producers — the fan-in the gathering redesign needs — is refused rather than made to wait, because nothing reconsults the graph when a producer answers. The readiness rule is already the fan-in rule: it waits on every producer that writes an input, proven by a test with two batch-keyed producers. What is missing is a loop that acts on it.

Re-pointing the engine at the two relations is done. The graph is drawn from the plan rows and their edges. The deadlines and the step words a screen shows come from the same rows. materialised_steps_json is gone from the model and from every generated artefact.

One thing about the write is worth recording, because the symptom points somewhere other than the cause.

The chain's rows carry tenant_id. Written with the engine's own tenant, the plan step's insert is refused by its own soft-FK check:

Invalid workflow_id: <id>. No active workflow instance found with this id.

The cause is not the row-level security policy, and the row was never hidden. The trigger's first statement rewrites the tenant it was handed:

NEW.tenant_id := ores_iam_validate_tenant_fn(NEW.tenant_id);

That function maps the system tenant to whatever the session says. Its own comment states the rule: Treat system tenant ID (nil UUID) as placeholder meaning "use session tenant". The engine holds the system tenant while it starts runs for everyone. So a row written with the engine's tenant becomes a lookup for instances under the system tenant, and the run's own instance carries the run's tenant. The lookup finds nothing.

Measured on one instance, in one session, with only the written tenant varying:

The execution rows insert in the same run because they already take instance.tenant_id. The engine states that three screens above the write, in a comment on the execution step.

So the caller was wrong, and the engine already held the answer. Each row of the chain takes the instance's tenant. Nothing in a policy, a check or a generator changes.

The rewrite inside ores_iam_validate_tenant_fn is a pre-existing compatibility affordance, and it is not the fault. A caller that states the wrong tenant must be corrected whether or not the function hides it. Removing the placeholder would change every generated insert trigger, so it is not part of this task.

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Delivered and merged as 2783.

A run's chain is two relations rather than a document. workflow_plan_step holds one step of the chain the run was started with and workflow_plan_dependency holds one edge, so "what is waiting", "what may run now" and "what rolls back, in what order" became queries over rows instead of a parse. materialised_steps_json left the model, and regeneration removed it from the domain, protocol, entity, mapper, service, shell commands and TypeScript bindings with it.

The engine dispatches every step the graph says may run rather than one per completion. A run completes when every step of its chain has answered – read from the chain, not counted from steps taken – and a run that can neither run a step nor has one in flight is failed rather than left in progress for ever. current_step_index went with the ordinal it belonged to; step_count stayed, because it is the chain's length fixed at start.

Declaring what a step reads stopped being optional: a chain that declares nothing now runs flat. So every definition that meant one step after another says so – identity (a sequence of outcomes), bundle publishing (display order), and the provisioning chain, which states its real dependencies, a kind's needs sitting beside its words and its budget, with a need the run did not order answered by the nearest step it did.

Measured on a live tenant bootstrap from acme_demo: import_lei_hierarchy and start_market_feeds were published in the same second, while provision_party did not yet exist. One completion made two steps ready.

What this did not settle. The engine asks the graph what may run, but a step's command is still built by the definition's own build_steps, so the rows are the authority for the graph and not yet for the builder, and a non-deterministic definition could still build a different chain than the run holds. Making the rows the only authority is the next slice.