Story: Gather a run's inputs by declared dependency

Table of Contents

This page documents a story in Sprint 27. It captures the goal, current status, acceptance criteria, and the tasks that compose it.

1. Goal

A step declares the artifacts it consumes and produces, and a run's chain is held as a persisted dependency graph that the engine dispatches from.

2. Status

Field Value
State DONE
Parent sprint Sprint 27
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-10-13

3. Acceptance

  • A step definition names the steps whose results it reads, and the declaration travels with the run.
  • The engine refuses a chain that names a step no earlier step produces, and fails a run naming the inputs it is waiting on.
  • The run's chain is two relations, a step per position and an edge per dependency, and materialised_steps_json is gone.
  • The engine dispatches every step the graph says may run, completes a run when every step has answered, and reconstructs the graph from the database after a restart.

4. Tasks

Task State Start End Description
Declare what a step consumes and produces DONE 2026-10-08 2026-10-08 A step is dispatched by its ordinal position and nothing else, so nothing states what it needs before it can run or what it leaves behind for the next step to read. Give a step a declared set of input artifacts and output artifacts, and make the engine refuse to dispatch a step whose inputs do not exist. This is the declaration half: without it the engine cannot know whether a run is waiting on work or is simply broken.
Hold a workflow as a dependency graph DONE 2026-10-08 2026-10-08 The chain is a std::vector built in code and the instance stores each step's ordinal, so the only edges are the ordinals. Persist the graph with the run — steps as vertices, artifact dependencies as edges — and reconstruct it as a boost::adjacency_list for dispatch, the way ores.dq's publish_service already orders datasets. A step with several producers then has a defined completion condition rather than an ordinal that no longer describes it.

5. Dependencies between the tasks

Two things gate everything: the artifact declaration and the dependency graph, because the first says what to wait for and the second says whether it has arrived. The batcher is the third, because the batch keys it produces are what the producers are named by. 

trade sets in ores.trading (Merry Newton)          [outside this tree]
  └─ partition a run's trade population into trade sets

declare what a step consumes and produces
  └─ hold a workflow as a dependency graph
       ├─ partition a run's trade population into trade sets
       └─ assemble the package once every input is present

batch keys
  ├─ export a trade set as an archive
  │    └─ map a trade set's archive into the ORE portfolio
  ├─ resolve a trade set's market data requirements
  │    └─ export the market data a trade set requires

hand each configuration to its owner by id at an as-of   [per run, not per set]

The producers are independent of each other, so they can be built in any order once the batch keys exist. What they cannot do is wait for each other: only the graph can say whether a run is still gathering.

Configurations are the exception to the fan-out: a configuration belongs to the definition, not to a trade set, so that task is independent of the batcher.

6. Decisions

The remainder is carried in Gather each trade set and its market data.

  • A run's chain is a relation, not a document. The chain a run was started with was a JSON snapshot on the instance, because build_steps is a function of the start request and the engine must not rebuild it after a restart. The other *_json columns in this schema are payloads — opaque wire messages a service must not interpret — and storing those verbatim is right. A chain is not a payload: the engine parses it, orders it and asks it which step reads which, which is a relation stored as a document and the reason the schema could not answer the graph's questions in SQL. The chain is settled as two relations instead — a step per position and an edge per dependency — and the document, the step count and the dispatch cursor go with it. Owned by Hold a workflow as a dependency graph.

Open, to settle inside the tasks. Each is recorded with the task that owns it, and none is closed:

Blocked outside this story. Trade sets do not exist yet: they are being built in ores.trading by Merry Newton. The batcher task is blocked on that landing, and the rest of the story is blocked on the batcher, so the critical path starts outside this tree. What the batcher needs from a trade set is small and stated on the task so the two efforts agree early: a name, an as-of, and the trades in it.

The as-of is cross-cutting. Every market and trade table is bitemporal, so an as-of read is a valid_at read and needs no new machinery — but the run document's asofDate is currently passed to nothing, and every producer has to be handed it for the same run to be reproducible. The story's acceptance says every gathered artifact is reproducible for a stated as-of; that is the test.

What this story does not do. It does not change the compute side. The package the run assembles, the assignment that carries it and the node that runs it keep the shape the storage-capability work gave them.

7. Out of scope

  • The compute side: the package, the assignment and the node keep the shape the storage-capability work gave them.
  • The near-term portfolio unblock, which maps today's blob so the engine proceeds before the archive is redesigned: Export the trades as the ORE portfolio the engine reads. The reverse mappers are shared, so whichever lands first builds them.

Emacs 29.3 (Org mode 9.6.15)