Report execution pipeline
Table of Contents
1. Summary
A report run is a sequence of phases. Pre-processing produces the input the engine will consume; submission turns it into work on the grid; execution runs the engine on a worker node; retrieval records where the output is; post-processing reads it; and finalisation closes the run. No phase calls another: the workflow engine invokes each step, and each step is handed the identifier or the location of what the previous step produced rather than its contents.
The definition's run configuration decides how much of that a run executes. Pre-processing and post-processing are each either executed or replaced by a prepared substitute, so a run executes the whole pipeline or a shorter chain. That is what lets the pipeline, and the workflow around it, be exercised before the trade and market data components can supply what pre-processing needs. The mechanism is described in Report execution substitution.
2. Detail
2.1. The phases
| Phase | Responsibility | Consumes | Produces | Substitutable |
|---|---|---|---|---|
| Pre-processing | produce the engine's input | the definition's configuration | one input artifact in object storage | yes |
| Submission | turn that artifact into work | the artifact's location | a batch and its workunits | no |
| Execution | run the engine | the engine package and the input | each output file and a manifest, and the output archive | no |
| Retrieval | bring the result back | the result identifier | the output location, recorded on the run | no |
| Post-processing | record what the engine produced | each result's manifest and the files it names | rows in the result and market state stores | yes |
| Finalisation | close the run | the outcomes above | the instance in a terminal state | no |
The four non-substitutable phases are the reason the two substitutable ones can be replaced at all. Submission, execution and retrieval are the same whatever produced the input and whatever will be done with the output, so a run that substitutes both ends still exercises the whole of the middle.
2.2. The phases as the workflow runs them
The pipeline decomposes into six phases; the workflow implements them as steps, and the two decompositions do not map one to one. Pre-processing is four steps rather than one, and retrieval and post-processing are one step rather than two. The steps are what a definition's run configuration actually selects.
| Run configuration | Steps that run |
|---|---|
pre_processing executes, post_processing executes |
gather_trades, gather_market_data, assemble_bundle, prepare_ore_package, submit_compute, collect_compute_results, post_process_results, finalise |
pre_processing substituted |
resolve_prepared_input, submit_compute, collect_compute_results, post_process_results, finalise |
post_processing ignored |
As above, with ignore_compute_results in place of collect_compute_results and post_process_results. |
2.3. What each step hands the next
No step calls another and no step hands on contents: each is invoked by the workflow and is given the identifier or the location of what came before it.
| Step | Does | Hands on |
|---|---|---|
gather_trades |
reads the trades the definition binds, as ORE XML | the trades object |
gather_market_data |
exports the tenant's series and fixings for the valuation date | the market data and fixings objects |
assemble_bundle |
records which objects this run reads | the bundle id |
prepare_ore_package |
builds the archive the engine reads, with the run document | the input archive |
submit_compute |
turns the archive into a batch and its workunits | the batch id |
collect_compute_results |
waits for the batch and records the run's log | the result ids and their output locations |
post_process_results |
reads the output and stores what it says | rows in the result stores |
finalise |
closes the run | the instance in a terminal state |
Post-processing is the one step whose work is proportional to the size of the result rather than to the size of the input, and it is the step that decides throughput. What it writes, and why the stores are shaped as they are, is stated in Report post-processing.
The chain is a pure function of the run configuration, because the engine rebuilds it on every dispatch and again when it recovers an interrupted run. A step addresses an earlier step's result by the name of the step that produced it, and a builder that cannot find what it consumes stops the run. The workflow itself is on Report execution.
Figure 1: One report run end to end: the trigger, the chain the workflow builds from the run configuration, and the messages each step uses.
Source: report_processing_pipeline.puml. Regenerate with
plantuml -tpng doc/knowledge/architecture/report_processing_pipeline.puml.
2.4. Pre-processing
Pre-processing turns a definition's scope into a document the ORE engine accepts. It reads the trades in scope and the market data those trades require, maps both into the engine's input files, and packages them into a single archive. Its interface is specified in Report pre-processing.
Its output is one object in storage, named by a digest of everything that determines it. Two runs over the same scope and the same market data timestamp name the same object, so the second finds the first's work already done. This is the only phase whose cost grows with the size of the data rather than with the number of runs.
2.5. Submission
Submission turns an artifact into work on the grid. It creates a batch and one workunit per job, records the package and the input, and leaves the grid to dispatch. Its contract is specified in Report compute handoff.
2.6. Execution
Execution happens on a worker node and is not a step any ORE Studio service performs. The node downloads the engine package and the input, runs the engine, uploads each output file as its own object with a manifest that names them, and uploads the output directory as one archive, the run's audit copy. The run's only involvement is to have made the work available and to wait, which is why execution is the phase most likely to outlast everything around it.
2.7. Retrieval
Retrieval records where the engine's output is. It does not read the output. Separating retrieval from post-processing means a run can complete knowing its result exists without paying to parse it, which is what makes the post-processing substitute meaningful: the output is genuinely optional.
2.8. Post-processing
Post-processing reads the files each result's manifest names and records what they contain. Its design is specified in Report post-processing. It is the only phase whose work is proportional to the size of the result rather than to the size of the input.
2.9. Finalisation
Finalisation closes the run. It is what moves the instance to a terminal state, and it is the only phase that touches the state machine. Keeping it separate means the state transition happens once, in one place, whatever the phases before it did.
2.10. What every phase may assume
- The previous phase's artifact is complete. A phase is invoked only after its predecessor reported completion, and the artifact it is handed is either whole or absent. No phase reads a partially written object.
- No phase holds another's data in memory. Phases are separate invocations and may be served by different processes. Everything passed between them is an identifier or a storage location.
- A phase may run more than once. Dispatch does not guarantee exactly-once delivery, so every phase must converge when repeated rather than accumulate. The consequences are set out in Report execution orchestration.
- A phase may be replaced. A phase that is substitutable must be defined so that its replacement is indistinguishable to the phases around it.
2.11. What a phase must leave behind
Every phase records enough for two readers: the next phase, which needs a location or an identifier, and an operator, who needs to know what happened. A phase that succeeds silently and a phase that fails silently are both defects; the run's record must name the phase, its outcome and the artifact it produced.
3. See also
- Report Processing — the structure note that orders this cluster, and where to read this page in it.