Task: Prove the run end to end and keep it repeatable

Run the ACME definition against a freshly recreated database, record the instance's FSM walk, the compute job and the retrieved output, and keep the run as a check that can be repeated.

• The instance travels pending, running and completed through the FSM guard, witnessed on a database built from nothing.
• The evidence is recorded on the task, including the compute job and where its output landed.
• The run is repeatable from the recorded steps.

Run the chain the way an operator does: recreate the database, provision it from acme_demo, trigger a definition the session's party owns, and watch the instance, the batch and the result. Every step that could differ across environments is read from the database rather than assumed, because the previous attempts failed on data shape rather than on code.

The chain was found sound where it was suspected: the config lookup resolves the binding to the risk configuration row before resolving scope, so the scope returns books. The investigation that mattered was the failure path, described under * Notes.

The repeatable artefact is a recipe, not a bespoke script, because the run needs prose steps between the commands: How do I prove a report runs end to end?.

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. The data-shape numbers below were measured at 8a7478393e; the run outcomes and engine time were measured on this branch, and the rows behind them are named under * Notes.

What the chain does, confirmed. gather_trades resolves the definition's binding to the central configuration row, then reads the risk_report_configs row whose configuration_id holds it, then resolves scope with the risk configuration id and the definition id. Joining on the central id returns no scope for every definition, which is what made the first reading look like a failure. It was a wrong join, not a defect.

A working run. Instance 97e9d3c2 walked pending, running, running and completed in nine seconds. The batch closed with one workunit, the result had outcome 1 and server state 5, and the output archive held log.txt, runtimes.csv, pricingstats.csv, counterparties.csv and the tradewise analytics. The engine logged Built Portfolio. Initial size = 8, size now 8, built 0 failed trades, so the package the chain builds is ORE-readable and the engine proceeds past Build Portfolio.

The defect the proof exposed. A job whose engine failed was recorded as Success. ORE writes runtimes.csv, pricingstats.csv, log_structured.json and todaysmarketcalibration*.csv whether or not the analytic it was asked for ran, and judge_engine_run treated any file other than log.txt as evidence that the run worked. It returned success before it ever read the log. The same broken package that returned outcome 1 before the change returns outcome 3 after it, carrying ORE's own line:

The engine's analytic failed. The engine reported:
  Error in ORE analytics: Failed to run analytics PORTFOLIO_DETAILS

The fix reads the engine's own verdict: a line whose message carries Error in ORE analytics fails the run whatever else it wrote, while a run that wrote analytics and logged no such verdict still succeeds. The file-based rule stays for the case where nothing but the log was written.

The job behind the evidence. The last run on this branch, after the fix, left these rows. They are the ones a reader can look up.

The gap that remains. Seven of the eight bonds in the run have no reference data the engine can build, so the engine warns and prices nothing for them. The chain completes and the run is honest about its output, but a pricing analytic will produce no numbers until the bond reference data reaches the package. This is recorded here rather than fixed, because it belongs to the instrument export rather than to the chain.

Where it runs. The proof uses the five local wrapper hosts of this environment. The LAN hosts serve other environments and were a release or two behind, so a run there would prove a different tree. They are not part of this environment's grid, so they are not part of this proof.

The run works end to end, and the failure path now reports failure.

• A recreated database is runnable after one provision. The recreate leaves schema and staged artefacts only. bootstrap setup ... --profile acme_demo publishes 78 report definitions, 46 portfolio scope rows and both ORE engine packages. Every one of the 78 definitions then resolves to books, through the risk configuration row rather than the central one.
• Four report runs completed through the grid. Each walked pending, running and completed, each left reporting/runs/<instance>/output.tar.gz in storage, and each closed a compute batch whose external reference is the instance id, with outcome 1 and server state 5.
• The engine reads the package the chain builds. ORE logged Built Portfolio. Initial size = 8, size now 8, built 0 failed trades and ran PORTFOLIO_DETAILS for ten seconds.
• A failed analytic is now a failed result. judge_engine_run read the files alone, and ORE's bookkeeping files made every failed run look successful. The function now fails a run whose log carries the engine's own Error in ORE analytics verdict. The same broken package returns outcome 1 before the change and outcome 3 after it, with the engine's message on the result, and a working run still returns outcome 1.
• The run is repeatable. How do I prove a report runs end to end? records the steps and its script tangles into the shell library. The generated script, with the two ids substituted, ran an instance to completion, and a fourth run confirmed the working path after the verdict fix.
• Re-verified after review. The branch was rebuilt, the database recreated and provisioned again. Instance 25922e99 completed in ten seconds with outcome 1, and the package with no portfolio, dispatched as batch e2e-broken-portfolio-5, returned outcome 3 carrying "The engine's analytic failed." The rows the earlier runs left behind were replaced when the gate recreated the database, which is captured above.