Story: Prove the report grid run end to end — the remaining work

Table of Contents

This page documents a story in Sprint 29, carried unfinished from Sprint 28 at close. It captures the goal, current status, acceptance criteria, and the tasks that compose it.

1. Goal

The ACME report definition runs to completion against the compute grid from an input bundle that already exists, the compute results are retrieved, and the run is reproducible from a freshly recreated database.

2. Status

Field Value
State DONE
Parent sprint Sprint 29
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-11-08

3. Acceptance

The story closes on the tasks it finished:

  • Add the report result rows table.
  • Ask a run's health from the shell.
  • Compare a run with an earlier one.
  • Complete the result row's generation.
  • Draw the plots the run can answer.
  • Group books under their portfolios.
  • Marshal the engine's output into result rows.
  • Nest the portfolios into a tree.
  • Post-process a run's results into rows.
  • Prove the run end to end and keep it repeatable.
  • Read a report's runs and one run's results.
  • Read a run's errors and its job chain from the shell.
  • Read a run's results at trade, book or counterparty level.
  • Read a run's results in the reporting area.
  • Read the trade a run failed on, at its version.
  • Read a trade from a run's results.
  • Record the date a run priced at.
  • Record the trade version a run read.
  • Restore the engine log work.
  • Roll a run's counterparties up to their groups.
  • Run a report from its page and read its rows.
  • Say the run failed in one sentence.
  • Seed ACME with the report type, the bundle and the definition.
  • Store a run's errors once, and keep the rows the vocabulary names.
  • Work out a run's error cause on the server.

4. Tasks

Task State Start End Description
Prove the run end to end and keep it repeatable DONE 2026-10-24 2026-10-25 Task for: Run an example bundle through the compute grid from a report definition
Seed ACME with the report type, the bundle and the definition DONE   2026-10-25 Task for: Run an example bundle through the compute grid from a report definition
Read a run's errors and its job chain from the shell DONE   2026-10-25 The shell could not read a run's errors at all. The generated report_error_event and report_error_scope command units existed but nothing registered them, so every recipe for a run's reasons aborted with 'Wrong command', and the reporting service subscribed only the counts read, so even a registered command would have reached no responder. The other job and run reads were already reachable: report_instances list and get, batches, workunits by-batch-id and results by-workunit-id.
Work out a run's error cause on the server DONE   2026-10-25 The run page decided what an error was about by reading the engine's message in TypeScript: it took the curve name out of IndexCurve(…), matched trades to curves by substring, truncated sentences and chose a label per scope. The shell could not do any of it without a second copy, which is how the duplication showed.
Read the trade a run failed on, at its version DONE   2026-10-25 The error screen links a failed trade to /trades?trade=<id>&version=<n>, and the trades page reads neither, so the link lands on the unfiltered blotter. Make the trades list open the trade the link names, at the version the run read, and say when that version is no longer current. The page must ask the server for it rather than filter a list it already fetched.
Ask a run's health from the shell DONE   2026-10-25 A run's health is a server-side read that the BFF calls and the shell cannot: the operations menu has the trigger and the gather verbs but no health verb, and the run entity carries state rather than health. Serve the health read under the report operations menu, generated the same way as the other operations, so the shell and the web ask the same question.
Store a run's errors once, and keep the rows the vocabulary names DONE   2026-10-25 The step that reads a run's output runs at least once, and it wrote its rows every time, so a redelivery stored the same reasons twice. One engine scope the vocabulary did not name made it return before writing anything, so a single unrecognised group cost the run every reason it did name. Writing is now guarded by the run, and an unknown scope costs only its own rows, which are said as a warning on the run.
Say the run failed in one sentence DONE   2026-10-25 The row that says a run failed carried the whole engine account, so the errors screen showed the same five sentences twice: once on the Report row and again on the rows that name each reason. The row now says the failure in one sentence and leaves the reasons to the rows beside it. The instance keeps the engine's full account.
Record the date a run priced at DONE   2026-10-25 The run document is written for the date the run asked for, and the package handler moves it to the date the gathered market data carries when the two disagree, with only a warning in the log. Nothing on the run says which date it used or why it moved, so no client can say it. The report instance gains priced_at and price_note, the package handler reports the date it used through a reporting operation, and the reporting service writes it on the instance.
Record the trade version a run read DONE   2026-10-25 The error screen links a failed trade at a version, and the error rows recorded no version, so every reference said v?. The export carries each trade's version, and the bundle now keeps the manifest the gather received. The recorder looks a trade up in that manifest and writes subject_version, with subject_as_of set to the date the instance priced at. A trade missing from the manifest records with both null rather than failing.
Read a run's results in the reporting area DONE   2026-10-25 The report page gains Definition, Runs and Results tabs, and the results are read from the run's output archive.
Read a report's runs and one run's results DONE   2026-10-25 The report page keeps the definition first and lists the runs beside it; a run's details and results live on the run's own page.
Read a trade from a run's results DONE   2026-10-26 Each result row names its trade and its counterparty, links to both, and opens onto that trade's figures and the error the run recorded against it.
Read a run's results at trade, book or counterparty level DONE   2026-10-26 One control chooses the level a run's results are read at, and the server does the aggregation.
Roll a run's counterparties up to their groups DONE   2026-10-26 The counterparty level states each group's rolled-up figure above its members, so a group can be checked against the parts beneath it.
Compare a run with an earlier one DONE   2026-10-26 A run's aggregate levels can be stated against an earlier completed run of the same report, with each row's change and share.
Group books under their portfolios DONE   2026-10-26 The book level states each portfolio's rolled-up figure above the books it holds.
Nest the portfolios into a tree DONE   2026-10-26 The book level reads as a tree: a global portfolio over banking books over the ports and books beneath them, each row rolled up from what it holds.
Draw the plots the run can answer DONE   2026-10-27 The results screen draws a donut of each book's share of the run, and a donut of the run's problems by scope, each with its figures beside it.
Add the report result rows table DONE 2026-10-27 2026-10-27 A hypertable of one row per subject, measure and currency at the run's priced date, with the security, indexes and retention the market data ticks use.
Post-process a run's results into rows DONE 2026-10-27 2026-10-27 A post-processing step after retrieval, one reader per analytic, rows written in one transaction per run, and the run stating what was written.
Marshal the engine's output into result rows DONE 2026-10-27 2026-10-27 The compute wrapper writes one canonical result file from the engine's own files, a projection of their headings rather than a parse, before it uploads.
Run a report from its page and read its rows DONE 2026-10-27 2026-10-27 A report definition runs on demand from its own page, the run is followed to the grid and its jobs, and the results the screen draws are the rows the post-processor loaded rather than the archive it used to parse.
Complete the result row's generation DONE 2026-10-28 2026-10-28 The result row entity's generated layers were never committed, and its repository was hand-edited to add read_by_run. Regeneration now wins: the layers are committed, the run read goes through the generated filter, and the read is paged and permission-checked.
Restore the engine log work DONE 2026-10-28 2026-10-28 Four commits that read the engine's log in one place, wrote it beside a run's rows and loaded it in reporting were merged as part of #3929 and are not on main: the branch that PR merged was rebuilt and left them behind. They are restored here, adapted to today's tree, before the duplicate parser is deleted.

5. Decisions

  • Split from Prove the report grid run end to end on 2026-10-24 at sprint 28 close. The acceptance above is copied from that story; its done tasks are recorded there.
  • 2026-10-24: The proof runs on this environment's five local wrapper hosts. The LAN hosts belong to other environments and were one and two releases behind main, so a run there would prove a different tree. The version table is recorded on Prove the run end to end and keep it repeatable.
  • 2026-10-24: The acceptance bullet about a failed job was not true when the run was first made. The engine's bookkeeping files counted as analytics, so a failed analytic was recorded as Success. The verdict now reads the engine's own Error in ORE analytics line, and the same broken package fails as it should.
  • 2026-10-24: Seven of the run's eight bonds have no reference data the engine can build, so they price nothing. The chain is correct and the run is honest about its output; the gap belongs to the instrument export, not to this story.
  • 2026-10-24: The acceptance bullet about a stored assignment whose result is gone is already met by an earlier decision, not by this run. Sprint 28 retired the assignments stream and the wrapper claims work from the ledger, so no assignment outlives a recreate. That task was abandoned for the same reason.
  • 2026-10-24: The proof rides with a test-harness fix. Local test runs share the NATS server with a running fleet, so the suites now use a <prefix>.tests namespace and a fleet cannot answer their announces. The streams that namespace creates are not purged yet; that is captured in Purge the streams a test run creates.
  • 2026-10-27: The run's results are read from the table by run, and no request path parses the archive. Merged as #3929 with four review points open: the read has no bound, four filter members are ignored, no read permission is checked, and the BFF keeps dead tar constants. They are recorded on Run a report from its page and read its rows.
  • 2026-10-27: The report_result_row entity generates twelve layers that the branch did not commit, because the generated CRUD service cannot serve a hypertable whose key is (id, as_of). check_component_drift.py --component reporting therefore fails on main, and the drift job did not see it because it does not watch projects/ores.reporting/modeling/. The model must state the layers it does not generate, or the job must watch the model.
  • 2026-10-30: The decision above no longer holds. The generated layers are committed, and check_component_drift.py --component reporting reports no drift at 55e53d5c56. The drift job watches projects/*/modeling/**.
  • 2026-10-30: The table, the node's projection and the post-processing step all landed inside #3916, and the PR was recorded on the post-processing task alone. The three tasks are closed now. The load-bearing fact — only npv.csv is projected, so a run of any other analytic loads no rows however many analytic codes are seeded — is written on the two tasks it changes.
  • 2026-10-30: The result rows table keeps its rows indefinitely. It declares no :hypertable_retention_interval:, and no Timescale retention job runs on any table in this environment. The interval is not settled, and it is filed as a capture rather than claimed by a task that is closed.
  • Closed on 2026-11-08 at sprint 29 close with its done tasks. The open tasks and the original acceptance moved to Prove the report grid run end to end — the remaining work.

6. Out of scope

Emacs 29.3 (Org mode 9.6.15)