Task: Post-process a run's results into rows

Table of Contents

This page documents a task in the Prove the report grid run end to end — the remaining work story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

After the compute results are retrieved, the rows the node marshalled become the figures the run's screens read, and the run states what was written.

2. Status

Field Value
State DONE
Parent story Prove the report grid run end to end — the remaining work
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-10-30

3. Acceptance

  • The run's workflow carries a post_process_results step after submit_compute, so the rows are loaded once the batch exists.
  • For each result of the batch the step downloads the canonical rows the node wrote, and reads them with one reader.
  • A refusal names the file, the header or the row that caused it, and loads nothing: a partial load would read as a complete one.
  • Every row of every result lands in one insert for the run.
  • The step answers the number of rows it loaded and a message naming the run.
  • The run's log rows are written once, before the rows are loaded, so a run whose rows are refused still states what the engine said.
  • Nothing in the step parses the run's archive; the rows come from the canonical file.

4. Plan

The step asks the batch's workunits for their results, checks each result's outcome, downloads each result's rows into a read_outcome, maps them onto ores.reporting.api::domain::report_result_row with the instance's party, definition and priced date, and inserts them once. The archive stays the run's output and is not read here.

5. Measurements

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. A number without both is not a measurement, because a clean-up record describes a tree and trunk moves under a branch: a consumer count, a gate's component count and a suite's size all change with no code changing, and the reader of a bare number cannot tell which tree it came from. Where the raw output matters, put it in an evidence/ file beside the task and name that file in the Command column.

What Value Command Commit
Rows one run loaded 24: 8 trades for each of npv, npv_base and notional compass sql -- -c "select measure, count(*) from ores_reporting_report_result_rows_tbl where report_instance_id = 'e31abdee-0dbe-404c-977d-80539b44fe17' group by measure"; the measurement is recorded on Run a report from its page and read its rows 3ffd149319

6. Notes

  • The description asks for "one reader per analytic". One reader serves every analytic, because the canonical file states its measures as rows rather than as columns; what is per analytic is the projection the node runs before upload. Only npv.csv has one today, so one analytic's rows load and 34 of the 35 seeded analytic codes would load none. Filed as Project each analytic's file into canonical rows.
  • The step records the run's account through record_run_logs before it loads the rows, so the ordering is a property of the step rather than of the archive.

7. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
Run a report through the grid and read every result PENDING The scenario checks the rows the step loaded against the run's figures.

8. PRs

PR Title
#3916 [reporting] Post-process a run's results into rows

9. Review

Comment summary File Decision Notes
       

10. Result

The run's workflow loads its figures after the batch is submitted. The step asks each workunit for its results, refuses a result whose outcome is not success, and downloads the canonical rows the node wrote for each one. One reader parses the file; a header it does not know, a field it cannot read or a figure it cannot parse is refused with the reason, and a refusal loads nothing.

The rows are mapped onto the run's party, definition and priced date and inserted in one call, and the step answers the count it loaded. The run's log rows are written first, so a run whose rows are refused still carries the engine's account.

The service no longer parses the run's archive for errors: the account comes from the rows the node wrote, which is where the figures come from too.

Emacs 29.3 (Org mode 9.6.15)