Story: Model every output ORE writes

Every output family ORE writes is named in one catalogue, read by ores.ore from the samples checked in, and stored by the component that owns the data.

• A catalogue names every output family, the file parameter that names it, its key dimensions and its columns, and agrees with the samples checked in.
• ores.ore reads every results family and holds no store: no session, no table, and no mapping to another component's entity.
• Every run-result table in ores.reporting is fed by the post-processor, which calls the readers and does not parse a file of its own.
• Every market-state output is fed by ores.marketdata's ingest, which calls the readers.
• Every output family is parsed into rows in a table; no family is stored only as a blob.
• A report definition states which families its runs store as rows, and a new definition stores every family.
• A test reads every checked-in expected output and lands it in the model, or states why the file is out of scope.
• A run's outputs load in blocks, in one transaction for each run, so a refusal leaves nothing behind.
• One knowledge page states every family's file, key and columns with their types and units, generated from the catalogue so the page and the readers cannot drift.

Every output ORE writes is catalogued, read into rows and stored by the component that owns it. The catalogue names 81 families, each proven by a sample ORE ships, a run here wrote, or a fixture written from its writer, and the generator refuses a seed output that is neither a family nor excused with its reason. Run results land in reporting's metric, cashflow and text value tables; the scenarios, calibration and market inputs a run used land in marketdata's tables; every row is keyed by the result, file and line that state it. A report definition chooses the families its runs skip, storing every family by default. A run's rows load a block at a time in one transaction, which cut peak memory on a 500000-row cube from 1992 MiB to 202 MiB. ORE output formats documents every family, generated from the catalogue, and Report post-processing states the flow as built. A defect found on the way is captured, Every report run fails on an orphan commodity future part: since the conventions split, no report run reaches post-processing. The block load's wall clock is set by the per-row insert trigger, a limit recorded on its task.

The end-to-end check closed the loop. With the orphan part fixed, runs reach post-processing, and the run page shows each run's results, its errors and its files named as the ORE user guide names them. The tester signed off the NPV: on a fresh environment the group values all 73 of its trades. A first pass over every group and Hong Kong report stored valuation, curves and market, portfolio details and run statistics; the analytics that store nothing yet wait on the trade XML gaps, now the story Prove generated trades value against the ORE samples, and on the market data work.

• 2026-11-01: Every output family is parsed into rows, and the payload store is removed. This supersedes the 2026-10-30 and 2026-10-31 decisions that kept a market-state cube as one compressed payload. The user's requirement is that the parsers put all of a run's data into timeseries tables, because the intermediate state is what explains what the computation did: the NPV cubes explain an exposure and let XVA be aggregated again without a new simulation, the simulated factor values validate the model, the stress shifts explain a stress P&L, and the calibrated market explains a price. The volume concern is real, so it becomes the user's choice and not ours: a report definition states which families its runs store, and the default stores every family until we trust the parsers. The raw files stay in the object store, so a family turned off can be parsed later.
• 2026-11-01: The NPV cubes (rawcube, netcube) are run results, not market state, because each value is the NPV of a trade or a netting set. They go to ores.reporting. scenario.csv is sensitivity output (base NPV, scenario NPV, difference for each trade and shifted factor), so it is a run result too. The simulated factor values, the stress shifts and the calibrated market stay in ores.marketdata.
• 2026-10-30: The layering. ores.ore holds the format, the read model and one reader for each family, and no store: no session, no table and no mapping to another component's entity. The component that stores the data owns its tables and the mapping from the read model: ores.reporting for run results, ores.marketdata for market state.
• 2026-10-30: The post-processor lives in reporting and calls the readers directly. That gives reporting a core link to ores.ore.core, the same edge marketdata already holds, and the core-link baseline records it as a reviewed exception. The alternative — the compute node marshals every family — puts sixty-six formats in the node and leaves reporting unable to read what it stores.
• 2026-10-31: Three storage shapes, not one table for each family. A family's columns fall into three jobs: keys that identify the row, qualifiers that state its context, and measures that are its figures. About fifty families share one shape once the roles are known, so one metric hypertable takes them with the family code as a column; the three child-row families keep a typed table because their meaning is a relation between columns; the wide cubes stay payloads in marketdata. Sixty tables would be forty identical shapes and a UNION for every cross-family read.
• 2026-10-31: One reader driven by the declaration, not one reader for each family. ORE declares its columns imperatively and adds them in loops over the configuration — curves.csv adds a column for every curve the run built — so a family's shape is not a fixed list. The reader requires the keys, carries the qualifiers and turns each measure column into one metric named by its heading. A variant then costs nothing: an extra figure is another metric, an extra context column another qualifier, and a missing open measure is not an error.
• 2026-10-31: The 76 header variants are three different things. Configuration columns (curves, dim regression, var quantiles, scenario factors), analytic augmentation (the sensitivity analytic prepends Type, IsPar, Factor_1 and the shift to an exposure file), and one stale expected file: the 25-column flows.csv in MinimalSetup is a subset of the writer's 27. Only the first two are shapes the engine can write; the third is a corpus artefact, and the required-subset rule accepts it without pretending it is a variant.
• 2026-10-30: The screens keep one shape. A view over the family tables serves the run, level and compare reads, so no screen learns a family and the rollups are written once.
• 2026-10-30: Wide market state is market data, not a result, so it lands in ores.marketdata. A scenario cube is kept as one compressed payload with an index of what it holds, because a production cube holds millions of values and exploding it into rows would cost more than it answers.
• 2026-10-30: The node stops projecting npv.csv into canonical rows. The archive it already uploads is what the post-processor reads, so there is one writer of an ORE format and not two. The run's log rows keep their own file.
• 2026-10-31: A cashflow family is read by the same declaration-driven reader, and the three typed readers it replaces are gone. A typed row could not carry every figure a family states — flows.csv states ten, and the stressed family states each of them three times — and the table stores them all, so the reader reads each heading the catalogue calls a measure. The stressed suffix _Base, _Scen and _Diff becomes the part of the figure rather than part of its name, so a reader compares the three without learning it.
• 2026-10-31: The metric table carries the scenario and the risk factor as columns of their own, because the corpus states them by name and a screen filters on them, and it does not carry the shift: the sensitivity files state one shift column for each factor of the run, so the shift of a figure is not one field of its row. Every other qualifier travels as one JSON object.
• 2026-10-31: The screen reads a read model, not a second table. ores.reporting.run_result is a view over the metric table that names the engine's headings once — NPV is npv — and reads the counterparty out of the qualifiers. The result-row table it replaces held the same figures under the same names, so keeping it would have been two stores of one fact.
• 2026-10-31: A market-state cube is stored as the file the engine wrote, not as rows. stress_scenarios.csv states a column for every curve and tenor the run built — 829 in the family's union, 789 in the widest file — so a row per value would be millions of rows per run and a read that needs one factor would unpack all of them. The payload is one compressed blob with an index beside it: the headings, the row count, and the distinct values of each key the catalogue names, capped and flagged when a key states more.
• 2026-10-31: The index is the distinct values of the family's key columns, not a list of known column names. The catalogue already says which columns are keys, so the dates, scenarios, factors and market objects an index names are the keys of the family, and a family whose keys change needs no change here.
• 2026-10-31: The payload is held as base64 text rather than a binary column, because the store's columns are text and a binary column would be a second way to hold a blob. The compression happens first, so the cost is a third of the compressed size and not of the file.
• 2026-10-30: The tests read the checked-in expected outputs; the sample suite does not run. At this commit the corpus is 473 files under 192 file names, which group into 66 family shapes by name, and the widest file states 789 columns. The count is a grouping of the corpus, not a list ORE publishes.