ORE output processing

Every output is a CSV whose first line names the columns, written through one interface:

// OREData/ored/report/report.hpp
virtual Report& addColumn(const string& name, const ReportType&, Size precision = 0,
                          bool scientific = false) = 0;
virtual Report& next() = 0;
virtual Report& add(const ReportType& rt) = 0;   // positional, in column order
virtual void end() = 0;

ReportType is a variant of Size, Real, string, Date and Period. CSVReport writes a # then the headings, then the rows. There is no column count check, no type check and no registry of what a name means.

The per-family knowledge lives in writer functions — 115 column blocks across 39 files - and it is imperative. curves.csv shows the shape of it:

// reportwriters/pricingreportwriter.cpp
report.addColumn("Tenor", Period()).addColumn("Date", Date());
for (auto it : discountCurves)  report.addColumn(it.first, double(), 15);
for (auto it : yieldCurves)     report.addColumn(it.first, double(), 15);
for (auto it : indexCurves)     report.addColumn(it.first, double(), 15);

So a family is not a fixed list of columns. The other analytics take the same sink and prepend their own columns: the sensitivity analytic adds Type, IsPar, Factor_1, Factor_2 and the shifts before it writes an exposure_trade_<id>.csv.

File names come from the run's parameters in most cases — npv.outputFileName, stress.scenarioOutputFile — which our file_type seed already catalogues. The rest are conventions of the engine, such as exposure_trade_<trade id>.csv and simm.csv, and no parameter names them at all.

A figure the engine has no value for is written as its null string (#N/A in the samples), not as an empty cell.

The checked-in expected outputs show 76 header variants across 60 families, and they are three different things:

Only the first two are shapes the engine writes. The third is an artefact of the corpus, which is worth knowing before treating a variant as a case.

The repository vendors ORE's samples under external/ore/examples/*/ExpectedOutput. At this writing they hold 473 CSV files under 192 file names. The catalogue those files generate states 60 families and 1 584 columns: 1 250 decimal, 280 text, 36 date and 18 integer. The widest file, stress_scenarios.csv, states 789 columns.

The file_type seed declares 55 output parameters and the samples ship the file of 17 of them. The other 38 name files no sample carries — jacobi, crossgamma, calibration, creditcurves, crif, the scenario dumps and cubes, dim evolution and regression — so their shape can only be read from the writers.

The layering is fixed before any reader is written:

• ores.ore holds the format, the read model and one reader for each row shape. It holds no store: no session, no table and no mapping to another component's entity.
• The component that stores the data owns its tables and the mapping: ores.reporting for what a run produced, ores.marketdata for market state.
• The post-processor calls the readers. That gives ores.reporting.core a core link to ores.ore.core, the same edge ores.marketdata.core already holds.

The catalogue is generated from the corpus by projects/ores.ore/scripts/generate_output_catalogue.py, and the ore-output-catalogue local check fails when the corpus and the data disagree. Each family states the roles of its columns — keys that identify a row, qualifiers that state the context of a figure, and measures that are the figures — and whether its measure set is open.

Reading follows from the roles:

• One metric reader serves every family whose row is keys, qualifiers and figures. It requires the keys, carries the qualifiers, and turns each measure column into one metric named by its heading. A curve, a risk factor and a quantile are therefore all measures, and a new one costs no code.
• Typed readers serve the three families whose row is a child of a trade: flows, stresstest_flows and todaysmarketcalibration_cashflows. There the meaning is the relation between the columns — trade, leg, cashflow number and pay date — and flattening it to metrics would lose the flow.
• Payloads serve the wide market state. A scenario cube is stored as one compressed object whose index names the dates, scenarios and factors it holds, because exploding 789 columns into rows costs more than it answers.

The stores follow the same three shapes: a metric hypertable with the family as a column, a cashflow table, and a view that keeps the one shape the screens read, so no screen learns a family. Sixty tables would be forty identical shapes and a UNION for every cross-family read.

The full inventory and the task list behind it are on Model every output ORE writes.

The 38 output parameters whose files no sample ships are not in the catalogue, because the corpus cannot state their shape. They are filed as Read the outputs the samples do not ship.