Report post-processing

The run's compute results, one for each workunit. Each result names a manifest, which lists every file the engine wrote with its size, its digest and where it is. The step also holds the run's own token, which is the authority to read them, and the families the run's definition said not to store. It is handed locations, never contents.

Nothing in reporting or the market data store parses an ORE file: the readers belong to ores.ore, and the components call them.

The catalogue says which family a file name belongs to, under its own name or the name the engine uses when the run's configuration names none, and the family says which shape its row is. One rule decides, stated once in ores.ore:

A file no family covers is counted and logged rather than refused: a run may write a file this version does not model, and the run's archive still holds it. Every output the engine's configuration can name is either a family or is stated, with its reason, on the formats page; the catalogue generator refuses one that is neither.

ores.ore declares each family's columns and their roles — key, qualifier, measure — and derives its readers from that declaration rather than writing one for each family:

• read_metrics turns each measure column into one figure named by its heading. A curve, a risk factor and a quantile therefore need no reader of their own, and a family that gains a column for each curve costs nothing.
• read_cashflows reads a family whose row is a payment. The stressed families state each figure three times; the reader turns Amount_Base, Amount_Scen and Amount_Diff into one figure named Amount with the part beside it.
• read_text_values reads the families that state a name and a text: the extra results a pricer reports, the lists of a portfolio's details, and the calibrated model as one XML document.
• read_calibration reads the calibration of the market the run built, one value for each line, keeping the text and the number beside it when there is one.
• read_market_inputs reads the quotes, fixings and dividends the run loaded.

The metric and cashflow readers hand their rows over a block at a time, because those are the families a large run makes large: a cube states a row for every netting set, date and sample.

reporting writes the run's results.

• A metric row is the run, the result, the file, the line, the family, the subject, the observation date, the measure, the currency, the scenario, the risk factor, every other qualifier as one JSON object, and the figure. It is a hypertable on the observation date, and the family is a column rather than a table: dozens of tables would be dozens of identical shapes and a union for every cross-family read. The cubes and the sensitivity scenarios are here too: they state a trade's or a netting set's value, which is a result.
• A cashflow row is the payment's trade or market object, its scenario, its leg, its number, its pay date, its flow type and its currency, one figure named by its heading, and the part of a stressed comparison it states.
• A text value row is a name and its text, about a trade when the file names one, with the number beside it when the engine stated a double.

The market data store writes the market the run saw, with the same readers.

• A scenario value is one risk factor's value under one scenario at one date.
• A calibration value is one line of the calibration file.
• A market input is one quote, fixing or dividend the run read. It is not written as an observation: the observations a run reads come from this store, and the run's copy written back would duplicate them and replace their provenance.

Every row is keyed by where the engine stated it: the compute result, the file, the line and the heading. A sensitivity, a cube or a scenario file states many rows with the same subject, date and heading, so nothing else tells them apart. Every row is written once per run and never edited; a correction is a new run.

A report definition names the families its runs skip. The list is empty by default, so a new definition stores every family, and a family the catalogue gains later is stored without anyone editing a definition. The list is copied into the run when it starts, so a run stores what its definition said then. A skipped family's file is not fetched and stays in storage, so a run that did not store a family can be read later, while storage keeps the run's objects.

• Every output is rows, none is a blob. What a run computed, and the market it computed it in, is queryable. Keeping a file whole in a row was considered for the market state and rejected: it cannot be queried, and it hides the run's intermediate state from the person trying to understand the run.
• One table for each shape, not one for each family. The columns of the metric families fall into three jobs, and once the roles are known the shapes are one shape.
• A separate table for the payments. A cashflow's keys are what make it a cashflow, and flattening one into a metric loses them.
• The engine's headings are stored, not translated. A family's measure is the engine's own word, so nothing in the store has to be listed when ORE adds a column.

A screen does not read a family. ores.reporting.run_result is a read model: a view over the metric table that names each engine heading once — NPV is npv — and reads the counterparty out of the qualifiers. The screen's shape therefore does not change when a run states a new family, and no screen states a rollup twice.

A refusal stops the run and records the reason on the instance. The step refuses when a family's row cannot be read whole, when a figure is not a number, when a date cannot be read, when a file's bytes do not match the digest its manifest states, and when the run's output states nothing to store. Because the whole run is one transaction, a refusal in the last file rolls back every block the earlier files wrote. It does not refuse a file no family covers, and it does not treat the engine's null string as a zero: a figure that was not stated is not stored at all.

The compute wrapper runs the engine with the output directory on local disk, then uploads it twice over. It packs the directory into one archive, which retrieval copies under the run as its audit copy: whatever the engine wrote is in it, modelled or not. And it uploads each file as its own compressed object, with a manifest that names them, their sizes and their digests, so each component fetches only the files it models, one at a time.

The node projects nothing. It knows which files it wrote but not what they mean, and everything a reader would call judgement — which run this is, which party owns it, what date it priced at, whether the rows load at all — belongs to the component that holds the run. Its reach is unchanged: the job directory, object storage over HTTP with the token of the claim it was given, and compute.v1.ops to take work and say what happened.

Source: report_post_processing_interaction.puml. Regenerate with plantuml -tpng doc/knowledge/architecture/report_post_processing_interaction.puml.

When the run's configuration selects the substitute, the phase is not executed. The workflow dispatches a step that records the output location on the run and completes. Nothing is fetched, parsed or loaded.

This is what allows a run to complete while the result store is unwritten, and it is deliberately the weakest of the substitutes: it proves that a run can reach a terminal state and that its output exists, and it proves nothing about the contents. Three conditions hold, as for pre-processing:

• The output must still be recorded. A run that ignores its results must still name where they are, or the work is lost the moment the archive expires.
• The parallel phase must be indistinguishable. Finalisation knows that the run completed and where the output is; it is not told whether the results were read.
• The substitution must be recorded on the run. A completed instance whose results were never ingested must not be mistaken for one whose results are queryable.

A grid finishes many runs at once, so the cost of one run matters.

• Each component fetches only its own files, once. The output is one object for each file, so reporting and the market data store each fetch what they model, and a family a definition skips is not fetched at all.
• Memory is bounded by a block, not by a family. A block is 10000 rows, written as one pipeline with one round trip. Measured on a 500000-row raw cube, peak memory was 1992 MiB when the family was read whole and 202 MiB in blocks; the bound is one file's text and one block of rows.
• The insert is the limit. The same cube took about 205 s either way, because each row runs the bitemporal insert trigger, which looks for a current row of the same key to close. That trigger, not the round trips, sets the rate of about 2500 rows a second.
• A large run holds one transaction for the whole load. That is the price of a refusal leaving nothing behind; a database statement or idle timeout must allow it.