Task: Design: the market data data model on oresmd

Asset Classes analyses seven published classifications and proposes one taxonomy. Seven classes, all at the same level:

Two changes come with it. cross_asset is removed, because a correlation between two classes is a relation between them rather than a third kind of thing. rates is renamed interest_rates, because with inflation beside it the bare word no longer reads as the interest rate class on its own.

The analysis task decided the following, and they stand:

The inverse-projection task shipped in PR #1963. The canonical form is therefore a fact about the code, not a design question:

• oresmd_projections::from_ore_key maps an ORE key string to a market_data_identifier, seeded by the series_key_registry.
• oresmd_parser::to_uri(identifier, canonical_values) emits the one canonical string per identifier: fixed parameter order per asset class, lower-cased codes, absent fields skipped.
• The builder matches tenor and point against a supplied canonical_values container and rejects unknown spellings.
• Round trips are pinned by tests: parse(to_uri(id)) = id= and to_uri(parse(uri)) = uri=.
• The stored string is the percent-encoded canonical form, and matching uses the same form.

A swaption is an option on a swap, so it needs two periods. The example quote file prices a full grid, every expiry from 1M to 30Y crossed with every underlying swap tenor from 1Y to 50Y. So 10y,2y,atm is a ten-year option into a two-year swap, at the money, and a two-part point could not name a cell in that grid. The third part is not always the literal ATM: the same grid carries Smile/-0.01 and Smile/0.02.

The point is not always three parts. Counting the shapes actually present in market_20160205.txt, by segments after the TYPE segment:

The arity varies by (quote type, metric), from absent to three parts. One text column holding the validated composite is the right shape for that. A column per axis would be a union for the same reason as section 4.3, and would have to be revisited every time ORE adds a quote family.

Two of the twelve are not business vocabularies. instrument_type (fixing, curve, quote, vol) and curve_role (discount, projection, self_discounting) are structural discriminators. The parser and the projections branch on them by name, as in id.type ! instrument_type::quote=. Nobody adds a fifth instrument type without a code change, so making them data buys nothing and costs compile-time exhaustiveness.

They stay as code constants. The remaining ten become reference tables.

A vocabulary held as data cannot be switched on exhaustively. The compiler stops proving that every quote type is handled. What replaces it is a run-time check against the loaded table at the validation boundary, which is where the analysis already moved validation. The trade is deliberate: a missing handler becomes a rejected identifier at import rather than a compile error.

Requirement. Import all the market data in the ORE samples, store it, confirm it reads correctly in the database, export it back to ORE format, and have the result identical.

This contradicts the parent story. Its out-of-scope section reads: "The market data values themselves (quotes, fixings, curves): only identity is in scope." The requirement cannot be met under that boundary, so the boundary moves. The story's out-of-scope line needs amending, and this is a ruling for the owner rather than a decision this task can take.

This is measured against external/ore/examples/Input/, not assumed.

20160205 BMA_SWAP/RATIO/USD/3M/3M 0.8
20160205 BMA_SWAP/RATIO/USD/3M/1Y 0.8
...
#### Fake ####

#artificial
20160205 ZERO/RATE/USD/2D/1D/1M 0.0043

Four properties matter for a round trip, and three of them are hostile to it.

1. A uniform data row. YYYYMMDD KEY VALUE, space separated. 8149 rows in market_20160205.txt. This part is easy, and the key is exactly what from_ore_key consumes.
2. Hand-written section comments. The file carries #### Fake ####, ###fake###, #####, #artificial, # artificial, # Normal volatility cap surfaces, # YY Inflation Volatilitites and more. The styles are inconsistent, and one carries a spelling mistake.
3. Blank lines as separators, in places that follow the sections rather than the data.
4. A hand-chosen row order that is not a function of the keys. The groups run BMA_SWAP, CC_BASIS_SWAP, BASIS_SWAP, MM, IR_SWAP, FRA, FX, FXFWD, ..., which is neither alphabetical nor by segment count. Within a group the order is by tenor as a duration, where = {3M, 1Y, 2Y, …, 10Y, 12Y, 15Y, 20Y, 25Y, 30Y}= is the file's order and a string sort would put 10Y before 2Y.

So the file is not a pure function of its data. Comments, blank lines and group order are all editorial choices made when the file was written.

An exporter that emits rows in a canonical sort order cannot reproduce these files byte for byte. Three positions are available, and the design does not pick between them because the answer is a product call about what identical means.

The recommendation is canonical plus a normalising comparison, with replay available later if byte identity turns out to be required for a specific delivery. The reasoning is that the round trip exists to prove no data is lost, and a normalising comparison answers that question directly. Byte identity also answers it, but it additionally asserts that our exporter reproduces somebody else's whitespace, which is a different and much larger claim.

What "normalising" has to mean. Parse both files into a multiset of (date, key, value) triples and compare those. This catches every real loss, including a dropped row, a changed value, a changed date and a re-keyed series. It deliberately ignores order, comments and blank lines, which is exactly the set of things section 4.8.2 showed to be editorial.

Measured, not assumed:

The same exporter must emit both, so the date format is a property of the file kind and not a global setting.

The fixings file groups by index code and sorts by date ascending within a code, which is a different editorial rule again.

The round trip imports every key in the samples, and the samples contain ORE families that from_ore_key rejects today: BOND, the option and capfloor families, and the volatility sub-families. Under the taxonomy resolution in section 4.2 some of these gain a mapping, and the rest do not.

Two ways out, and they are not equivalent.

• Map everything. Extend oresmd until every sampled key has an identifier. Complete, and it forces the capfloor grammar question in section 4.4 to be answered rather than deferred.
• Pass through unmapped keys. Store the raw ORE key for anything the parser rejects, flagged as unmapped. Gets the round trip green sooner and leaves a permanent second identity form in the table, which is the drift the analysis ruled against.

The recommendation is to map everything. It costs the capfloor answer, and it keeps one identity form, which is the whole point of the story.

Requirement. We already bootstrap curves, so the model must represent the curves we create.

Section 4.7 covers it. What is worth stating here is that the bootstrap is the first producer of curve_build rows, so the round trip in 4.8.1 applies to them too: a curve we build must export to ORE format as well as a curve we read. That is a stronger requirement than storing it, and it is the reason the point composite is reused rather than a second coordinate form being invented for curves.

Requirement. We generate market data, so the model must represent what we generate.

This is already covered and needs no new tables, which is worth recording because it is easy to assume otherwise. Generated data lands in the same two tables as vendor data:

• A generated observation is a market_observation row whose source column names the producer channel, as in synthetic.v1.tick.fx_spot.eur-usd.
• Its provenance is an observation_lineage row: the derivation_config_id and derivation_config_version that produced it, the source_as_of it was built from, and the source_series_ids it consumed.

So the model already distinguishes the three provenances in one place: a vendor observation has a source and no lineage, a derived observation has both, and a bootstrapped curve has a curve_build.