Investigation: Review of ores.marketdata after the grammar story

ORE defines 45 instrument types (MarketDatum::InstrumentType, without NONE). oresmd names 41 of them. These four have no quote type:

None of the four is in the corpus, so the coverage gate, which reads only the corpus and the shape table, cannot see that they are missing.

ORE also accepts three aliases: FX_SPOT for FX, FX_FWD for FXFWD, and the deprecated RATE_GVOL for RATE_LNVOL. oresmd rejects all three. An import drops such a line with a warning. It does not normalise it.

ORE defines 16 quote types. The oresmd metric enum holds 8 of them. CREDIT_SPREAD, CONV_CREDIT_SPREAD, HAZARD_RATE and BASE_CORRELATION are not members, so metric is not set for credit quotes. The vol models (RATE_LNVOL, RATE_NVOL, RATE_SLNVOL, SHIFT, PRICE) are in the separate volatility_model_subtype enum. This is the gap the previous close-out summary recorded for BASE_CORRELATION. It is wider than that one member.

This is where the real distance is. ORE accepts many forms for most types. For example, CDS has nine documented forms, and CAPFLOOR, SWAPTION, EQUITY_OPTION, COMMODITY_OPTION and INDEX_CDS_OPTION each have between four and eight. oresmd was shaped by the example corpus, so it reads the forms the corpus uses and few others.

Of the 40 documented forms that are not in the corpus, oresmd names 17. At least four of those 17 are named wrongly: the reader puts a token in the wrong field. The text round-trips, so no current test sees the fault.

The corpus has the same kind of fault. EQUITY_OPTION/.../ATM/AtmFwd and COMMODITY_OPTION/.../ATM/AtmFwd store AtmFwd as call_put. The 9-segment ATM/AtmDeltaNeutral/DEL/Spot form stores DEL as call_put and Spot as strike. The readers choose a field by segment count alone. ORE chooses it by the token values.

Forms that oresmd rejects outright include: the CDS PRICE, CONV_CREDIT_SPREAD, 8-segment and 4-segment (MDX) forms; CAPFLOOR with an index token or a C=/=F flag; the CAPFLOOR, SWAPTION and BOND_OPTION shift forms with no surface; the equity MNY/Spot/1.2 moneyness strike; and COMMODITY_OPTION/SHIFT/<name>.

• Fixings are in good shape. All 158 corpus index names except the two that the corpus misfiles as fixings round-trip through the stored URI (probe 3).
• Dividends (ORE's dividendDataFile) have no reader and no entity. The corpus has no input dividends file, so nothing has asked for one yet.
• Curve and market configuration (curveconfig.xml, todaysmarket.xml) are out of scope by decision, and Add proper curve support to ores.marketdata holds the curve question.

oresmd_ore_coverage_tests.cpp checks to_quote_key(from_ore_key(k)) = k=. That never builds a URI string. Production stores the URI string: import_service writes to_series_uri to market_series.oresmd_uri and to_uri to market_observations.oresmd_uri. Probe 1 runs the stored path:

The 4,269 rejected URIs come from five types. The writer makes a URI that its own parser refuses:

Probe 2 shows that the series identity fails too for the three credit index families: 3,007 corpus keys import under a series URI that does not parse. The rows are in the database. Nothing reads them back through the parser yet, so nothing fails.

The other losses are case changes. The writer lower-cases the entity path segment, and the parser upper-cases it and most coordinates. So EQUITY/PRICE/Lufthansa/EUR comes back as EQUITY/PRICE/LUFTHANSA/EUR, and .../ATM/AtmFwd comes back as .../ATM/ATMFWD. ORE names are case sensitive. 1,845 of 1,866 COMMODITY_OPTION keys and 1,118 EQUITY_OPTION keys lose their case this way. The capture The forward projection uppercases free-text identifiers recorded this fault in the in-memory path. That path is fixed. The URI path still has it, and the capture still sits in the inbox as if nothing changed.

import_service.cpp:358 stores the file's raw key in market_observations.key. ore_export_service.cpp:54 writes that column back out. So export_reproduces_the_market_data_file_it_imported proves that a text column survives a database round trip. It does not prove that the grammar can produce the key.

The same column makes one row hold two identities that can disagree. When the FX convention checker reverses a pair, the series and the datum URI take the corrected pair, and key keeps the reversed one. Other writers (curve_republish_service.cpp:344, feed_ingest_loop.cpp:491) store the projected key instead. So the meaning of the column depends on which writer wrote the row.

service_corpus_import_tests.cpp compares the sorted multiset of values in the file with the sorted multiset of values in the rows. Two values that change places between keys pass. The test does not check that a value arrives under its own key.

The * Round-trip tables in the quote-type models check URI → identifier → URI. That proves that the parser and the writer agree. It cannot prove that a token is in the correct field, which is the fault in section 1.

• oresmd_ore_coverage_tests.cpp has its own key_of tokeniser. It needs two commas on a CSV line, which is the fault that PR 2 fixed in the production parser. Lines with one comma drop out of the coverage count without notice.
• The comment above expected_fully_round_tripped says that four types still lose keys. The gap list is empty, and the four types (COMMODITY_FWD, EQUITY, EQUITY_FWD, CORRELATION) are missing from the expected set.
• corpus_files.hpp excludes MD_* files, but the rule is dead: those files do not contain "market" in their name, so the earlier rule already excludes them. Its comment also names todaysmarketcalibration.csv, which the code does not test for. The three MD_*.csv files are real market data in DD.MM.YY comma form, and the three FD_*.csv files are fixings in semicolon form. Neither reaches any corpus walk.
• I count 96 market payload files with the selection rule. The clean task's V08 row says 118.

The oresmd quote-type models declare the enums, the fields, the coordinates, the reject keys and the test tables. Those parts are generated. The key grammar itself is not:

• oresmd_projections.cpp.mustache has 2,585 lines and 187 template tags. oresmd_parser.cpp.mustache has 1,074 lines and 200 tags. Almost every from_* reader and to_quote_key branch is hand-written C++ inside the template.
• The model's qualifier_shape column only selects one of those hand-written functions. The segment layout is in the notes column as prose ("6-segment: ZERO/RATE/CCY/CURVE_ID/DAY_COUNT/POINT"), and no code reads it.
• model.org carries variant_order and parse_order columns, and the IR model has a URI order table. Their only job is to reproduce the order of the hand-crafted files that the generator replaced.

So to add an ORE key form, you edit a 2,585-line template, not a model row. That is the main obstacle to the code-generation goal.

The same ORE key shapes are written down three times:

1. ores_series_key_shapes_populate.sql (203 lines), read by ores.ore's series_key_registry and used by parse_market_data and by import_service to split a key into (series_type, metric, qualifier).
2. The oresmd models' qualifier_shape and notes columns.
3. The hand-written C++ in the two templates.

The import uses (1) to parse and classify, and (3) to name. A change to one is not checked against the others except through the corpus.

The story's own * Status table still says "No implementation unit has started".

• oresmd_resolver (306 generated lines) and market_data_requirement have no production caller. Under B06 that is dead code. The clean task's B06 row says the oresmd test-only symbols were closed.
• to_curve_key and instrument_type::curve have no production caller. The curves story already records this.
• Each of market_observations and market_fixings has two repositories: a generated singular one and a hand-written plural one. The record keeps the plural ones for read_as_of. H02 asks for one survivor.
• The market_series table comment (generated from the model prose) says that "those three columns carry the registry's decomposition". The columns are gone.

The hand-written code and the oresmd templates have a very high comment density. Some functions have more comment than code. 14 comments narrate edit history ("The silence the old grammar read as a default", "used to arrive inside the point", "no longer"). Some of them come from the templates, so every regeneration writes them again. Two hand-written end-of-line comments remain (crm_ingest_bridge.hpp:176, curve_republish_resolver.hpp:47). The project comment rules forbid all three patterns.

A name match finds no test for 73 of the 102 source files outside main.cpp. Some of them are covered through other tests. But no repository test exists for feed_binding, observation_lineage or series_classification_rule. No test exists for any generated message handler, for publish_from_dq_handler, for curve_republish_service, or for market_data_client. Only series_classification_rule has an eventing integration test.

The capture for M06 and P04, Six ores.marketdata protocol headers carry wire types no model covers, describes work that is done. Its body is still the empty scaffold.