Investigation: The oresmd URI grammar: silent defaults and the untyped coordinate

The walk imports every market payload under external/ore/examples through the import service, then reads the rows back. It is the only test that measures the whole path. It is also hidden behind Catch2's .corpus-full tag, so ctest never runs it.

It failed on its 32nd file:

AmericanMonteCarlo/Input/market_20160205_flat_fixed_fxfwd.txt
Repository error: Executing INSERT failed: ERROR: duplicate key value violates
unique constraint "_hyper_13_8_chunk_market_observations_observations_current_uniq"
DETAIL: Key (tenant_id, party_id, series_id, observation_datetime, point_id)
=(..., 76778039-..., 2016-02-05, 6M/10Y/ATM) already exists.

The file carries both SWAPTION/RATE_LNVOL/CHF/6M/10Y/ATM and SWAPTION/RATE_NVOL/CHF/6M/10Y/ATM. The probe printed:

ln data   : oresmd://ir/chf?tenor=10y&type=vol&point=6m,10y,atm
nv data   : oresmd://ir/chf?tenor=10y&type=vol&point=6m,10y,atm&model=rate_nvol
ln series : oresmd://ir/chf?tenor=10y&type=vol
nv series : oresmd://ir/chf?tenor=10y&type=vol

The two data identities differ by model. The two series identities are equal. The observation's unique key is the series identity, the date and the coordinate, so the second row cannot be inserted. The failing file is not unusual: 22 of the 96 market payloads carry a lognormal and a normal vol at the same date and coordinate, and the scan counts 23,828 such groups.

The only path set aside was treating the failure as a duplicate-data problem. It is not: the two keys are different quotes, and the file is a faithful ORE example.

to_uri() writes model only when it differs from rate_lnvol (oresmd_parser.cpp, six copies, one per asset class). The struct's member carries the default, so the parser knows the value and the writer discards it. Absence therefore means "lognormal" to a reader who knows the default, and "unspecified" to every other reader. The design record says the parser "must not silently default it"; the writer silently omits it.

This is the only place the writer discards a value it holds. It is enough to lose an entire volatility surface.

One query key carries between one and five dimensions, in a comma-joined string, with the meaning of each position defined by the quote type:

The reader must know the quote type, and often the family's own conventions, to read the string. This is the opacity the grammar exists to remove.

The struct volatility_surface_point has a delta_type member. Its own comment says it holds "DEL, ATM or ATMF". The parser fills it only for the four-token smile form (oresmd_parser.cpp:420). For the common three-token form it assigns the third token to strike (:416). So ATM, a moneyness dimension, is kept in the strike slot. ORE's own curveconfig.xml calls it a dimension: [[<Dimension>ATM</Dimension>]].

The consequence is not only cosmetic. A strike and a delta are different coordinates. Storing one in the other's field makes a real strike and a moneyness marker indistinguishable in the data, and it is why the URI cannot name the dimension the user asked about.

SWAPTION/RATE_LNVOL/CHF/6M/10Y/ATM parses to tenor=10y and point=6m,10y,atm. The writer emits both, because tenor is set and point is set. The same fact appears in two slots.

The round-trip tests do not catch it. They assert parse(uri) = parse(to_uri(parse(uri)))= — identifier equality, not URI-text equality (oresmd_oresmd_parser_tests.cpp:1108-1123). A URI may therefore normalise on the way out without any test noticing.

to_series_uri() clears point and then vol wholesale (oresmd_parser.cpp:1193-1200), on the reasoning that "the surface is its vol point". That reasoning is right for the coordinates and wrong for the model. The model subtype is not a coordinate: it selects which surface is being sampled. Two ORE quote types, RATE_LNVOL and RATE_NVOL, are two surfaces over one underlying, and the corpus publishes both. Clearing the struct merges them.

This is the root cause of the corpus failure. It is also invisible to the coverage tests, because every key still names and still round-trips: the failure appears only when two surfaces meet in one database key.

to_series_uri() is a std::visit with three special cases for IR (a discount curve's tenor, a future's contract fields, an overnight future's contract month). Everything else is a consequence of where the parser happened to put a value. The model declares point_semantics — scalar, tenor, surface, composite — but that is a class of coordinate, not the list of keys or their role. Nothing in the model says "model is identity, strike is a coordinate", so the rule lives in one function and the models cannot check it.

The URI's coordinate is comma-joined and lower-cased (6m,10y,atm). The database's coordinate is the registry's slash-joined ORE form (6M/10Y/ATM, ores_series_key_shapes). The import bridges them at import_service.cpp:329-331: take the registry's point_id when it has one, else the parser's. Two spellings of one coordinate, bridged in a caller.

The observation table's uniqueness rule makes this worse: the coordinate is part of the key (market_observations_observations_current_uniq_idx), so the two spellings must agree exactly for the identity to be stable.

Some defaults are written into the identifier and some are not. FX spot sets quote_type=fx_quote_type::spot (oresmd_projections.cpp:1430), so the writer emits quote=spot. An IR metric is left unset when the ORE key does not carry the second segment, and the projection defaults it later (default_metric, oresmd_projections.cpp:213). A reader of a URI cannot tell which keys were defaulted, because both spellings are legal for the same datum.

A key the grammar defines for a (type, quote) pair is always present. Where the grammar has a default, the parser materialises it and the writer emits it. model is always present for type=vol, metric always for type=quote, and type always. A reader never has to know a default to read a URI.

A consequence worth stating: four volatility families have no quote type of their own today. An IR swaption is the implicit default for type=vol, and the FX, equity and commodity options are the same on their asset classes. P1 removes the right to leave the key out, so each gains a member: swaption for IR, and an option member for FX, equity and commodity. Keeping quote absent for those four would leave exactly the silence P1 removes.

point is removed. Each dimension is a query key named after the dimension. Which keys a quote type carries is declared in the model, so the parser rejects an unknown or misplaced one by construction rather than by hand-written checks.

The vocabulary, with the existing IR fields reused:

A worked example. The series identity is the datum identity with the coordinate keys removed, so the two spellings differ by exactly the coordinates:

before  datum   oresmd://ir/chf?type=vol&point=6m,10y,atm
        series  oresmd://ir/chf?tenor=10y&type=vol      <- model dropped: the bug

after   datum   oresmd://ir/chf?type=vol&quote=swaption&model=rate_lnvol
                       &tenor=10y&expiry=6m&delta=atm
        series  oresmd://ir/chf?type=vol&quote=swaption&model=rate_lnvol&tenor=10y

after   datum   oresmd://ir/chf?type=vol&quote=capfloor&model=rate_lnvol
                       &tenor=6m&shift=0&strip=0&expiry=5y&strike=0.03
        series  oresmd://ir/chf?type=vol&quote=capfloor&model=rate_lnvol
                       &tenor=6m&shift=0&strip=0

after   datum   oresmd://credit/vod?ccy=eur&type=quote&quote=cds&metric=credit_spread
                       &seniority=snrfor&restructuring=xr14&maturity=10y
        series  oresmd://credit/vod?ccy=eur&type=quote&quote=cds&metric=credit_spread

Every query key the model declares gains one attribute: role, either identity or coordinate. Then:

• the series identity is the identifier with the coordinate keys removed;
• the datum identity is the whole identifier;
• to_series_uri() is the rule, not a visitor with special cases.

For a volatility surface, model moves from being dropped to being identity, while expiry, strike, delta, call_put and premium stay coordinates. tenor, shift, strip and the rest of the surface's own conventions are identity today and stay identity; that is a deliberate choice to keep one series per quoted surface rather than one per currency. The one change is that the model joins them. That single classification fixes F5: lognormal and normal vol become two series, and the corpus import can store both.

The database coordinate is the typed keys rendered in the model's declaration order, for example seniority=snrfor,restructuring=xr14,maturity=10y. It is self-describing, so the row and the URI agree without translation. The registry's ORE-form point_id (ores_series_key_shapes) is derived for key projection and no longer written to the observation.

Each of the twelve *.org quote-type models gains a Coordinates table:

The point_semantics column stays, as the class, and the table becomes its expansion. The generator reads the table to emit the identifier struct, the parser, the writer and the tests. The hand-written per-family blocks in oresmd_parser.cpp and the three special cases in to_series_uri() are deleted against it.

The observation row already stores the producer's own key in market_observations.key (for example SWAPTION/RATE_LNVOL/CHF/6M/10Y/ATM). A migration can therefore re-derive both the series identity and the coordinate from data the row already holds, rather than parse the old URI. Rows whose key is null cannot be re-derived and must be reported.

Existing databases hold only one of each colliding pair, because the insert of the second always failed. The migration splits the stored row only where the retained key names a different surface from the row's model.

TodaysMarket binds a role to a named set and a name to a curve. It holds no values. Two layers:

<Configuration id="ois">
  <DiscountingCurvesId>xois_eur</DiscountingCurvesId>
  <YieldCurvesId>xois_eur</YieldCurvesId>
  <IndexForwardingCurvesId>default</IndexForwardingCurvesId>
</Configuration>
<DiscountingCurves id="ois">
  <DiscountingCurve currency="EUR">Yield/EUR/EUR1D</DiscountingCurve>
</DiscountingCurves>

The leaves are ORE curve keys, which oresmd already names as type=curve: Yield/EUR/EUR1D is oresmd://ir/eur?curve_id=eur1d&type=curve. So today's market is the binding between the market series, the curve definitions (curveconfig.xml) and a named market a run selects.

"If ORE names the document in the Analytics block it belongs to ores.analytics; if the run or an analytic merely references it, it belongs to the component that owns the concept, which is ores.refdata for curves, conventions, calendars, currencies and tenors, and ores.reporting for the run document and the registry."

• curveconfig.xml names curves and quotes. In scope of the reporting story (tasks DB247C1C and E13E6DCD). Owner: ores.refdata.
• todaysmarket.xml names sets of curves and the role each set plays. The concept is a curve selection. By the same rule its owner is ores.refdata. simm_todaysmarket.xml has the same root and goes with it.
• The <Simulation><Market> documents (crifmarket.xml, sensimarket.xml, stressmarket.xml, xvasensimarket.xml) configure an analytic's market. Owner: ores.analytics, with simulation.
• ore_market.xml has root <ORE>. It is a run document, and it goes to ores.reporting with the other run documents.

So the recommendation is that today's market goes to ores.refdata, with curveconfig.xml and the conventions. One note for the record. ores.reporting.report_market_binding's generated comment says the value it holds "names a configuration set built in todaysmarket.xml, which is market data and belongs to another component". The phrase "market data" is loose. The document is market configuration, and the binding row correctly stays in reporting. That comment should be corrected in the same unit that models today's market, so the record does not point at ores.marketdata by accident.