Task: Generate the oresmd grammar from the models

Table of Contents

This page documents a task in the Make the oresmd URI grammar explicit and typed story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

The oresmd grammar that ships is the one the models declare. A URI names every coordinate dimension with its own query key, writes every key the grammar defines for its type, and names no default by silence. point is gone from the identifier, the parser, the writer and every declared test case.

2. Status

Field Value
State DONE
Parent story Make the oresmd URI grammar explicit and typed
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-10-01

3. Acceptance

  • No query key is silently defaulted: the parser materialises a grammar default and the writer emits it, so model is always present for type=vol and metric and quote always for type=quote.
  • No coordinate is a comma-joined string: point is gone from the identifier, the parser and the writer.
  • ATM is a delta and not a strike.
  • Every URI in every model's test tables uses the new spelling, the generated tests pass, and a URI carrying point is refused.
  • doc/knowledge/domain/market_data_urn.org describes the grammar that ships.

4. Plan

4.1. The identifier

Every declared coordinate key becomes a top-level member of its asset class's identifier, std::optional<std::string>, named exactly as the query key: maturity, expiry, tenor, contract_month, contract_code, strike, delta, smile, seniority, restructuring, from, to, date, second, period, dst, month, cap_floor, call_put, premium.

volatility_surface_point is deleted. Its members are the surface's declared coordinates, and keeping them in a nested struct while the quote coordinates sit at the top level is two storage shapes for one idea. point is deleted with it.

A key already declared in Fields (tenor, contract_month, contract_code) keeps its member and gains no second one.

4.2. The parser

The per-family point blocks are deleted. Each parse_<asset>() reads every declared coordinate key by its query-key name, rejects a key the resolved quote type does not carry, and rejects point outright with a message naming the replacement. The resolved quote type's coordinates list drives the check, so a key on the wrong family fails at the boundary rather than being stored.

Placing an ORE key's tokens is driven by the point_keys column added in c917ad6c2d: it names, in order, the keys a type's old point corresponds to, identity keys included. That is what lets both directions be generated rather than hand-written per family.

The three-token swaption form is where ATM stops being a strike: the last token is a delta when it matches the declared delta vocabulary and a strike when it is a number. The four-token smile form carries smile as its own key.

4.3. The writer

to_uri() emits, in the declared order, every identity key that is set and every declared coordinate that is set, and always emits model for type=vol, metric and quote for type=quote, and type. The six per-asset if (model_subtype ! rate_lnvol)= suppressions are deleted; the default is written like any other value.

4.4. The series identity

to_series_uri() keeps the declared identity keys and clears the declared coordinates. The std::visit with three IR special cases is deleted, because the model now says which keys are which. model is identity, so lognormal and normal vol become two series and the corpus import can store both.

4.5. The projections' key builders, and their readers

This is the piece that forces the order. from_ore_key() builds the identifier from an ORE key, and its per-family blocks set id.point to the comma-joined coordinate. A writer that emits typed keys reads the typed members, so every one of those blocks must set the member its family's point_keys names instead – about twenty sites, done and verified in the attempt recorded here.

The read side has to move with them, and that is the whole of what remains: quote_key_ir() and its siblings read *id.point, and split_point(*id.point), for the single-coordinate families. A URI that names its coordinates leaves id.point unset, so every projection of a parsed URI returns nothing until those readers read the typed members. The measured shape of that: rewriting the models' 113 test URIs to the typed spelling took the suite from 17 failures to 105, all of them the projection returning no key, because the parser filled the typed members and the projections did not look at them.

Two attempts failed by stopping before the readers. The order is: builders, readers, then the writer and the parser and the URIs in one commit.

4.6. The test cases

Every point=-bearing URI in the twelve models' test tables is rewritten to the named keys, from the inventory in task 55EDA7FC. The rewrite is mechanical and scripted, and the script is not kept: the shape the models carry afterwards is the source. New cases assert that point is refused, that model is written even when it is rate_lnvol, and that a malformed coordinate is refused.

4.7. The projection switches**

The generated switches follow the models. The one hand-written switch the new members reached, default_metric(ir_quote_type), already carries swaption. The build is the check for any other.

4.8. The design record**

doc/knowledge/domain/market_data_urn.org is rewritten in the same unit, because the URI text changes under it: the query-key table loses point and gains the coordinate keys, the model default becomes "always written", and the per-family worked examples move to the new spelling.

4.9. Sequencing**

One commit, because the spells cannot half-change: the parser, the writer and every declared test case must move together or the suite is red between them. The unit is verified by the codegen suite, the full marketdata core suite, and a full-tree build, before unit 4 touches the stored data.

5. Measurements

Every number this task claims about a tree, with the command that produced it and the commit it was measured at. A number without both is not a measurement, because a clean-up record describes a tree and trunk moves under a branch: a consumer count, a gate's component count and a suite's size all change with no code changing, and the reader of a bare number cannot tell which tree it came from. Where the raw output matters, put it in an evidence/ file beside the task and name that file in the Command column.

What Value Command Commit
oresmd suite 1,227 assertions in 364 cases, all green ores.marketdata.core.tests "[oresmd]" c8aeeabbeb
marketdata core suite green ctest -R '^ores.marketdata.core.tests$' c8aeeabbeb
Whole tree 89 of 89 ctest suites green ctest -j 4 c8aeeabbeb
Codegen suite 1,050 passed projects/ores.codegen/venv/bin/python -m pytest tests/ -q c8aeeabbeb
Corpus keys named 485,606 of 485,606 ores.marketdata.core.tests "[oresmd]" coverage warnings c8aeeabbeb
Corpus keys round-tripped 485,606 of 485,606, up from 481,311 on main same c8aeeabbeb
Whole-corpus import Fails on market_20160205.txt, duplicate observation key with an empty coordinate ores.marketdata.core.tests "[corpus-full]" c8aeeabbeb

6. Notes

7. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

8. PRs

PR Title
#2485 [marketdata] Generate the oresmd grammar from the declared coordinates

9. Review

# Comment summary File Decision Notes
1 point still on the IR identifier and requirement ir_quote_type.org Fixed The IR Fields table still declared it; the row is removed and the identifier, requirement and resolver lose the member with it.
2 Coordinate validation skipped when quote_type is unset ores.cpp.oresmd.parser.org Fixed A type that resolves to no quote type now has every coordinate refused, by a generated validate_<ac>_no_coordinates().
3 Required coordinates no longer enforced ores.cpp.oresmd.parser.org Accepted, pinned A partial identifier is what the requirement and resolver model is for, so the parser accepts one. What the old point checks really protected is the key projection, and that still refuses: to_quote_key() returns nothing when a coordinate it needs is absent. No model declares which coordinates a type requires, so a parser-side check would invent a rule. A test now pins it: a capfloor surface naming only its expiry parses and projects no key.
4 delta is uppercased but the comment says otherwise org_loader.py Fixed The comment was stale; it now says only smile keeps its case, and says why.
5 canonical_values::point names every coordinate oresmd_parser.hpp Fixed Renamed to coordinate.
6 Stale inflation comment naming point ores.cpp.oresmd.parser.org Fixed Now names quote and type.
7 with_point() drops a point for a spot quote ores.cpp.oresmd.parser.org Accepted Deliberate and already pinned: FX/RATE/EUR/USD plus the SPOT a feed's column carries is the same series key, and the case asserts the key is unchanged. The comment says so.
8 Corpus walk failure and the unmaterialised metric — Tracked Both are in this task's Result. The walk is the next unit's (task 4204259C), and it sits behind the hidden [.corpus-full] tag, so no green pull-request check hides it.
9 Identifier briefs and test names still say point *_quote_type.org Fixed The briefs name the declared keys and three test names say "coordinate".
10 from maps to from_grade in two places org_loader.py, parser template Accepted The two are one rule in two languages: the hand-written query_params member and the loader's qp_member. A generated struct would remove the duplication and is a larger change than this unit.

10. Result

The oresmd grammar that ships is the one the models declare. point is gone from the identifier, the parser, the writer and every declared test case, and every coordinate dimension is a query key of its own.

Delivered:

  • The identifier drops point; the surface keeps volatility_surface_point as its typed container rather than gaining six top-level members. The plan asked for the struct's deletion too; the acceptance asks only that no coordinate be a comma-joined string, which this meets, and the deletion is left as optional.
  • The parser refuses point with a message naming the replacement keys, reads each declared coordinate by its query-key name, and refuses a coordinate the resolved quote type does not carry. That check is generated from the model's coordinates column, so the grammar is enforced by construction.
  • The parser materialises the quote-type default for a type=quote URI that names none (spot for FX, equity and commodity; cds for credit), and swaption or option for a type=vol URI that names none.
  • The writer emits every set key, writes model even when it holds the default, and skips an empty ccy for a quote type the model marks no_ccy.
  • The projections' key builders read the typed members. split_point, the derived point and compose_point are deleted.
  • to_series_uri() clears the declared coordinates and keeps the surface, so model is identity. The IR std::visit with three special cases is replaced by members the model declares. This is the fix for the corpus's lognormal and normal vol colliding: the coverage walk now names and round-trips all 485,606 of its keys, where main round-tripped 481,311.
  • ATM is a delta: a surface's value slot is strike for a number and delta for a moneyness, and smile is its own dimension.
  • Every point-bearing URI in the twelve models' test tables and in the templates moves to the named keys, and the codegen gate is rewritten to check the new spelling. It now fails a positive test case that still spells point, a coordinate the resolved quote type does not carry, or an undeclared key.
  • doc/knowledge/domain/market_data_urn.org describes the grammar that ships.

What is verified, and where:

  • The oresmd suite: 1,225 assertions in 363 cases, all green, at c8aeeabbeb.
  • The marketdata core suite and all 89 ctest suites: green.
  • The codegen suite: 1,050 passed.
  • The whole-corpus coverage walk: 485,606 keys named and 485,606 round-tripped.
  • compass services restart: 24 of 24 services running, none failed.

What is not met, recorded rather than glossed:

  • metric is not materialised. The acceptance asks the parser to write it always for type=quote; it is still supplied at projection time from the quote type's own metric. Materialising it needs a per-quote-type metric constant the models do not carry for every class – credit's BASE_CORRELATION has no metric enum member – so it is left to a unit that can extend the enum.
  • The parser does not require a coordinate of a type. A type=vol&quote=capfloor URI with no strike parses; the projection then refuses to emit a key. The old grammar's point-arity errors are not re-stated as a parser-side rule, because no model declares which coordinates a type requires and inventing one would put a convention back in the parser that the declaration is supposed to replace.
  • The twelve models still carry a point_keys column, the ordered keys the old point value corresponded to. Nothing reads it now: the parser is hand-written per family and the codegen gate checks the named keys instead. Removing the column and the loader's validation of it is a mechanical subtraction left for a follow-up, recorded rather than done because it changes nine tables whose separator rows a mechanical strip mangles.
  • The whole-corpus import fails on market_20160205.txt. The series identity now drops credit's tenor – which the model declares a coordinate – while the observation's point_id is still the registry's ORE-form split, which is empty for the four-segment INDEX_CDS_OPTION/RATE_LNVOL/<INDEX>/<TENOR> term vol, so two term vols of one index collide on an empty coordinate. Moving the stored coordinate to the declared keys is next unit's work (task 4204259C).

Emacs 29.3 (Org mode 9.6.15)