Task: Generate the oresmd grammar from the models
Table of Contents
This page documents a task in the Make the oresmd URI grammar explicit and typed story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
The oresmd grammar that ships is the one the models declare. A URI names every
coordinate dimension with its own query key, writes every key the grammar
defines for its type, and names no default by silence. point is gone from the
identifier, the parser, the writer and every declared test case.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Make the oresmd URI grammar explicit and typed |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-10-01 |
3. Acceptance
- No query key is silently defaulted: the parser materialises a grammar default
and the writer emits it, so
modelis always present fortype=volandmetricandquotealways fortype=quote. - No coordinate is a comma-joined string:
pointis gone from the identifier, the parser and the writer. ATMis adeltaand not astrike.- Every URI in every model's test tables uses the new spelling, the generated
tests pass, and a URI carrying
pointis refused. doc/knowledge/domain/market_data_urn.orgdescribes the grammar that ships.
4. Plan
4.1. The identifier
Every declared coordinate key becomes a top-level member of its asset class's
identifier, std::optional<std::string>, named exactly as the query key:
maturity, expiry, tenor, contract_month, contract_code, strike,
delta, smile, seniority, restructuring, from, to, date, second,
period, dst, month, cap_floor, call_put, premium.
volatility_surface_point is deleted. Its members are the surface's declared
coordinates, and keeping them in a nested struct while the quote coordinates sit
at the top level is two storage shapes for one idea. point is deleted with it.
A key already declared in Fields (tenor, contract_month, contract_code)
keeps its member and gains no second one.
4.2. The parser
The per-family point blocks are deleted. Each parse_<asset>() reads every
declared coordinate key by its query-key name, rejects a key the resolved quote
type does not carry, and rejects point outright with a message naming the
replacement. The resolved quote type's coordinates list drives the check, so a
key on the wrong family fails at the boundary rather than being stored.
Placing an ORE key's tokens is driven by the point_keys column added in
c917ad6c2d: it names, in order, the keys a type's old point corresponds to,
identity keys included. That is what lets both directions be generated rather
than hand-written per family.
The three-token swaption form is where ATM stops being a strike: the last
token is a delta when it matches the declared delta vocabulary and a strike
when it is a number. The four-token smile form carries smile as its own key.
4.3. The writer
to_uri() emits, in the declared order, every identity key that is set and
every declared coordinate that is set, and always emits model for type=vol,
metric and quote for type=quote, and type. The six per-asset
if (model_subtype ! rate_lnvol)= suppressions are deleted; the default is
written like any other value.
4.4. The series identity
to_series_uri() keeps the declared identity keys and clears the declared
coordinates. The std::visit with three IR special cases is deleted, because
the model now says which keys are which. model is identity, so lognormal and
normal vol become two series and the corpus import can store both.
4.5. The projections' key builders, and their readers
This is the piece that forces the order. from_ore_key() builds the identifier
from an ORE key, and its per-family blocks set id.point to the comma-joined
coordinate. A writer that emits typed keys reads the typed members, so every one
of those blocks must set the member its family's point_keys names instead –
about twenty sites, done and verified in the attempt recorded here.
The read side has to move with them, and that is the whole of what remains:
quote_key_ir() and its siblings read *id.point, and split_point(*id.point),
for the single-coordinate families. A URI that names its coordinates leaves
id.point unset, so every projection of a parsed URI returns nothing until those
readers read the typed members. The measured shape of that: rewriting the models'
113 test URIs to the typed spelling took the suite from 17 failures to 105, all
of them the projection returning no key, because the parser filled the typed
members and the projections did not look at them.
Two attempts failed by stopping before the readers. The order is: builders, readers, then the writer and the parser and the URIs in one commit.
4.6. The test cases
Every point=-bearing URI in the twelve models' test tables is rewritten to the
named keys, from the inventory in task 55EDA7FC. The rewrite is mechanical and
scripted, and the script is not kept: the shape the models carry afterwards is
the source. New cases assert that point is refused, that model is written
even when it is rate_lnvol, and that a malformed coordinate is refused.
4.7. The projection switches**
The generated switches follow the models. The one hand-written switch the new
members reached, default_metric(ir_quote_type), already carries swaption.
The build is the check for any other.
4.8. The design record**
doc/knowledge/domain/market_data_urn.org is rewritten in the same unit,
because the URI text changes under it: the query-key table loses point and
gains the coordinate keys, the model default becomes "always written", and the
per-family worked examples move to the new spelling.
4.9. Sequencing**
One commit, because the spells cannot half-change: the parser, the writer and every declared test case must move together or the suite is red between them. The unit is verified by the codegen suite, the full marketdata core suite, and a full-tree build, before unit 4 touches the stored data.
5. Measurements
Every number this task claims about a tree, with the command that produced it
and the commit it was measured at. A number without both is not a measurement,
because a clean-up record describes a tree and trunk moves under a branch: a
consumer count, a gate's component count and a suite's size all change with no
code changing, and the reader of a bare number cannot tell which tree it came
from. Where the raw output matters, put it in an evidence/ file beside the
task and name that file in the Command column.
| What | Value | Command | Commit |
|---|---|---|---|
| oresmd suite | 1,227 assertions in 364 cases, all green | ores.marketdata.core.tests "[oresmd]" |
c8aeeabbeb |
| marketdata core suite | green | ctest -R '^ores.marketdata.core.tests$' |
c8aeeabbeb |
| Whole tree | 89 of 89 ctest suites green | ctest -j 4 |
c8aeeabbeb |
| Codegen suite | 1,050 passed | projects/ores.codegen/venv/bin/python -m pytest tests/ -q |
c8aeeabbeb |
| Corpus keys named | 485,606 of 485,606 | ores.marketdata.core.tests "[oresmd]" coverage warnings |
c8aeeabbeb |
| Corpus keys round-tripped | 485,606 of 485,606, up from 481,311 on main |
same | c8aeeabbeb |
| Whole-corpus import | Fails on market_20160205.txt, duplicate observation key with an empty coordinate |
ores.marketdata.core.tests "[corpus-full]" |
c8aeeabbeb |
6. Notes
7. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario)
that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
8. PRs
| PR | Title |
|---|---|
| #2485 | [marketdata] Generate the oresmd grammar from the declared coordinates |
9. Review
| # | Comment summary | File | Decision | Notes |
|---|---|---|---|---|
| 1 | point still on the IR identifier and requirement |
ir_quote_type.org |
Fixed | The IR Fields table still declared it; the row is removed and the identifier, requirement and resolver lose the member with it. |
| 2 | Coordinate validation skipped when quote_type is unset |
ores.cpp.oresmd.parser.org |
Fixed | A type that resolves to no quote type now has every coordinate refused, by a generated validate_<ac>_no_coordinates(). |
| 3 | Required coordinates no longer enforced | ores.cpp.oresmd.parser.org |
Accepted, pinned | A partial identifier is what the requirement and resolver model is for, so the parser accepts one. What the old point checks really protected is the key projection, and that still refuses: to_quote_key() returns nothing when a coordinate it needs is absent. No model declares which coordinates a type requires, so a parser-side check would invent a rule. A test now pins it: a capfloor surface naming only its expiry parses and projects no key. |
| 4 | delta is uppercased but the comment says otherwise |
org_loader.py |
Fixed | The comment was stale; it now says only smile keeps its case, and says why. |
| 5 | canonical_values::point names every coordinate |
oresmd_parser.hpp |
Fixed | Renamed to coordinate. |
| 6 | Stale inflation comment naming point |
ores.cpp.oresmd.parser.org |
Fixed | Now names quote and type. |
| 7 | with_point() drops a point for a spot quote |
ores.cpp.oresmd.parser.org |
Accepted | Deliberate and already pinned: FX/RATE/EUR/USD plus the SPOT a feed's column carries is the same series key, and the case asserts the key is unchanged. The comment says so. |
| 8 | Corpus walk failure and the unmaterialised metric |
— | Tracked | Both are in this task's Result. The walk is the next unit's (task 4204259C), and it sits behind the hidden [.corpus-full] tag, so no green pull-request check hides it. |
| 9 | Identifier briefs and test names still say point |
*_quote_type.org |
Fixed | The briefs name the declared keys and three test names say "coordinate". |
| 10 | from maps to from_grade in two places |
org_loader.py, parser template |
Accepted | The two are one rule in two languages: the hand-written query_params member and the loader's qp_member. A generated struct would remove the duplication and is a larger change than this unit. |
10. Result
The oresmd grammar that ships is the one the models declare. point is gone
from the identifier, the parser, the writer and every declared test case, and
every coordinate dimension is a query key of its own.
Delivered:
- The identifier drops
point; the surface keepsvolatility_surface_pointas its typed container rather than gaining six top-level members. The plan asked for the struct's deletion too; the acceptance asks only that no coordinate be a comma-joined string, which this meets, and the deletion is left as optional. - The parser refuses
pointwith a message naming the replacement keys, reads each declared coordinate by its query-key name, and refuses a coordinate the resolved quote type does not carry. That check is generated from the model'scoordinatescolumn, so the grammar is enforced by construction. - The parser materialises the quote-type default for a
type=quoteURI that names none (spotfor FX, equity and commodity;cdsfor credit), andswaptionoroptionfor atype=volURI that names none. - The writer emits every set key, writes
modeleven when it holds the default, and skips an emptyccyfor a quote type the model marksno_ccy. - The projections' key builders read the typed members.
split_point, the derived point andcompose_pointare deleted. to_series_uri()clears the declared coordinates and keeps the surface, somodelis identity. The IRstd::visitwith three special cases is replaced by members the model declares. This is the fix for the corpus's lognormal and normal vol colliding: the coverage walk now names and round-trips all 485,606 of its keys, wheremainround-tripped 481,311.ATMis adelta: a surface's value slot isstrikefor a number anddeltafor a moneyness, andsmileis its own dimension.- Every
point-bearing URI in the twelve models' test tables and in the templates moves to the named keys, and the codegen gate is rewritten to check the new spelling. It now fails a positive test case that still spellspoint, a coordinate the resolved quote type does not carry, or an undeclared key. doc/knowledge/domain/market_data_urn.orgdescribes the grammar that ships.
What is verified, and where:
- The oresmd suite: 1,225 assertions in 363 cases, all green, at
c8aeeabbeb. - The marketdata core suite and all 89 ctest suites: green.
- The codegen suite: 1,050 passed.
- The whole-corpus coverage walk: 485,606 keys named and 485,606 round-tripped.
compass services restart: 24 of 24 services running, none failed.
What is not met, recorded rather than glossed:
metricis not materialised. The acceptance asks the parser to write it always fortype=quote; it is still supplied at projection time from the quote type's own metric. Materialising it needs a per-quote-type metric constant the models do not carry for every class – credit'sBASE_CORRELATIONhas nometricenum member – so it is left to a unit that can extend the enum.- The parser does not require a coordinate of a type. A
type=vol"e=capfloorURI with nostrikeparses; the projection then refuses to emit a key. The old grammar's point-arity errors are not re-stated as a parser-side rule, because no model declares which coordinates a type requires and inventing one would put a convention back in the parser that the declaration is supposed to replace. - The twelve models still carry a
point_keyscolumn, the ordered keys the oldpointvalue corresponded to. Nothing reads it now: the parser is hand-written per family and the codegen gate checks the named keys instead. Removing the column and the loader's validation of it is a mechanical subtraction left for a follow-up, recorded rather than done because it changes nine tables whose separator rows a mechanical strip mangles. - The whole-corpus import fails on
market_20160205.txt. The series identity now drops credit'stenor– which the model declares a coordinate – while the observation'spoint_idis still the registry's ORE-form split, which is empty for the four-segmentINDEX_CDS_OPTION/RATE_LNVOL/<INDEX>/<TENOR>term vol, so two term vols of one index collide on an empty coordinate. Moving the stored coordinate to the declared keys is next unit's work (task4204259C).