Story: Rewrite the oresmd grammar as a hand-written codec checked against ORE

An ORE market data key read into ORE Studio means what ORE means by it, and it comes back out as the same key. The oresmd grammar is one hand-written, typed C++ codec. It reads every key form ORE's own parser accepts, and it puts each token in the field ORE puts it in. The proof is a field-by-field comparison with ORE's parseMarketDatum, and a round trip through the URI that the database stores. The same story closes the clean-standard findings that the 2026-10-02 review recorded, except P01, H01 and H03, which Return marketdata to the clean-standard registry holds.

On 2026-10-05, at the sprint 26 close, this story was reworded to the scope it delivered and closed DONE. The remainder — the point-in-time read P01, the composite root diagram H01, and the namespace closer comments H03 — is Return marketdata to the clean-standard registry, and its two tasks moved there.

• Every key form that ORE's parseMarketDatum accepts, and every key in the corpus, reads into the fields ORE assigns. A checked-in catalogue that ORE produced is the reference.
• Every ORE InstrumentType and QuoteType is modelled, and the aliases FX_SPOT, FX_FWD and RATE_GVOL are accepted.
• A quote URI is always fully specified and maps one to one onto the canonical ORE key: no default, no partial form, no extra key.
• Every datum round-trips key → datum → key and datum → URI → datum. Writing a parsed URI again gives the same text. Every series URI parses.
• The key grammar exists once: no generated oresmd code, no oresmd quote-type models, no SQL key shape table and no series_key_registry.
• The export builds each key from the stored URI. No row stores the raw key. The corpus tests compare each value under its own key.
• market_data_urn.org describes the grammar that ships.
• The marketdata clean-standard record matches the tree for the items this story closed: W02, P02, G02, G04, S01, S02, V04 and V08 pass. P01, H01 and H03 are open, and Return marketdata to the clean-standard registry holds them.

• The grammar is hand-written, not generated. The generated grammar was 2,585 lines of hand-written C++ inside a template with 187 tags. The model rows only chose which hand-written function ran. Generation added a build step and an indirection, and it removed no code. ORE's key grammar changes slowly and has a fixed shape. A typed class tested against ORE is easier to read, to change and to trust.

• The design, agreed on 2026-10-02. It follows the data-oriented design criteria: the identity is a closed value type with identity by key, and per-type behaviour is a switch, not a class hierarchy.

  1. One typed record, with its schema as data. market_datum holds the ORE instrument type (an enum of all 45), the ORE quote type (an enum of all 16), and a short list of (field, value) pairs. field is an enum of every field any ORE key carries. value is a small variant of value types: string, ORE period, ORE date, decimal, strike, and enum code. A strike is a variant of absolute, ATM with its ATM type, moneyness and delta. Free-text names (ticker, reference entity, curve id, commodity) keep the case ORE wrote. Equality is the defaulted operator=.
  2. The schema is a table, not a type per instrument. A constexpr table has one row per instrument type. The row lists the type's fields in key order, the role of each field (identity or coordinate), and the fields that may hold none. The only constructor checks a datum against its row. A traits map gives each field its value type, so get<field::expiry>() is typed at compile time. The series identity is the datum without its coordinate fields.
  3. Per-type logic is a switch on the instrument type. Code that differs per instrument type, chiefly the ORE key codec, switches on instrument_type with one function per type, in one translation unit per asset class. The build treats a missing case as an error (-Werror=switch-enum on GCC and Clang, C4062 as an error on MSVC), so a new type cannot be forgotten. Where such a function wants typed access, a non-owning view, for example cds_view(const market_datum&), gives typed getters inside that translation unit only.
  4. Nothing is silently absent. Where ORE has an optional token (a CDS doc clause, a running spread, a cap/floor flag), the datum holds an explicit value such as none, never a missing field that means a default.
  5. The ORE key codec reads and writes ORE key text. It follows the structure of ORE's marketdatumparser.cpp: one reader per instrument type, and each reader chooses a field by the token's value (period, date, number, C=/=P, ATM, MNY, DEL), not by segment count. Each form cites the ORE form it reads. An alias is read and then written in its canonical spelling, and the import reports the change.
  6. The URI codec is generic. It walks the datum's fields in schema order. The authority is the asset class. The path is the row's subject field, percent-encoded with its case kept, and the query does not repeat it. The query starts with type (quote or series), instrument and quote, then holds each other field under its schema name, in schema order, with the case of its value kept. The writer leaves out a field that holds none. The reader takes a missing key as none only where the schema row says the field may be none; it refuses a missing required key, an unknown key and a repeated key. It also refuses any text but the writer's own spelling, so one datum has one URI and stored URIs compare as text. No per-family URI code exists, so the URI cannot drift from the schema. Agreed on 2026-10-02: oresmd://credit/ACME?type=quote&instrument=cds&quote=credit_spread&seniority=SNRFOR&ccy=USD&doc_clause=XR14&term=5Y.
  7. One grammar. ores.ore keeps only the file reader, which yields date, key text and value. The SQL key shape table and series_key_registry go. Classification reads the asset class and the instrument type from the datum.
  8. Storage. market_series.oresmd_uri holds the series URI and market_observations.oresmd_uri holds the datum URI. The raw key column goes. The export writes write(read_uri(datum_uri)). The database is recreated and the corpus re-imported, so no migration exists.
  9. A quote URI has a strict contract. type=quote names exactly one ORE datum, so its URI is always fully specified: every required field of the type's schema row is present, a field that may be none is present only when it holds a value, and nothing else is present; no default applies, no partial or requirement form exists, and the URI maps one to one onto the canonical ORE key with case kept. "Canonical" means the spelling after an alias is resolved: FX_SPOT/RATE/EUR/USD reads as FX/RATE/EUR/USD, and a round trip of an alias returns the canonical spelling, not a failure. role and the requirement reading of index and tenor belong to fixings and curves, which keep the richer contract. market_data_urn.org states this as a rule of the grammar.

  9a. Fixings have their own hand-written codec. Agreed on 2026-10-03: a fixing is named by ore_index_codec, which reads an ORE index name the way parseIndex does and accepts by structure, as ORE does once a convention defines a name. Its URI is decomposed like a quote's, for example oresmd://ir/EUR?type=fixing&index=ibor&name=EURIBOR&tenor=6M, and GENERIC-* indices take a generic asset class.

  1. Tests, in order of authority. (a) A catalogue that ORE produced: an offline harness, linked against the local libOREData.so, reads every documented form and every distinct corpus key and writes ORE's fields to a checked-in fixture. The codec must match it field by field. (b) Key and URI round trips over the whole fixture, with series URIs parsed. (c) Property tests that generate datums from each schema row and each field's domain. (d) Keys that ORE rejects are rejected. (e) The database, corpus and export tests, which compare by key.
• One market tick for every feed. Agreed on 2026-10-04, after the market tick investigation: a live tick is one observation of one datum, keyed by its oresmd quote URI, with a value, an observation time and a source. Every feed takes its owner from its feed binding, and no asset class has a tick type of its own. Replace the asset-specific ticks with one market tick builds it and closes P02.
• No variant of instrument types. A std::variant with one alternative per ORE instrument type (45) was the first proposal. It was rejected because every translation unit that includes it compiles all 45 types, every std::visit instantiates 45 branches, and the variant approaches compiler limits. A new ORE type would also change a header the whole component includes. The schema table moves that cost to one checked constructor, and the only variant left has about six value types.
• No visitor hierarchy. A classic visitor over one class per instrument type was considered next. It keeps compile cost local and gives full typing, but it puts every datum behind an owning pointer with virtual dispatch. The data-oriented design criteria reject that shape for persisted data (value semantics, explicit variation, no owning pointers) and for kernels (no per-object dispatch). The enum switch gives the same exhaustiveness without the hierarchy.
• The asset class model stays. The model from Declare the asset class lists once generates SQL seeds and DQ badges. It is not part of the key grammar.
• Market reference data reaches a tenant by dataset publish. Decided on 2026-10-04 in Provision the FOMC meeting schedule into every tenant. Provisioning copies vocabulary with no dependencies, such as the diary entry types. Data that refers to a tenant's countries or calendars, such as the FOMC calendar, the tenor schedules and the meeting dates, is published after them. A column that can only be set once published data exists, such as a resolution's schedule, is set by that publish.

• Modelling curveconfig.xml or todaysmarket.xml, and the curve entity. Add proper curve support to ores.marketdata holds them.
• ORE's dividend history file. Nothing in the corpus or the product asks for it yet.
• The point-in-time read (P01). Backlog story F15A8480 holds it.
• The hot side of the data: a series' points as parallel arrays per coordinate shape, for curve and surface building. The 45-way variation lives in the identity, which is cold; the points vary over about six coordinate shapes. The curve story picks this up after this story closes.