Task: Write the ORE index codec
Table of Contents
This page documents a task in the Rewrite the oresmd grammar as a hand-written codec checked against ORE story. It captures the goal, current status, acceptance, and any notes or results.
1. Goal
A fixing is named by a hand-written index codec that reads every index name ORE reads, as ORE reads it, and writes it back; the fixing import, export and classification use it, and the generated grammar has no remaining consumer.
2. Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Rewrite the oresmd grammar as a hand-written codec checked against ORE |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-10-03 |
3. Acceptance
- A catalogue of index names, taken from every fixing file in the corpus and from the forms ORE's parseIndex documents, records ORE's own reading of each, regenerated from the local ORE build like the key catalogue.
- The codec accepts the index names ORE accepts and refuses the ones it refuses, and write(read(n)) == n or the canonical spelling where ORE reads an alias.
- The fixing URI shape is agreed with the product owner before the codec writes it, and a fixing URI is one to one with the canonical index name.
- The fixing import, the export and the classifier's index-name rule use the codec; oresmd_parser and oresmd_projections have no production consumer.
- The SQL seeds that hold fixing URIs hold the new form, and a test checks every one.
4. Plan
Added on 2026-10-03, when the retire task found that fixings still need the
generated grammar: a fixing is named by an ORE index name such as
EUR-EURIBOR-6M, UKRPI or FX-ECB-EUR-USD, and only the old grammar's
from_index_name and to_index_name read and write one. The product owner
chose to replace it before the retire task removes the grammar, rather than
keep the fixing half or flatten the fixing identity.
The same method as the key codec:
- Extend the catalogue harness to ORE's
parseIndex, and record ORE's reading of every index name in the corpus's fixing files and of the formsparseIndexdocuments. - Model the index identity as data, the way the schema models a quote.
- Agree the fixing URI shape with the product owner.
- Write the codec, test it against the catalogue, and move the fixing import, the export and the classifier onto it, with the seeds.
5. Measurements
Every number this task claims about a tree, with the command that produced it
and the commit it was measured at. A number without both is not a measurement,
because a clean-up record describes a tree and trunk moves under a branch: a
consumer count, a gate's component count and a suite's size all change with no
code changing, and the reader of a bare number cannot tell which tree it came
from. Where the raw output matters, put it in an evidence/ file beside the
task and name that file in the Command column.
| What | Value | Command | Commit |
|---|---|---|---|
| Index forms in the catalogue | 69, of which ORE accepts 53 with the corpus's conventions | index_forms.jsonl |
9f124dec1f |
| Corpus index names | 158; ORE accepts 156 without conventions and 158 with them | index_corpus.jsonl |
9f124dec1f |
| Index codec tests | 5 cases, 32 assertions, all passed; every corpus name reads, writes back and matches ORE's family fields | ores.marketdata.core.tests "[ore_index_codec]" |
9f124dec1f |
| Tests bite | with the FX currencies swapped in the reader, 4 of 5 cases failed | the same, with the defect | 9f124dec1f |
| Whole suite after recreating the database | 90 of 90 passed (5 suites after the recreate) | compass test run |
9f124dec1f |
| Production consumers of the old grammar | none | grep of oresmd_parser and oresmd_projections over projects/ |
9f124dec1f |
6. Notes
6.1. Decisions agreed on 2026-10-03
Decomposed fixing URIs, in the pattern of quote URIs: the authority is the asset class, the path the index's subject, and the query
type=fixing,index=<family>and the family's own fields.EUR-EURIBOR-6M oresmd://ir/EUR?type=fixing&index=ibor&name=EURIBOR&tenor=6M FX-ECB-EUR-USD oresmd://fx/EUR?type=fixing&index=fx&source=ECB&ccy=USD COMM-NYMEX:CL-2025-01 oresmd://commodity/NYMEX:CL?type=fixing&index=commodity&expiry=2025-01- Structural acceptance. ORE reads any IBOR or inflation index a conventions
file defines, such as
CZK-CZEONIAandMXN-FTIIEin the corpus, so the codec accepts any name of the right shape. The catalogue harness loads the corpus's conventions, so ORE stays the oracle. - A generic authority.
GENERIC-*indices take a newgenericasset class, which maps to no refdata class, likecorrelation.
6.2. What ORE's parseIndex reads
Eleven families, tried in this order: equity (EQ-*), bond (BOND-ID[-YYYY-MM[-DD]],
a bond future when dated), bond future (BOND_FUTURE-*), commodity
(COMM-NAME[-YYYY-MM[-DD]]), intraday power (POWER-NAME-DATE[-START-END]), FX
(FX-SOURCE-CCY1-CCY2), generic (GENERIC-*), constant maturity bond
(CMB-FAMILY-TENOR), IBOR and overnight (CCY-NAME[-TENOR]), CMS swap
(CCY-CMS[-TAG]-TENOR), and zero inflation (a fixed list with spaced aliases
such as UK RPI, or a convention). QuantLib's name() is an internal label
(Euribor6M Actual/360), not an ORE name, so the harness reports the family's
inspectors rather than a canonical spelling.
Without conventions ORE accepts 156 of the 158 corpus index names; with
examples/Products/Input/conventions.xml it accepts all 158.
6.3. Where the codec and ORE differ, on purpose
Every name in this table is a negative probe written into
external/ore/tools/datum_catalogue/index_forms.txt, not data from ORE: the
codec and ORE agree on all 158 corpus names. The ORE column is the verdict
external/ore/catalogue/index_forms.jsonl records.
| Name | ORE | Codec | Why |
|---|---|---|---|
EQ-, BOND-, COMM- |
accepts, empty name | refuses | the model refuses empty text; an index with no name is no identity |
POWER-ICE:PDQ |
accepts, delivery from the evaluation date | refuses | the name cannot keep a date ORE takes from the clock |
FX-ECB-EUR-XXX |
refuses, unknown currency | accepts | ORE checks its currency list; the codec takes three letters, as the key codec does |
CMB-US-TREASURY-10Y, CMB-BUND-5Y |
refuses without bond reference data | accepts | a data dependency, not grammar |
EUR, NOSUCHCPI |
refuses without a convention | accepts | the agreed structural reading: a hyphen-free name is an inflation index a convention may define |
UK RPI, EU HICPXT |
reads as UKRPI, EUHICPXT |
writes the unspaced name | the canonical spelling |
A power index with a DST flag, POWER-ICE:PDQ-2021-03-28-3600-7200-DST (a
hand-written negative probe), is refused by ORE's catalogue, so the codec refuses the flag too. ORE's grammar
names it, but no corpus name carries it, and ORE's error does not say which
hour it would accept.
6.4. A regression from the cut-over, found here
The tenant provisioning handler in ores.iam still wrote feed bindings through
oresmd_projections and oresmd_parser, in the old URI form. The cut-over's
consumer map, taken by grep over ores.marketdata and ores.synthetic, did
not include ores.iam, so from PR 2545 a newly provisioned tenant's FX feed
bindings named series the ingest loop could not read, and its ticks were
dropped. This task moves it onto the codecs; a grep over the whole of
projects/ now finds no production consumer of the old grammar.
7. Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario)
that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
8. PRs
| PR | Title |
|---|---|
| #2570 | [marketdata] Write the ORE index codec and name fixings with it |
9. Review
| # | Comment summary | File | Decision | Notes |
|---|---|---|---|---|
| 1 | Local review: the first draft of market_index::make checked for unknown fields after moving the known ones out. |
market_index.cpp |
Accepted | Unknown fields are checked first. |
| 2 | Local review: the consumer map in the cut-over missed ores.iam. |
tenant_provisioning_handler.hpp |
Accepted | Fixed here; the map is now a grep over all of projects/. |
| 3 | PR 2570: storing the canonical name as the qualifier could split a spaced alias's series. | import_service.cpp |
Declined | A series is found by its oresmd URI, not its qualifier, and the fixings tokenizer splits on spaces, so a spaced alias never reaches the import. |
| 4 | PR 2570: a power index with a DST flag is refused. | ore_index_codec.cpp |
Declined | ORE refuses it too (index_forms.jsonl); the name is a hand-written negative probe, noted under the deviations. |
| 5 | PR 2570: std::stoi copies strings already checked to be digits. |
ore_index_codec.cpp |
Accepted | A from_chars helper reads them. |
| 6 | PR 2570: starts() only forwards to starts_with. |
ore_index_codec.cpp |
Accepted | Removed. |
| 7 | PR 2570: say why is_currency checks only three upper-case letters. |
ore_index_codec.cpp |
Accepted | A comment says ORE checks its configurable currency table and the codec checks the shape. |
| 8 | PR 2570: move the skip lambda and the namespace alias to the top of the function. |
tenant_provisioning_handler.hpp |
Declined | The lambda reads the loop's config, and the alias is scoped to its one use. |
| 9 | Own finding while checking 3: an import test comment named the deleted from_ore_key() and from_index_name(). |
service_import_service_tests.cpp |
Accepted | It names the two codecs now. |
10. Result
ore_index_codec reads every index name ORE's parseIndex reads, by
structure, and writes it back; market_index holds a fixing's identity as data,
one row per family; and oresmd_uri_codec writes and reads the agreed fixing
URI. The fixing import, the export, the classifier and the tenant provisioning
handler use them, so the generated grammar has no production consumer and the
retire task can delete it. Along the way this found and fixed a regression from
the cut-over: tenant provisioning still wrote old-form feed bindings.