Story: Migrate to oresmd, delete market_series qualifier and ore_key
Table of Contents
This page documents a story in Sprint 25, carried unfinished from Sprint 24 at close. It captures the goal, current status, acceptance criteria, and the tasks that compose it.
Goal
Migrate ORE Studio's market data identifiers onto oresmd and delete the legacy schemes it replaces — a hard cutover, not permanent coexistence. Per-field migration:
fx_spot_generation_config.ore_key: the URI's FX branch (oresmd://fx/eurusd?type=quote→FX/RATE/EUR/USD) already produces exactly this field's existing value —ore_keybecomes thetype=quoteprojection of an FX-scopedoresmdURI. Once every consumer reads the projected value fromoresmdinstead, delete theore_keycolumn/field.market_seriesseries_type/metric/qualifier[/point_id]: the URI'stype=quote=/=type=volprojection rules produce exactly theTYPE/SUBTYPE/dim...shapemarket_series'sseries_type=/=metriccolumns already mirror;qualifier=/=point_idare currently free text, hand-populated per row — underoresmdthey would be deterministically generated from the URI'sentity=/=index=/=tenor=/=pointfields instead. Migrate every write path to generatequalifier=/=point_idfrom anoresmdidentifier, then the free-text population path can be deleted (theseries_type=/=metric=/=qualifier=/=point_idcolumn shape itself can stay — this is a generation-source change, not necessarily a schema change, though revisit once migration is underway).ir_curve_generation_config's (currency_code,index_name) pair: replaced by atype=curveoresmdURI (entity=currency_code,index=+=tenorreplacing free-textindex_name, plus therolefield this table has no equivalent of today).curve_feed_controller's qualifier-based collision check should be rebuilt on top oforesmdfields (asset class, entity, index, tenor, role) so it can finally express "these two curves legitimately coexist because one discounts and one projects" instead of colliding on a flat string.
Status
| Field | Value |
|---|---|
| State | DONE |
| Carried from | Sprint 24 (unfinished at close) |
| Now | 5 of 6 tasks DONE; XML round-trip descoped to Extend oresmd story. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-08-07 |
Acceptance
- Every consumer of
ore_key,market_seriesqualifier/point_id, andir_curve_generation_config's (currency_code, index_name) is migrated ontooresmd(or the migration for it is scoped as an explicit follow-on task with rationale for deferral). - The legacy free-text fields (
ore_key, and the free-text write path formarket_seriesqualifier/point_id) are deleted once every consumer reads/writes throughoresmdinstead – a hard cutover, no permanent coexistence. curve_feed_controller's qualifier-based collision check is rebuilt onoresmdfields so it can express discount-vs-projection curve coexistence instead of colliding on a flat string.- Normalize floating-rate index reference data is either superseded/closed or explicitly reconciled with this migration, not left as a competing scheme.
– descoped: the task attempting this was abandoned once scoping against the real corpus showed it depends on closing oresmd's quote-type representation gap first, now tracked as its own story (Extend oresmd to full ORE quote-type coverage). This story's remaining acceptance is the internal-consumer migration (oresmdidentifiers can round-trip through ORE's own =curveconfig.xml=/=todaysmarket.xml=/market-data-quote file formatsore_key,market_series,ir_curve_generation_config), which is complete.
Tasks
| Task | State | Start | End | Description |
|---|---|---|---|---|
| In-depth analysis: migrate legacy identifiers onto oresmd | DONE | 2026-07-28 | 2026-07-28 | Ground the per-field migration plan in the current codebase: enumerate every read/write consumer of ore_key, market_series qualifier/point_id, and ir_curve_generation_config's (currency_code, index_name), and confirm the oresmd projection rules cover every one before any schema/code change starts. |
| Add oresmd support to ores.marketdata | DONE | 2026-07-28 | 2026-07-28 | Implement parse_oresmd, to_uri, and resolve inside ores.marketdata (api/core) – not a separate library/component – per the oresmd design's variant-based data model, with projection rules into ORE's index/curve/quote keys. No consumer migration yet. |
| Migrate fx_spot_generation_config.ore_key onto oresmd | DONE | 2026-07-29 | 2026-07-29 | Switch ore_key's generation source from the currency-pair derivation to oresmd's FX projection rule, verify against existing rows, then delete the old derivation code. |
| Migrate ir_curve_generation_config onto oresmd, add curve role | DONE | 2026-07-29 | 2026-07-30 | Replace index_name with oresmd's index_family (backed by overnight_index_convention/ibor_index_convention) plus tenor, add the missing role field, and rebuild curve_feed_controller's collision check on the full oresmd field set so discount/projection coexistence works. Close Normalize floating-rate index reference data as superseded once this lands. |
| Migrate market_series qualifier/point_id generation onto oresmd | DONE | 2026-07-30 | 2026-07-30 | Generate market_series.qualifier and market_observation.point_id from oresmd fields at every write path instead of free text; column shapes stay as-is. Once every consumer reads/writes through oresmd, delete the legacy free-text generation paths across all three migrations. |
| Build ORE curve-config and market-data XML round-trip via oresmd | ABANDONED | 2026-07-30 | 2026-07-30 | Abandoned: scoping against the real corpus found oresmd represents ~7 of 49 ORE quote types (split into its own story) and most YieldCurve segment subtypes need that gap closed first. Revisit via the new story once its IR-coverage task lands. |
Decisions
ir_curve_generation_configmigrates directly ontooresmd, not via a standaloneindex=/=currency_indexrefdata model: the in-depth analysis task confirmedindex_nameis already an FK'd, catalog-backed value (floating_index_type.code), not the free-text problem that design was built to solve, so building it would have created a third competing scheme. Normalize floating-rate index reference data will be closed as superseded once this migration lands.oresmd'sindex_familyenum is backed byovernight_index_convention=/=ibor_index_convention, the tables ORE's realconventions.xmlimporter actually writes into –floating_index_typeis a separately hand-seeded, unreconciled catalog and should be retired or turned into a generated view over the other two, not the source of truth.oresmdsupport lives insideores.marketdata, not a separate library/component –oresmdnames a URI scheme (aconstexprstring constant), not a C++ namespace boundary; the types/functions are plainores::marketdataadditions, keeping every consumer's existing dependency edge onores.marketdataunchanged.- Migration order:
ores.marketdata's oresmd support first, thenore_key(lowest risk, pure derivation), thenir_curve_generation_config(unblocks the superseded-task closure), thenmarket_series(widest internal consumer surface), then the ORE curve-config/market-data XML round-trip – each its own task on this story. - ORE XML round-trip is new capability, not a migration:
conventions.xmlalready round-trips both ways (ores.ore.core/xml/importer.hpp=/=exporter.hpp), but there is no existing import/export for ORE'scurveconfig.xml,todaysmarket.xml, or market-data quote-string formats at all today –oresmdproducing the right key strings internally does not by itself connect to a real ORE run's actual input/output files, so building that round-trip is included as its own task rather than assumed to fall out of the internal-consumer migrations. - Widget/soft-FK convention for new oresmd-shaped fields: a
dynamic_comboonly when a real, persisted entity backs the field (tenor->refdata::domain::tenor, matchingcalendar_exception.calendar_code -> fetch_calendars); astatic_combofor a closed, intrinsic vocabulary with no entity to reuse (role), or where the "real" enforcement is better done server-side than via a new cross-table aggregation query just to populate a combo (index_family, validated againstovernight_index_convention=/=ibor_index_conventionin the C++ service layer instead). - Composite (multi-column) soft FKs live in the C++ service layer, not
SQL: the codegen Insert-trigger Validations mechanism only supports
fn(tenant_id, NEW.column)(one column in, one value out); a two-column lookup like(currency_code, index_family)againstovernight_index_convention=/=ibor_index_conventionis validated inir_curve_generation_config_serviceinstead, matching the precedentir_curve_template_entry's own tenor codes already set (also app-layer-only, no DB trigger). curve_feed_controller's collision key includesrole, not just qualifier: a discount curve and a projection curve for the same(currency_code, index_family, tenor)are expected to coexist, so role is part of the conflict comparison rather than excluded from it – two feeds only conflict when qualifier AND role both match.ores.synthetic's legacy = Primary key=/=* Natural keys= model format needs migrating to the unifiedColumns+:primary_key:=/:natural_key:= flags, entity by entity, as each is touched* – codegen no longer supports the old dual-heading format at all (discovered while regeneratingir_curve_generation_config); this story migrates each entity it touches, not the whole component at once.market_seriesqualifier/point_id generation needed no migration work beyond consolidation: the two prior tasks on this story (ore_key,ir_curve_generation_config) already made FX's and IR's qualifier/point_id generation oresmd-derived as a side effect of their own scope, andmarket_serieshas no hand-entry UI to migrate. The only real gap was three independent hand-written copies of the same projected-key-splitting logic, consolidated intooresmd_projections::split_market_series_key(). Lesson for future tasks on a migration story: always landscape-survey before assuming a task's title describes its full remaining scope – prior tasks can close out later ones' acceptance criteria for free.
Out of scope
Promoted from capture
Captured 2026-07-27 in the product backlog; promoted preserving the UUID.
What
Migrate ORE Studio's market data identifiers onto oresmd and delete the legacy schemes it replaces — a hard cutover, not permanent coexistence. Per-field migration:
fx_spot_generation_config.ore_key: the URI's FX branch (oresmd://fx/eurusd?type=quote→FX/RATE/EUR/USD) already produces exactly this field's existing value —ore_keybecomes thetype=quoteprojection of an FX-scopedoresmdURI. Once every consumer reads the projected value fromoresmdinstead, delete theore_keycolumn/field.market_seriesseries_type/metric/qualifier[/point_id]: the URI'stype=quote=/=type=volprojection rules produce exactly theTYPE/SUBTYPE/dim...shapemarket_series'sseries_type=/=metriccolumns already mirror;qualifier=/=point_idare currently free text, hand-populated per row — underoresmdthey would be deterministically generated from the URI'sentity=/=index=/=tenor=/=pointfields instead. Migrate every write path to generatequalifier=/=point_idfrom anoresmdidentifier, then the free-text population path can be deleted (theseries_type=/=metric=/=qualifier=/=point_idcolumn shape itself can stay — this is a generation-source change, not necessarily a schema change, though revisit once migration is underway).ir_curve_generation_config's (currency_code,index_name) pair: replaced by atype=curveoresmdURI (entity=currency_code,index=+=tenorreplacing free-textindex_name, plus therolefield this table has no equivalent of today).curve_feed_controller's qualifier-based collision check should be rebuilt on top oforesmdfields (asset class, entity, index, tenor, role) so it can finally express "these two curves legitimately coexist because one discounts and one projects" instead of colliding on a flat string.
Findings from a parallel implementation attempt (2026-07-27/28)
While starting
Normalize floating-rate
index reference data (a pre-existing BACKLOG task predating this
capture, aimed at consolidating floating_index_type=/
=overnight_index_convention into a normalized index=+
=currency_index refdata model), two things surfaced that belong here
rather than in that task:
- That task would have built a third competing scheme. Its
index=/=currency_indexdesign (anindexentity keyed on a bare family code, plus a junction asserting (currency, index, tenor) and deriving aCCY-INDEX[-TENOR]soft-FK string) solves the exact same free-text-index_nameproblemoresmd'sindex_familyenum +tenorquery key already solves, but as a separate ORE-Studio-refdata mechanism rather than an application oforesmditself — precisely the "no three competing schemes" anti-pattern this capture's ownWhysection warns against. Recommendation: do not build a standaloneindex=/=currency_indexrefdata model; migrateir_curve_generation_configdirectly ontooresmdper this capture's existing per-field migration plan instead. The normalize-floating-index-refdata task should be re-scoped or closed as superseded once this migration lands. floating_index_typeisn't the only legacy catalogoresmd'sindex_familyenum should absorb.floating_index_type's 25 rows are a flattened mix of two other real, populated tables:overnight_index_convention(SOFR/SONIA/ESTR/…-shaped rows) andibor_index_convention(EURIBOR/LIBOR-shaped rows with tenor variants) — both are live destinations ofores.cli's real OREconventions.xmlimport command (application::import_conventions()writesconvs.overnight_index=/=convs.ibor_indexstraight into them via their repositories), not orphaned scaffolding as a first pass incorrectly concluded from grepping only for SQL soft-FK/validator references. Anyindex_familyenum design should map onto (or absorb) both tables' real fixing-calendar/day-count/settlement-lag conventions, and theconventions.xmlimporter is itself a consumer that will eventually need to write throughoresmd's data model instead of straight into either legacy table.
Why
The oresmd design task explicitly scoped out any code/schema migration
("no three competing schemes remain in the mental model" was a design
goal, not an implementation one) — this capture exists so that
per-field migration plan isn't lost once the design doc itself is
trimmed down to describe only the oresmd scheme, not the legacy
schemes it will eventually replace. Implementing this closes the
tenor-index and curve-role gaps
the gap analysis task
found in production code, not just in a design document.
References
- oresmd: ORE Studio Market Data URI — the design this migrates onto.
- Market Data Identifier — documents the
market_serieskey being replaced. - Gap analysis: ORE Studio identifiers vs ORE's real addressing scheme — the gaps this migration closes in code.
See also
- Market data notation: design an ORE Studio canonical URN mapping to ORE/Bloomberg/Reuters — the design story this is follow-on work from.
- Dual-curve discount+projection model — the still-BACKLOG task whose data model should agree with
oresmd'srolefield on one representation.