Story: Migrate to oresmd, delete market_series qualifier and ore_key

Table of Contents

This page documents a story in Sprint 25, carried unfinished from Sprint 24 at close. It captures the goal, current status, acceptance criteria, and the tasks that compose it.

Goal

Migrate ORE Studio's market data identifiers onto oresmd and delete the legacy schemes it replaces — a hard cutover, not permanent coexistence. Per-field migration:

  • fx_spot_generation_config.ore_key: the URI's FX branch (oresmd://fx/eurusd?type=quoteFX/RATE/EUR/USD) already produces exactly this field's existing value — ore_key becomes the type=quote projection of an FX-scoped oresmd URI. Once every consumer reads the projected value from oresmd instead, delete the ore_key column/field.
  • market_series series_type/metric/qualifier[/point_id]: the URI's type=quote=/=type=vol projection rules produce exactly the TYPE/SUBTYPE/dim... shape market_series's series_type=/=metric columns already mirror; qualifier=/=point_id are currently free text, hand-populated per row — under oresmd they would be deterministically generated from the URI's entity=/=index=/=tenor=/=point fields instead. Migrate every write path to generate qualifier=/=point_id from an oresmd identifier, then the free-text population path can be deleted (the series_type=/=metric=/=qualifier=/=point_id column shape itself can stay — this is a generation-source change, not necessarily a schema change, though revisit once migration is underway).
  • ir_curve_generation_config's (currency_code, index_name) pair: replaced by a type=curve oresmd URI (entity = currency_code, index=+=tenor replacing free-text index_name, plus the role field this table has no equivalent of today). curve_feed_controller's qualifier-based collision check should be rebuilt on top of oresmd fields (asset class, entity, index, tenor, role) so it can finally express "these two curves legitimately coexist because one discounts and one projects" instead of colliding on a flat string.

Status

Field Value
State DONE
Carried from Sprint 24 (unfinished at close)
Now 5 of 6 tasks DONE; XML round-trip descoped to Extend oresmd story.
Waiting on Nothing.
Next Nothing.
Last touched 2026-08-07

Acceptance

  • Every consumer of ore_key, market_series qualifier/point_id, and ir_curve_generation_config's (currency_code, index_name) is migrated onto oresmd (or the migration for it is scoped as an explicit follow-on task with rationale for deferral).
  • The legacy free-text fields (ore_key, and the free-text write path for market_series qualifier/point_id) are deleted once every consumer reads/writes through oresmd instead – a hard cutover, no permanent coexistence.
  • curve_feed_controller's qualifier-based collision check is rebuilt on oresmd fields so it can express discount-vs-projection curve coexistence instead of colliding on a flat string.
  • Normalize floating-rate index reference data is either superseded/closed or explicitly reconciled with this migration, not left as a competing scheme.
  • oresmd identifiers can round-trip through ORE's own =curveconfig.xml=/=todaysmarket.xml=/market-data-quote file formats – descoped: the task attempting this was abandoned once scoping against the real corpus showed it depends on closing oresmd's quote-type representation gap first, now tracked as its own story (Extend oresmd to full ORE quote-type coverage). This story's remaining acceptance is the internal-consumer migration (ore_key, market_series, ir_curve_generation_config), which is complete.

Tasks

Task State Start End Description
In-depth analysis: migrate legacy identifiers onto oresmd DONE 2026-07-28 2026-07-28 Ground the per-field migration plan in the current codebase: enumerate every read/write consumer of ore_key, market_series qualifier/point_id, and ir_curve_generation_config's (currency_code, index_name), and confirm the oresmd projection rules cover every one before any schema/code change starts.
Add oresmd support to ores.marketdata DONE 2026-07-28 2026-07-28 Implement parse_oresmd, to_uri, and resolve inside ores.marketdata (api/core) – not a separate library/component – per the oresmd design's variant-based data model, with projection rules into ORE's index/curve/quote keys. No consumer migration yet.
Migrate fx_spot_generation_config.ore_key onto oresmd DONE 2026-07-29 2026-07-29 Switch ore_key's generation source from the currency-pair derivation to oresmd's FX projection rule, verify against existing rows, then delete the old derivation code.
Migrate ir_curve_generation_config onto oresmd, add curve role DONE 2026-07-29 2026-07-30 Replace index_name with oresmd's index_family (backed by overnight_index_convention/ibor_index_convention) plus tenor, add the missing role field, and rebuild curve_feed_controller's collision check on the full oresmd field set so discount/projection coexistence works. Close Normalize floating-rate index reference data as superseded once this lands.
Migrate market_series qualifier/point_id generation onto oresmd DONE 2026-07-30 2026-07-30 Generate market_series.qualifier and market_observation.point_id from oresmd fields at every write path instead of free text; column shapes stay as-is. Once every consumer reads/writes through oresmd, delete the legacy free-text generation paths across all three migrations.
Build ORE curve-config and market-data XML round-trip via oresmd ABANDONED 2026-07-30 2026-07-30 Abandoned: scoping against the real corpus found oresmd represents ~7 of 49 ORE quote types (split into its own story) and most YieldCurve segment subtypes need that gap closed first. Revisit via the new story once its IR-coverage task lands.

Decisions

  • ir_curve_generation_config migrates directly onto oresmd, not via a standalone index=/=currency_index refdata model: the in-depth analysis task confirmed index_name is already an FK'd, catalog-backed value (floating_index_type.code), not the free-text problem that design was built to solve, so building it would have created a third competing scheme. Normalize floating-rate index reference data will be closed as superseded once this migration lands.
  • oresmd's index_family enum is backed by overnight_index_convention=/=ibor_index_convention, the tables ORE's real conventions.xml importer actually writes into – floating_index_type is a separately hand-seeded, unreconciled catalog and should be retired or turned into a generated view over the other two, not the source of truth.
  • oresmd support lives inside ores.marketdata, not a separate library/component – oresmd names a URI scheme (a constexpr string constant), not a C++ namespace boundary; the types/functions are plain ores::marketdata additions, keeping every consumer's existing dependency edge on ores.marketdata unchanged.
  • Migration order: ores.marketdata's oresmd support first, then ore_key (lowest risk, pure derivation), then ir_curve_generation_config (unblocks the superseded-task closure), then market_series (widest internal consumer surface), then the ORE curve-config/market-data XML round-trip – each its own task on this story.
  • ORE XML round-trip is new capability, not a migration: conventions.xml already round-trips both ways (ores.ore.core/xml/importer.hpp=/=exporter.hpp), but there is no existing import/export for ORE's curveconfig.xml, todaysmarket.xml, or market-data quote-string formats at all today – oresmd producing the right key strings internally does not by itself connect to a real ORE run's actual input/output files, so building that round-trip is included as its own task rather than assumed to fall out of the internal-consumer migrations.
  • Widget/soft-FK convention for new oresmd-shaped fields: a dynamic_combo only when a real, persisted entity backs the field (tenor -> refdata::domain::tenor, matching calendar_exception.calendar_code -> fetch_calendars); a static_combo for a closed, intrinsic vocabulary with no entity to reuse (role), or where the "real" enforcement is better done server-side than via a new cross-table aggregation query just to populate a combo (index_family, validated against overnight_index_convention=/=ibor_index_convention in the C++ service layer instead).
  • Composite (multi-column) soft FKs live in the C++ service layer, not SQL: the codegen Insert-trigger Validations mechanism only supports fn(tenant_id, NEW.column) (one column in, one value out); a two-column lookup like (currency_code, index_family) against overnight_index_convention=/=ibor_index_convention is validated in ir_curve_generation_config_service instead, matching the precedent ir_curve_template_entry's own tenor codes already set (also app-layer-only, no DB trigger).
  • curve_feed_controller's collision key includes role, not just qualifier: a discount curve and a projection curve for the same (currency_code, index_family, tenor) are expected to coexist, so role is part of the conflict comparison rather than excluded from it – two feeds only conflict when qualifier AND role both match.
  • ores.synthetic's legacy = Primary key=/=* Natural keys= model format needs migrating to the unified Columns + :primary_key:=/:natural_key:= flags, entity by entity, as each is touched* – codegen no longer supports the old dual-heading format at all (discovered while regenerating ir_curve_generation_config); this story migrates each entity it touches, not the whole component at once.
  • market_series qualifier/point_id generation needed no migration work beyond consolidation: the two prior tasks on this story (ore_key, ir_curve_generation_config) already made FX's and IR's qualifier/point_id generation oresmd-derived as a side effect of their own scope, and market_series has no hand-entry UI to migrate. The only real gap was three independent hand-written copies of the same projected-key-splitting logic, consolidated into oresmd_projections::split_market_series_key(). Lesson for future tasks on a migration story: always landscape-survey before assuming a task's title describes its full remaining scope – prior tasks can close out later ones' acceptance criteria for free.

Out of scope

Promoted from capture

Captured 2026-07-27 in the product backlog; promoted preserving the UUID.

What

Migrate ORE Studio's market data identifiers onto oresmd and delete the legacy schemes it replaces — a hard cutover, not permanent coexistence. Per-field migration:

  • fx_spot_generation_config.ore_key: the URI's FX branch (oresmd://fx/eurusd?type=quoteFX/RATE/EUR/USD) already produces exactly this field's existing value — ore_key becomes the type=quote projection of an FX-scoped oresmd URI. Once every consumer reads the projected value from oresmd instead, delete the ore_key column/field.
  • market_series series_type/metric/qualifier[/point_id]: the URI's type=quote=/=type=vol projection rules produce exactly the TYPE/SUBTYPE/dim... shape market_series's series_type=/=metric columns already mirror; qualifier=/=point_id are currently free text, hand-populated per row — under oresmd they would be deterministically generated from the URI's entity=/=index=/=tenor=/=point fields instead. Migrate every write path to generate qualifier=/=point_id from an oresmd identifier, then the free-text population path can be deleted (the series_type=/=metric=/=qualifier=/=point_id column shape itself can stay — this is a generation-source change, not necessarily a schema change, though revisit once migration is underway).
  • ir_curve_generation_config's (currency_code, index_name) pair: replaced by a type=curve oresmd URI (entity = currency_code, index=+=tenor replacing free-text index_name, plus the role field this table has no equivalent of today). curve_feed_controller's qualifier-based collision check should be rebuilt on top of oresmd fields (asset class, entity, index, tenor, role) so it can finally express "these two curves legitimately coexist because one discounts and one projects" instead of colliding on a flat string.

Findings from a parallel implementation attempt (2026-07-27/28)

While starting Normalize floating-rate index reference data (a pre-existing BACKLOG task predating this capture, aimed at consolidating floating_index_type=/ =overnight_index_convention into a normalized index=+ =currency_index refdata model), two things surfaced that belong here rather than in that task:

  • That task would have built a third competing scheme. Its index=/=currency_index design (an index entity keyed on a bare family code, plus a junction asserting (currency, index, tenor) and deriving a CCY-INDEX[-TENOR] soft-FK string) solves the exact same free-text-index_name problem oresmd's index_family enum + tenor query key already solves, but as a separate ORE-Studio-refdata mechanism rather than an application of oresmd itself — precisely the "no three competing schemes" anti-pattern this capture's own Why section warns against. Recommendation: do not build a standalone index=/=currency_index refdata model; migrate ir_curve_generation_config directly onto oresmd per this capture's existing per-field migration plan instead. The normalize-floating-index-refdata task should be re-scoped or closed as superseded once this migration lands.
  • floating_index_type isn't the only legacy catalog oresmd's index_family enum should absorb. floating_index_type's 25 rows are a flattened mix of two other real, populated tables: overnight_index_convention (SOFR/SONIA/ESTR/…-shaped rows) and ibor_index_convention (EURIBOR/LIBOR-shaped rows with tenor variants) — both are live destinations of ores.cli's real ORE conventions.xml import command (application::import_conventions() writes convs.overnight_index=/=convs.ibor_index straight into them via their repositories), not orphaned scaffolding as a first pass incorrectly concluded from grepping only for SQL soft-FK/validator references. Any index_family enum design should map onto (or absorb) both tables' real fixing-calendar/day-count/settlement-lag conventions, and the conventions.xml importer is itself a consumer that will eventually need to write through oresmd's data model instead of straight into either legacy table.

Why

The oresmd design task explicitly scoped out any code/schema migration ("no three competing schemes remain in the mental model" was a design goal, not an implementation one) — this capture exists so that per-field migration plan isn't lost once the design doc itself is trimmed down to describe only the oresmd scheme, not the legacy schemes it will eventually replace. Implementing this closes the tenor-index and curve-role gaps the gap analysis task found in production code, not just in a design document.

References

See also

Emacs 29.3 (Org mode 9.6.15)