Story: Adopt oresmd as the native market data identity across simulator, UIs, and shell
Table of Contents
This page documents a story in Sprint 25. It captures the goal, current status, acceptance criteria, and the tasks that compose it.
Goal
Make oresmd the native market data identifier across ORE Studio. The
market simulator (ores.synthetic), every Qt UI that displays or
edits market data, and ores.shell all talk oresmd natively, and the
persisted entities that identify market data (market_series,
market_observation, the generation configs, and related tables)
abandon their ORE-shaped representations (series_type=/=metric=/
=qualifier, point_id, ore_key, currency_code) in favour of
oresmd identity. Backwards compatibility is explicitly out of scope:
a hard cutover is the accepted strategy.
The first task is a thorough analysis. It inventories how the market
data tables connect to oresmd today and sets the target state. Its key
question: do entities store the individual components of an oresmd
identifier (ccy, index, tenor, role, point) as fields, or a
single column carrying the oresmd URI string? The hybrid – both
fields and the URI – must be explicitly ruled on; the working
assumption is that it is redundant: two canonical copies of the same
identity drift.
Status
| Field | Value |
|---|---|
| State | STARTED |
| Parent sprint | Sprint 25 |
| Now | Analysis task DONE; target state split into seven implementation tasks. |
| Waiting on | Review and merge of the analysis branch. |
| Next | Start the first implementation task: inverse projection and canonical form. |
| Last touched | 2026-08-11 |
Acceptance
- The analysis task (this story's first task) delivers: (a) an inventory of every market-data-bearing entity and its current identifier fields, with its read/write consumers; (b) a per-entity decision on fields-vs-URI storage with rationale; (c) a sequenced target state – the way forward – grounded in the actual codebase, not design-time assumptions.
- Every consumer lens is covered in the analysis: market simulator,
Qt UIs, and
ores.shell(including whatores.shellmust gain – today it has no market data surface at all). - The target state is implemented: simulator, UIs, and shell talk oresmd natively for market data identity.
- The legacy ORE representations for market data identity are deleted once the target state lands – no backwards compatibility, per the story's scope.
Tasks
| Task | State | Start | End | Description |
|---|---|---|---|---|
| Analysis: market data identity on oresmd — inventory, consumers, fields-vs-URI target state | DONE | 2026-08-11 | 2026-08-11 | Inventory every market-data-bearing entity and how it connects to oresmd today; enumerate consumers across the market simulator, all Qt UIs, and ores.shell; answer the key question per entity – component fields vs a single URI string (the hybrid of both must be explicitly ruled on); produce a sequenced target state. Analysis only, no schema or code change. |
| Inverse projection: ORE key to oresmd identifier, canonical form contract | DONE | 2026-08-11 | 2026-08-11 | Build the inverse projection (ORE key to oresmd identifier) in ores.marketdata.core/oresmd, seeded by the series_key_registry decomposition table; reject series types without an oresmd mapping. Pin the canonical URI form: round-trip tests and value-spelling rules (tenor and point via a supplied refdata container). |
| Identity core: market_series and feed_binding store oresmd URIs | BACKLOG | market_series and feed_binding gain oresmd_uri columns; backfill existing rows through the inverse projection; drop series_type/metric/qualifier and ore_key; repositories, registrars, and ingest loops read URIs; auto-create uses identifiers. | ||
| Observation identity: point_id carries the oresmd point | BACKLOG | market_observation.point_id carries the serialised oresmd point; scalar series have a null point (the SPOT sentinel dies); fixings and observation_lineages follow; the DQ artefact mirrors observation identity. | ||
| Generation configs move to oresmd URIs | BACKLOG | fx_spot_generation_config and ir_curve_generation_config switch to one oresmd_uri each; the component columns drop; validation moves from SQL constraints to the oresmd layer; the Qt editors round-trip URIs. | ||
| The NATS wire carries oresmd URIs | BACKLOG | Tick subjects and the ir_curve_tick payload carry oresmd URIs; every subscriber of marketdata.v1.tick.* and synthetic.v1.* updates; feed controllers resolve and collide on URIs. | ||
| ores.shell talks oresmd natively | BACKLOG | marketdata import stores oresmd URIs; new oresmd-native commands (list series, add series, validate); synthetic and crm command output prints URIs, never ORE keys. | ||
| Delete legacy ORE representations | BACKLOG | Drop the legacy identity columns (series_type, metric, qualifier, ore_key, currency_code), the import classification table, the split_market_series_key identity use, and the DB check constraints; update the tests. point_id stays, repurposed to carry the serialised oresmd point. |
Decisions
Decided by the analysis task (see its * Plan):
- Identity storage: one
oresmd_uricolumn per identity-bearing entity. No component fields. - The hybrid (fields and URI) is rejected: two writable copies of the same identity drift, and no constraint can detect it. No compensating benefit exists.
- Classification (
asset_class,series_subclass,is_scalar) is derived, not identity. Generated columns only where an index needs them. - Surrogate FKs to series rows are relationships, not identity copies. They stay.
- Validation moves from SQL check constraints into the oresmd parser and validator layer.
- The projection boundary is the ORE-format edge. Forward projections stay for ORE file and CSV export. The import boundary gains the inverse projection (ORE key to oresmd identifier). The NATS wire is oresmd-native, not a projection.
- ORE text (market.txt, fixings.txt) and CSV stay as boundary formats; nothing ORE-shaped is stored.
- Coverage gap: oresmd models seven asset classes. ORE series types without an oresmd identifier (e.g. BOND) are rejected at import until oresmd grows them.
- The URI builder decouples from refdata: the caller supplies a container of canonical values (tenor, point). The builder matches against it and rejects unknown spellings; no direct repository dependency.
- The inverse projection reuses the
fx_quote_convention_checker: the pair swap applies to the identifier at parse time (samecheck(), new target). The checker header moves PUBLIC on ores.marketdata.core. - Fixing keys (
FIXING/RATE/<index-name>) are an index-name key space the registry table does not seed. Their inverse lands with the identity-core task, which wires the fixing boundary.
Out of scope
- ORE XML round-trips beyond conventions (
curveconfig.xml,todaysmarket.xml): not planned. The earlier XML round-trip task was abandoned and tracked separately. The conventions import stays: it feeds the refdata catalogs behind oresmd's enums. - Projection fidelity to ORE/Bloomberg/Reuters key strings: oresmd's projection rules already exist and stay; the question of where the projection boundary lives (if anywhere) is for the analysis task to answer, not new notation work.
- The market data values themselves (quotes, fixings, curves): only identity is in scope.