Task: Review and extend ores.marketdata import for ORE reference vintages
This page documents a task in the Synthetic data collections: Basic and Realistic story. It captures the goal, current status, acceptance, and any notes or results.
Goal
ores.marketdata.core::service::import_service
(projects/ores.marketdata/core/include/ores.marketdata.core/service/import_service.hpp)
already exists. Review it (and import_protocol) against what's
needed to import ORE's market.txt reference data: plain
whitespace-delimited <date> <quote-name> <value> rows (e.g.
20160205 FX/RATE/EUR/USD 1.132337), written into market_observation
with a distinguishing source (e.g. ore.reference) and
observation_datetime set to the vintage date. Extend only the gaps
found — do not rebuild import functionality that already exists.
Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Synthetic data collections: Basic and Realistic |
| Now | Done. |
| Waiting on | Nothing. |
| Next | Move to the next task. |
| Last touched | 2026-07-03 |
Acceptance
- Documented gap analysis: what
import_service=/=import_protocolalready supports vs what's missing formarket.txt-shaped, source-tagged, dated FX reference imports. - Any extension is additive to the existing service, not a parallel import mechanism.
Plan
Gap analysis. ores.marketdata.core::service::import_service
(projects/ores.marketdata/core/src/service/import_service.cpp)
already does almost everything needed:
- Parses ORE's
market.txtformat exactly as needed — plain whitespace/comma-delimited<date> <quote-name> <value>, bothYYYYMMDDandYYYY-MM-DDdate formats (via the sharedores::ore::market::parse_market_dataparser) — no new parser required. - Decomposes ORE quote-name keys (
FX/RATE/EUR/USDetc.) intoseries_type=/=metric=/=qualifiervia the existing series key registry, auto-creating themarket_seriescatalog entry if it doesn't exist. - Writes
observation_datetimefrom the file's own date column — so importing20160205 FX/RATE/EUR/USD 1.132337already lands at the correct vintage date with zero extra plumbing. - Exposed over NATS (
marketdata.v1.import) and already has one caller (ImportTradeDialog.cppinores.qt.trading), so the request/response shape and wiring are proven.
The one real gap: market_observation.source was never set —
import_service.cpp:178 wrote point_id but nothing touched
source, so every imported observation landed with an empty source,
making "give me only the ore.reference vintage" unqueryable.
Extension made: added an optional source field to
import_market_data_request
(projects/ores.marketdata/api/include/ores.marketdata.api/messaging/import_protocol.hpp),
defaulting to empty string (preserves existing behaviour for the
ImportTradeDialog caller, which doesn't set it), and wired
obs.source = req.source; into the observation-building loop in
import_service.cpp. Purely additive — no signature break, no new
mechanism.
PR #1423 review round 1 caught: the fixings-import loop right below
the observations loop built domain::market_fixing without setting
its source field, even though it exists with identical intent
(market_fixing.hpp:80). Fixed symmetrically: fix.source = req.source;.
Notes
PRs
| PR | Title |
|---|---|
| #1423 | [marketdata] Synthetic data collections WIP: import source tagging, ORE key fixes |
Review
| # | Comment summary | File | Decision | Notes |
|---|---|---|---|---|
| 1 | market_fixing.source not wired despite identical intent to market_observation.source | import_service.cpp | Fixed | Added fix.source = req.source; symmetric to the observations loop |
| 2 | Shared external/ore/examples/Input/curveconfig.xml also wires FX/RATE/USD/GBP-CHF, resolved by Example_56/Example_44/InitialMargin-Simm (previously believed safe) via curveConfigFile=../../../Input/curveconfig.xml in their SIMM run configs | market.txt (3 files) | Fixed (reverted) | Reverted the 3 files back to original reversed keys — couldn't rule out the shared-config path being live without running the SIMM examples; only XvaRisk (x2) and Example_7 confirmed safe (own local curveconfig.xml, zero reference) |
| 3 | "4 more instances" undercounts the bug's footprint; story doc stale relative to later commits | story.org | Fixed | Decisions section rewritten with the full, accurate fixed/reverted/unverified file breakdown; filed capture 3BC19819-6BEB-456E-9647-F6AB258A6635 for the vendor-refresh-drift risk on the 3 files actually fixed |
| 4 | No unit test coverage for import_service | import_service.cpp | Declined | Pre-existing gap, not introduced by this PR; no import_service test suite exists at all yet |
Result
import_service already handled parsing, key decomposition, and
date-from-file — the only missing piece was tagging observations with
a source. Added an optional source field to
import_market_data_request and wired it through to
market_observation.source in the import loop; empty-string default
preserves the existing ImportTradeDialog caller's behaviour
unchanged. Built ores.marketdata.core.lib and ores.qt.trading.lib
clean (clang debug/make). Unblocks
Import the 2016-02-05 ORE reference vintage, which can now tag its
import with source = "ore.reference" and query for it precisely.