Story: Synthetic data collections: Basic and Realistic

Synthetic data librarian support: FX foundation gave every party a single synthetic FX collection (the ore_analytics dataset bundle). This story adds a second sample collection so parties can pick between two archetypes, and closes the gaps that surfaced while designing it:

• Two named collections: Basic (simple/exaggerated process dynamics, easy to eyeball on a chart while exercising every UI feature) and Realistic (calibrated to look like actual FX behaviour). Both seed their initial spot from the same real reference data — the difference between them is process dynamics only, not the starting point. A bad/arbitrary starting spot isn't "basic," it's just wrong.
• Reference/vintage data as market data, not a synthetic-service concept: a "vintage" is real, dated market_observation rows (asset_class::fx, tagged with a distinguishing source, e.g. ore.reference), imported once from ORE's own bundled example data into ores.marketdata — reusing the exact schema fields already designed for this (source, point_id, observation_datetime), not a new table or a synthetic-specific store. ores.synthetic is a consumer of this at feed-start (asks ores.marketdata "what was FX/RATE/EUR/USD as of 2016-02-05 from source ore.reference?"), never a store of it — so it's reusable by future IR/vol-surface work without any synthetic-specific plumbing, and scales to more vintages as just more dated rows, never more collection names.
• Market Simulator UI groups by collection, not by feed type: today the tree's top-level node is a feed-source label ("Synthetic FX Spot: EUR/USD") with all configured pairs flattened under it regardless of which collection they belong to. It should group by collection (Basic / Realistic) instead, since every entry can be enabled simultaneously across collections and users need to tell them apart at a glance.
• Feed/subject namespacing by collection: NATS subjects and source_name values collide across collections today (both would publish as synthetic.eurusd). Prefix with the collection: synthetic.basic.eurusd / synthetic.realistic.eurusd.
• Vintage-availability guard: if the reference-data import hasn't been run, starting a Realistic feed should fail with an actionable error pointing at the import, not silently default to a wrong value or crash deep in feed-start logic.

• ORE's bundled example data is anchored overwhelmingly to one date: 2016-02-05 appears in 288 of 392 example Input/*.xml asofDate parameters (≈73%); the remainder is a long tail of one-off dates, no second dominant cluster.
• Real FX spot values as of that date, from external/ore/examples/Legacy/Example_56/Input/market.txt (the richest FX quote set at that date): EURUSD 1.132337, GBPUSD ≈=1.3946= (derived via EUR/GBP), EURCHF 1.125046, USDCHF ≈=0.9936=, EURJPY 128.15.
• market.txt format: plain whitespace-delimited <date> <quote-name> <value>, e.g. 20160205 FX/RATE/EUR/USD 1.132337 — trivial to parse.
• ores.marketdata.core::service::import_service already exists (projects/ores.marketdata/core/include/ores.marketdata.core/service/import_service.hpp) — this story reviews/extends it rather than building import functionality from scratch.

• Existing ores.marketdata import functionality (import_service=/=import_protocol) is reviewed against the need to import ORE's market.txt reference data as dated market_observation rows tagged with a distinguishing source; extended if needed, not rebuilt.
• A Basic and a Realistic dataset bundle both exist (alongside the existing ore_analytics bundle, or replacing/extending it — decide during implementation), each seeding its FX configs' initial price from the same imported 2016-02-05 reference vintage.
• Basic and Realistic differ only in GMM/process parameters, not in starting spot.
• ores.synthetic validates vintage-data availability before a feed that depends on it starts — both as an on-demand "validate every entry" toolbar action in Market Simulator, and as an implicit check at feed-start — surfacing a clear, actionable error (pointing at the import step) rather than a silent bad default or a deep crash.
• Market Simulator's feed tree groups by collection (Basic / Realistic) as the top-level node, not by feed-source label.
• Synthetic feed source_name=/NATS subject is namespaced by collection: =synthetic.basic.<pair> / synthetic.realistic.<pair>, so two collections can run the same pair concurrently without colliding.

• Discovered while starting the vintage-import task: no non-interactive way exists to trigger import_market_data_request (only the Qt ImportTradeDialog calls it). Spun off as its own story, ores.shell command for market data import, rather than building shell tooling inline here — it's independently useful (any future scripted/provisioning-time import needs it too), and keeps this story's tasks focused on collections rather than shell infrastructure. Import the 2016-02-05 ORE reference vintage is blocked on it.
• Also found while sourcing reference values: a systematic bug across several vendored ORE example market.txt files — FX/RATE/USD/GBP and FX/RATE/USD/CHF rows storing values that are mathematically EURUSD/EURGBP=/=EURUSD/EURCHF (i.e. the GBP/CHF-in-USD rate), not USD-in-GBP/CHF as the key order implies, inconsistent with every other EUR/xxx row in the same files. Reported upstream to the ORE project with the full file list (10+ instances found across Legacy, ORE-API, ORE-Python, XvaRisk, InitialMargin). Only fixed locally where provably safe: XvaRisk/Input/market_20160205_eonia_200bp_up.txt, XvaRisk/Input/market_20160205_eur6m_200bp_up.txt, and ORE-Python/Notebooks/Example_7/Input/market.txt — each has its own local curveconfig.xml with zero reference to the old key anywhere. Left unfixed (bug still present in the vendored tree, deliberately not patched): Legacy/Example_56, Legacy/Example_44, InitialMargin/Input/Simm/simm_market.txt — these three have no local curveconfig.xml and resolve the shared external/ore/examples/Input/curveconfig.xml (via curveConfigFile = ../../../Input/curveconfig.xml in their SIMM run configs), which itself has a literal <SpotRate>FX/RATE/USD/GBP</SpotRate>=/=<SpotRate>FX/RATE/USD/CHF</SpotRate> wiring — initially fixed then reverted after a PR review caught that this shared-config path couldn't be ruled out as live without actually running the SIMM examples. ORE-API, ORE-Python/Notebooks/Example_9, Example_4, Example_3 were investigated and left unfixed for the same reason (their own local curveconfig.xml directly wires the old key). The ~20 further instances the upstream issue lists beyond these were not individually verified safe/unsafe — flagged in the issue for ORE to triage, not something this story attempted to fix.
• Fixing the same bug's blind spot: import_service's fixings-import loop didn't set market_fixing.source even though the field exists with identical intent to market_observation.source — the observations loop got tagged, fixings didn't. Fixed symmetrically.
• Vintage/reference data lives in ores.marketdata as ordinary (dated, source-tagged) market_observation rows, not as a new synthetic-specific table or store. ores.synthetic queries it at feed-start; it never owns or caches a private copy. Rationale: reuses the exact schema fields (source, point_id, observation_datetime) already designed for this in the IR Rates story's investigation; scales to more vintages as more rows, not more collection names or schema changes; reusable by future IR/vol-surface work for free.
• Collections differ only in process parameters, never in starting spot — both Basic and Realistic seed from the same reference vintage. A collection that starts from a nonsensical spot isn't "basic," it's broken.
• Vintage-availability validation is both a toolbar "validate all" action and an implicit start-time check — not one or the other. The toolbar action lets a user audit an entire collection proactively; the start-time check is the safety net for anyone who skips it.

• Adding vintages beyond 2016-02-05 (this story imports and uses one; the reference-data mechanism is designed to support more later without rework, but adding a second vintage is follow-on work).
• Asset classes beyond FX (matches the scope boundary already set by the FX foundation story).
• Curve bootstrapping / vol surfaces (separate stories, per IR Rates synthetic data generation's scope reasoning).

Both Basic and Realistic collections run concurrently for the same party with no NATS subject or market_observation.source collisions. What started as a namespacing task grew into a real, queryable hierarchy: a generic self-referencing ores.synthetic.folder entity (root > collection > asset class > instrument type > feed) drives the Market Simulator tree directly, and backs a folder-scoped start/stop request so "start everything under Basic" (or the whole Synthetic root) is one server-resolved call instead of per-feed client enumeration – usable identically from Qt, ores.shell, or a wt workflow step. A vintage-validity indicator (computed live, not persisted, to stay accurate across data imports with no feed edit) rounds it out, shown as small badge emblems on the tree rather than separate columns after manual QA flagged the first cut as unreadable.

Two real bugs were found and fixed along the way, both worth remembering: market_data_generation_config had party_id wrongly declared a Natural Key (generating a DB constraint that capped a party at one collection ever), and the folder-scoped start endpoint initially failed to forward the caller's bearer token into the vintage-availability check, so it ran under the service's own system-tenant identity instead of the caller's and silently saw no data for any real tenant.