Story: IR Rates synthetic data: dataset seeding, index cleanup, dual-curve, quoting conventions

Goal
Status
Acceptance
Tasks
Decisions
- A synthetic theme is one atomic DQ dataset, not one per asset class
- dt is a first-class, explicit process parameter (not caller-side pre-scaling)
dt)=, variance(speed, sigma, dt)), never a caller-pre-scaled
Out of scope

This page documents a story in Sprint 25. It captures the goal, current status, acceptance criteria, and the tasks that compose it.

Goal

Extend IR Rates synthetic data generation (Sprint 23, DONE) with realistic dataset seeding, floating-index catalogue cleanup, a proper dual-curve model, and correct IR quoting conventions — none of which its own acceptance (short-rate engine, Curve Template, tenor-collision validation, tick-batch publishing) required.

Status

Field	Value
State	STARTED
Parent sprint	Sprint 25
Now	Not yet started. Pulled into Sprint 24 (mission includes improving IR generation).
Waiting on	Nothing.
Next	Any of the four tasks is independently startable.
Last touched	2026-07-21

Acceptance

A realistic vintage IR rate dataset is seeded, with vintage-aware seeding and basic+realistic synthetic.ir_curve_configs DQ datasets mirroring the existing FX pattern.
The duplicate flat CCY-INDEX catalogs (floating_index_type, overnight_index_convention) are consolidated into an index entity + currency_index junction.
The single self-discounting curve is extended to a proper discounting/projection dual-curve model with a basis spread.
IR rate display/quoting conventions (basis points, per-index precision) mirror currency_pair_convention's pip_factor/tick_size/decimal_places pattern, with a shared Qt formatter.

Tasks

Task	State	Start	End	Description
Seed realistic IR curve sample data (vintage + DQ datasets)	DONE	2026-07-22	2026-07-26	Source a vintage IR rate dataset, add vintage-aware seeding to ir_curve_feed, day-scale default kappa/theta/sigma for the 1-tick=1-day convention, and build basic+realistic synthetic.ir_curve_configs DQ datasets mirroring the existing FX pattern.
Normalize floating-rate index reference data	ABANDONED	2026-07-28	2026-07-28	Superseded by oresmd; follow-on tracked on the migration capture instead.
Dual-curve (discount + projection) short-rate model	BACKLOG			Extend the single self-discounting curve to a proper discounting/projection dual-curve model with a basis spread, closer to real market practice.
IR rate display/quoting conventions (basis points, per-index precision)	ABANDONED	2026-07-22	2026-07-22	Superseded before implementation by Rate display conventions (FX has the same gap; solved together, not designed twice).
Source vintage historical IR rate dataset + populate script	DONE	2026-07-22	2026-07-24	Source a small set of real historical (currency, tenor, date) IR points from a public source (FRED/ECB/BoE) and add a vintage populate script mirroring marketdata_fx_driver_rates_populate.sql, covering enough currencies/tenors to seed DEPO/FRA/IRS entries for a basic Curve Template.
Fix day-scaled kappa/sigma calibration across short-rate processes	DONE	2026-07-22	2026-07-22	Root-cause and fix the day-scaling bug shared by Vasicek/Hull-White and CIR: day-scaling only divides kappa by 365, leaving sigma at its annual scale, so per-tick variance stays roughly annual-sized regardless of how small kappa gets.
Rename CIR, OU, and GMM processes to their proper names	DONE	2026-07-26	2026-07-26	Rename cir_process/ou_process/gmm_process/arithmetic_gmm_process to Cox-Ingersoll-Ross/Ornstein-Uhlenbeck/Gaussian Mixture Model, matching QuantLib's naming and this codebase's other spelled-out process names; includes fixing IrCurveEditor's combo, which shows the raw "CIR" code instead of a spelled-out label.
Merge per-asset-class synthetic collection folders into one theme unit	DONE	2026-07-27	2026-07-28	FX and IR curve publish-from-dq functions each create their own collection folder keyed by dataset-scoped config id, so a single theme (e.g. 2016 ORE Samples) shows as two duplicate top-level Market Simulator tree nodes instead of one; fix so all asset classes of a theme share one folder/start-stop unit, scaling to future asset classes without one click per class.
Publish FX driver rates for every synthetic theme vintage during party provisioning	BACKLOG			party_provisioning_bundle_plan() only publishes the marketdata.reference_vintage_2016_02_05 FX driver-rates bundle, so the 2026 Realistic synthetic theme (which needs fed.h10.2026-05-05) always shows all-red vintage-invalid badges after a standard party provision until someone manually publishes the missing bundle; publish every vintage bundle a party's synthetic themes actually need.
Fix vintage-validity false negative: find_series never matches an existing series	BACKLOG			Every FX feed shows vintage-invalid (red) even for themes with genuinely matching, correctly-scoped market_observation and market_series rows (verified directly in Postgres for 2016 ORE Samples/Uniform Volatility Demo); service logs show market_data_client::find_series() (ores.marketdata.client/src/market_data_client.cpp) never leads to a market_observations.list call for any of the 37 feeds checked, meaning it fails to match a series that demonstrably exists in its own NATS response – investigate and fix the match/response-handling bug.
Rename synthetic themes and add 2026 Realistic vintage dataset	DONE	2026-07-23	2026-07-24	Rename the 'basic'/'realistic' synthetic FX+IR themes to their clearer identities (Uniform Volatility Demo, 2016 ORE Samples), split out a new 2026 Realistic theme sourced from a real 2026-05-05 FX driver-rates vintage, add the QA screenshot capture fix needed to test it under WSLg, and file the follow-up tasks the manual test scenario surfaced.
Fix vintage-mode preview chart using stale r0/initial price instead of resolved rate	BACKLOG			IrCurveEditor's and the FX Spot editor's Process/sample-paths chart preview is parameterized from the local editor's r0/gmm_initial_price spin value, which for a vintage-mode config just holds an unused leftover default – the server resolves the real seed from the vintage observation instead. The chart should be centered on the resolved vintage rate, not the stale local value. Check both IR curve and FX spot editors.
Extend SyntheticBindingDialog to cover IR curve configs	DONE	2026-07-28	2026-07-29	SyntheticBindingDialog only ever lists fx_spot_generation_config records and bulk-creates feed bindings for them; ir_curve_generation_config has no equivalent, so IR curve feed ticks can never be persisted into market_observations even manually, unlike FX. Generalize the dialog (or add a sibling) to cover IR curves too, matching this story's own goal of bringing IR curves to FX parity.
Fix Browse-available-vintages dialog: slow unfiltered fetch and detached window	BACKLOG			Discovered while manually verifying vintage-aware IR curve seeding: both FxSpotRateEditor::onBrowseVintageClicked and IrCurveEditor::onBrowseVintageClicked fetch up to 10000 market series then up to 10000 observations per series with no server-side qualifier filter (get_market_series_request/get_market_observations_by_series_id_request only support offset/limit), making the resulting picker dialog very slow to appear. The picker (QDialog dialog(self)) is also not properly rooted to the MDI main window – it shows as a detached top-level window rather than centered/owned like other modals in the app. Pre-existing in both editors, not introduced by this story's IR curve work, which only mirrored the existing FX pattern.
Feed bindings have no product-kind filter: FX Spot grid renders IR bindings as garbage currency-pair rows	DONE	2026-07-29	2026-07-29	SyntheticBindingDialog can now bind IR curve configs (this story's own change), but ores_marketdata_feed_bindings_tbl / feed_binding has no kind/product-type discriminator at all – just ore_key and source_name. FxSpotGridWindow::reload() fetches ALL enabled feed_bindings via get_feed_bindings_request unfiltered and renders every row as a currency pair via pair_from_ore_key(), so an IR curve binding (e.g. USD/LIBOR-3M) now shows up in the FX Spot screen misparsed as an FX pair. Surfaced during manual QA of the SyntheticBindingDialog IR-curve task (scenario step 'Confirm no regression to the FX-only flow' failed) – separately, CRM topology (majors/minors/exotics) is also missing after Barclays Plc provisioning because the realistic_2026/base bundle used by 'provision system'/'provision party' never publishes refdata.crm_topology_bundles (only the older marketdata.reference_vintage_2016_02_05 bundle does); that's a provisioning-bundle gap, not a code filtering bug, and may warrant its own task if not already tracked. Root cause for the FX Spot issue: no product-kind field on feed_binding, and no consumer-side filtering by kind.
QA Market Simulator tree Start/Stop-at-Root for merged theme datasets	DONE	2026-07-29	2026-07-29	Verify the Market Simulator Qt tree's Start/Stop-at-Root behaviour and run the Synthetic data themes manual test scenario end-to-end against the new one-dataset-per-theme model, and fix any issues found.
Unify synthetic config/binding topology: single cross-asset-class query with a real kind discriminator	BACKLOG			Synthetic config/binding is siloed per asset class with no unified topology query: each asset class has its own request/response message pair and domain config shape, so any cross-asset-class consumer (SyntheticBindingDialog, grids, CRM screens) must know every asset class ahead of time, fire one request per kind, and manually merge/hand-tag results client-side. feed_binding itself has no kind/product-type field, so consumers fetching all bindings can't filter by kind (symptom: FxSpotGridWindow renders IR bindings as FX pairs). Replace the per-asset-class fan-out with a single query returning all asset classes' configs together with an explicit, wire-carried and persisted kind discriminator, so adding a new asset class doesn't require every cross-cutting consumer to learn a new message type.
Improve Market Simulator status icons: clearer running/stopped and an auto_start=false indicator	BACKLOG			The Market Simulator tree's blue/yellow icons for feed running/stopped state are not visually distinct enough – surfaced during manual QA of the merged-theme Start-at-Root flow, where a curve (2026 Realistic PLN-POLONIA) that backend logs confirm started and is publishing ticks fine looked, at a glance, like it hadn't started. There is also no visual indicator at all for auto_start=false feeds (e.g. Uniform Volatility Demo's IR curves, which deliberately don't bulk-start due to tenor collisions with 2026 Realistic) – a tester has no way to tell, without checking the config, why a feed didn't start along with its theme. Design and implement clearer status iconography for both dimensions.
Fix realistic_2026 IR curve seed data / index_family enum mismatch	DONE	2026-07-30	2026-07-30	Fixes the index_name/index_family regression left by df398d5f1 that currently blocks IR curve theme provisioning.
Fix missing dataset dependency edges for currency refdata datasets	DONE	2026-08-09	2026-08-09	Fresh db recreate + Acme provisioning aborts the base refdata publish at step 0: publication_service::resolve_publication_order() topologically sorts bundle members, and datasets with no declared dependency edges become sinks that float to the front. refdata.currency_pairs has no edge to iso.currencies, so it publishes before any currencies exist (error: Invalid base_currency EUR, no active currency found), the 38-step workflow aborts, and nothing else in the base bundle publishes, which cascades into the acme_lei_import failing on missing business centres (error: Invalid business_centre GBLO, must be one of WRLD). Same bug class fixed for refdata.calendars depending on iso.countries in commit 9527e7590; the currency siblings were never given edges.

Decisions

A synthetic theme is one atomic DQ dataset, not one per asset class

The merge-folders task found the atomicity gap one layer below where the tree-node duplication symptom appeared: publication_service::publish() already grouped a theme's datasets into a bundle, but looped over the bundle's members and published each independently, so a party could end up with FX published and IR not (or vice versa). Rejected folder-reuse-by-name (fixes the cosmetic symptom only) and caller-side transaction wrapping (atomicity as a promise, not a data-model property) in favour of collapsing each theme's two per-asset-class dataset rows into one (synthetic.themes.<theme>), published by a single ores_synthetic_publish_theme_from_dq_fn in one transaction. Design note: Synthetic Theme as One Atomic DQ Dataset.

dt is a first-class, explicit process parameter (not caller-side pre-scaling)

The calibration-fix task's root-cause analysis, cross-checked against QuantLib's own OrnsteinUhlenbeckProcess=/=CoxIngersollRossProcess=/=Vasicek=/ =HullWhite sources (Engine.remote/QuantLib/ql/processes/, ql/models/shortrate/onefactormodels/): every QuantLib formula keeps speed (kappa) and dt as two always-separate arguments (=exp(-speed

dt)=, `variance(speed, sigma, dt)`), never a caller-pre-scaled

composite. Our synthetic_ir_curve_configs_basic_populate.sql instead computed kappa/365 and sigma/sqrt(365) as SQL constants — quant scaling logic with no tests, living outside ores.analytics.quant entirely. Decision: dt (year-fraction per tick) becomes an explicit constructor parameter on hull_white_process=/=cir_process (and vasicek_process, forwarding), defaulting to 1.0 for full backward compatibility; kappa=/=theta=/=sigma=/=initial_rate stay in their natural annualised units always, and callers never pre-scale them. process_factory::make_yield_curve_process() gains the same dt parameter, threaded from ir_curve_template_resolver's already- documented "1 tick == 1 calendar day" convention. The SQL populate scripts revert to plain, real annualised Vasicek parameters.

This also fixed a second, independent bug the day-scaling investigation surfaced: discount_factor()'s backward recursion accumulated +1.0 of bond-time per tick regardless of what a tick represents, rather than +dt — for a 730-daily-tick 2Y entry this integrated ~730 years of a rate meant to run for 2, producing the ~150-250% published rates that triggered this investigation. cir_process::discount_factor()'s closed form had the identical defect (tau = ticks_ahead used as literal elapsed years).

Domain knowledge from QuantLib, not its OOP shape

QuantLib's StochasticProcess1D inheritance, Handle<YieldTermStructure> observer wiring, and virtual drift()=/=diffusion() are deliberately not adopted — this codebase stays data-oriented (plain structs/free functions over flat parameter arrays), both for its own conventions and because a GPU-batched stochastic- process backlog capture wants these processes parallel-simulation- friendly, which an object-per-process virtual-dispatch design works against. What is adopted is QuantLib's domain knowledge: the explicit speed=/=dt separation above, degenerate small-kappa algebraic-limit branches (QuantLib branches at speed < sqrt(QL_EPSILON) rather than only guarding speed < 0=), and its testExtendedCoxIngersollRossDiscountFactor test pattern (calibrate to a flat-rate curve, assert the model reproduces exp(-r*(T-t)) exactly) — adapted into a per-dt regression test in this codebase's own Catch2 style, not ported as QuantLib code.

Spelled-out process names extend to stored process_type values

The rename task's Goal originally scoped the cir=/=ou rename to identifiers the codebase controls directly, leaving the stored process_type wire/DB value as a decision to make explicit later. Decision: the stored values also become the full spelled-out form (cir=/=CIR → cox_ingersoll_ross=/=COX_INGERSOLL_ROSS, ou → ornstein_uhlenbeck), consistent with the class/file rename rather than leaving a short-code/spelled-out-class split. Since the dev database is rebuilt from scratch via db recreate (no live data to migrate), this needed no migration script — just updated SQL check constraints, the yield_curve_process_types catalogue row, and every C++/Qt string literal comparing against these codes.

Base-bundle datasets must declare their refdata soft-FK dependencies

The dependency-edge fix task codified the rule the calendars fix (commit 9527e7590) applied to one dataset: any DQ dataset whose publish-time insert validates values against another dataset's published rows (a soft FK, enforced by the insert trigger) must declare that dependency via ores_dq_dataset_dependencies_upsert_fn in its seed script. publication_service::resolve_publication_order() topologically sorts bundle members and floats edge-less sinks to the front, so a missing edge silently reorders the publish — with no error until the trigger hard-raises and aborts the whole workflow (the base bundle aborted at step 0 on refdata.currency_pairs, cascading into the LEI import). The currency refdata family now declares its edges (currency_pairs→iso.currencies, pair_conventions→currency_pairs, currency_calendars→refdata.calendars+iso.currencies, currency_countries→iso.countries+iso.currencies, conv_calendars→pair_conventions+refdata.calendars) with roles naming the reference kind; new datasets must follow suit.