Task: Fix day-scaled kappa/sigma calibration across short-rate processes

Table of Contents

This page documents a task in the IR Rates synthetic data: dataset seeding, index cleanup, dual-curve, quoting conventions story. It captures the goal, current status, acceptance, and any notes or results.

Goal

Make the "day-scaled default kappa/sigma" claim from the tick-batch-publishing task actually true — for every short-rate process, not just the one that first surfaced the bug. Manual verification of the published synthetic.ir_curve_configs.basic dataset produced unrealistic (~150-250) published rates despite day-scaled kappa=0.5/365.

Confirmed root cause on reading the code (not just the ad hoc verification): every process's one-tick transition treats kappa as a discrete per-tick AR(1) decay (exp(-kappa)), not as an annualised kappa*dt. Day-scaling only divided kappa by 365 and left sigma untouched. A correct discretisation of a continuous-time SDE under dt = 1/365 needs both kappa_dt = kappa*dt and sigma_dt = sigma*sqrt(dt) — omitting the sigma scaling means per-tick variance stays roughly annual-sized regardless of how small kappa gets, which is why every process's simulated rates move around far more than a single day's move should. This is shared, not Vasicek-specific:

  • vasicek_process composes hull_white_process directly (it is Hull-White with constant theta, see its own docstring) — one fix covers both.
  • cir_process::next_stochastic()'s per-tick noncentral-chi-squared scale c = sigma^2*(1-e^{-kappa})/(4*kappa) has the identical shape: as kappa -> 0, c -> sigma^2/4, i.e. it does not shrink with a smaller day-scaled kappa either.

Split out of Seed realistic IR curve sample data because this is a quant/numerics bug in the existing process recursions, independent of sourcing new vintage data or wiring it into the feed.

Status

Field Value
State DONE
Parent story IR Rates synthetic data: dataset seeding, index cleanup, dual-curve, quoting conventions
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-07-22

Acceptance

  • hull_white_process=/=vasicek_process=/=cir_process gain an explicit dt (year-fraction per tick) parameter, defaulting to 1.0 (full backward compatibility); kappa=/=theta=/=sigma=/ =initial_rate stay in their natural annualised units, never pre-scaled by callers.
  • discount_factor()'s per-tick bond-time accumulation is dt-aware for both Hull-White's recursion and CIR's closed form (not just the state-transition side) — this is the actual source of the ~150-250% published-rate bug, not sigma scaling.
  • process_factory::make_yield_curve_process() and every ores.synthetic call site thread dt through explicitly (1/365 for the IR curve feed's day-per-tick convention), replacing the SQL- embedded kappa/365=/=sigma/sqrt(365) scaling in synthetic_ir_curve_configs_basic_populate.sql=/ =_realistic_populate.sql, which revert to plain annualised parameters.
  • Root cause and the corrected recursion are documented in doc/knowledge/domain/vasicek_process.org, hull_white_process.org, cir_process.org, including the QuantLib cross-check and attribution.
  • The published synthetic.ir_curve_configs.basic dataset produces realistic rates end-to-end (verified via the same manual verification path that surfaced the bug) for at least a 2Y entry, for each process type the dataset exercises.
  • Comprehensive test coverage in vasicek_process_tests.cpp=/=hull_white_process_tests.cpp=/ =cir_process_tests.cpp (see Plan): flat-curve reproduction across a dt sweep (year/day/hour/minute), cross-dt consistency, degenerate small-kappa branch sanity, and default-dt backward-compatibility (existing assertions unchanged).

Plan

Cross-checked against QuantLib (/mnt/development/Development/OreStudio/Engine.remote/QuantLib): ql/processes/ornsteinuhlenbeckprocess.cpp, ql/processes/coxingersollrossprocess.cpp, ql/models/shortrate/onefactormodels/vasicek.cpp, hullwhite.cpp. Confirms the fix direction and supersedes the sigma*sqrt(dt) framing above — the actual numeric root cause is more specific and doesn't need caller-side sigma pre-scaling at all once dt is explicit (see the parent story's * Decisions for the full writeup):

  • QuantLib keeps speed=(kappa) and =dt always separate: exp(-speed*dt), variance = sigma^2*(1-exp(-2*speed*dt))/(2*speed) — never a pre-scaled composite kappa.
  • Applying that to hull_white_process: the only change needed in next() is decay = exp(-kappa_ * dt_) (was exp(-kappa_)); the existing var = (1-decay^2)/(2*kappa_) formula is already correct once decay itself is dt-aware (dividing by annualised kappa_ was never the bug — the missing *dt in the exponent was).
  • The actual, more damaging bug is in discount_factor(): its next_b = 1.0 + b*decay treats every tick as one full year of bond-time regardless of what a tick represents. Fix: next_b = dt_ + b*decay. For a 730-daily-tick 2Y entry this was integrating exp(-730*rate_) instead of exp(-2*rate_) — the actual source of the ~150-250% published rates, not sigma at all.
  • cir_process: next_stochastic()'s decay=/=c=/=lambda need the same exp(-kappa_*dt_) fix; discount_factor()'s closed form needs tau = ticks_ahead * dt_ (was raw ticks_ahead) — CIR already uses a closed form (no iteration), so this is a one-line fix once dt is threaded through, unlike Hull-White's recursion.
  • Degenerate small-kappa branches (Hull-White's kappa < 0=, both processes' driftless/ODE paths) get the same *dt treatment (b + dt_=, rate_ + sigma_*sqrt(dt_)*z=), following QuantLib's speed < sqrt(QL_EPSILON) algebraic-limit pattern rather than a bare < 0= guard.
  • dt defaults to 1.0 (one tick per year) on every constructor — exactly today's behaviour, so this is purely additive for every existing caller/test that doesn't pass it.
  • process_factory::make_yield_curve_process() gains the same dt parameter (default 1.0); ores.synthetic's three call sites (ir_curve_feed.cpp, ir_curve_preview_handler.hpp x2) pass 1.0/365.0, matching ir_curve_template_resolver's already- documented "1 tick == 1 calendar day" convention.
  • synthetic_ir_curve_configs_basic_populate.sql=/ =_realistic_populate.sql revert their kappa/365=/=sigma/sqrt(365) SQL-embedded scaling back to plain, real annualised Vasicek parameters — the scaling now lives in tested C++, not SQL comments.

Notes

Manual scenario run surfaced a real gap this fix's original scope missed: IrCurveEditor's (Qt) Process tab – the Simple-mode κ/σ sliders, their tooltips, and a new curve's default κ/θ/σ values – was never updated from the old day-scaled convention. Its slider ranges were hardcoded to [0, 0.02]=/[0, 0.005]= (assuming day-scaled kappa ~0.0007-0.0015, sigma ~0.0004-0.0007), so once the SQL populate scripts reverted to real annualised values (kappa ~0.2-0.55, sigma ~0.006-0.022) the sliders pegged visually at their max even though the underlying stored value was correct. Fixed: slider ranges to [0, 2.0]=/[0, 0.05]=, tooltips to describe annualised (not per-tick) magnitudes with the real seeded ranges, and a new curve's defaults from kappa=0.001,sigma=0.0005 to kappa=0.35,sigma=0.01.

Also flagged during the same run: published rates read as "a tad small" in the Curve Snapshot chart (raw decimal, e.g. 0.03022 rather than 3.022% or 302.2bp). The rates themselves are correct (EUR ESTR hovering ~2.98-3.04%, exactly theta≈3% plus a few bp of realistic vol) – this is a display-scale question, not a calibration bug, and is already the exact scope of the sibling IR rate display/quoting conventions task in this story, so it stays there rather than expanding this task's scope.

Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario, run through the QA Validation Runner panel) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
Verify day-scaled kappa/sigma calibration fix for IR curve feeds PASSED All 5 steps passed after the IrCurveEditor slider fix; AUD/AONIA curve snapshot showed plausible, moving rates (4.17-4.28%) across every tenor.

PRs

PR Title
#1661 [analytics.quant,synthetic,sql] Fix day-scaled kappa/sigma calibration across short-rate processes

Review

Three independent @claude review passes on #1661, all issue-level (no line comments): all three found no correctness/security/ maintainability issues and marked the PR ready to merge. Non-blocking observations only:

# Comment summary File Decision Notes
1 dt=1.0/365.0 duplicated as two separately-named constants (ir_curve_feed.cpp's dt, ir_curve_preview_handler.hpp's preview_dt) ir_curve_feed.cpp, ir_curve_preview_handler.hpp Accepted Consolidated into one ir_curve_feed_dt constant in ir_curve_template_resolver.hpp, next to the "1 tick == 1 calendar day" convention it implements; both call sites now reference it.
2 dt isn't validated by the shared process_parameter_validation layer (Qt client's validation surface), only by each process constructor's own throw process_parameter_validation.hpp/.cpp Declined dt is never user-facing input (fixed 1/365 convention set in code, not an editable config field), so there is nothing for the shared UI-facing validator to validate; the constructor throw is the only meaningful validation site.
3 preview_shape() passes a literal 42 seed rather than deriving one from req, unlike simulate_paths() ir_curve_preview_handler.hpp Declined Not a regression from this PR – matches prior behaviour (the old 4-arg call relied on the constructor's implicit seed=42 default); out of scope for a dt-focused fix.

Result

Root-caused and fixed the day-scaling bug shared by every discrete- time short-rate process (Vasicek/Hull-White, CIR) plus OU: the actual bug was discount_factor()'s bond-time accumulation treating every tick as a full year regardless of what a tick represents (+1.0 per tick instead of +dt), not the sigma scaling first suspected — confirmed by cross-checking against QuantLib's own process sources, which always keep speed=/=dt as separate arguments.

Shipped: an explicit dt (year-fraction per tick) constructor parameter on all four processes, defaulting to 1.0 for full backward compatibility; process_factory and every ores.synthetic call site thread dt=1/365 through explicitly; the SQL populate scripts reverted to plain annualised parameters; 191 new dt-tagged Catch2 assertions across all four processes (flat-curve exactness, discretisation convergence, statistical variance scaling, degenerate- branch sanity); three knowledge docs updated with the QuantLib cross- check; and a real regression the fix's own manual verification surfaced — IrCurveEditor's κ/σ sliders/tooltips/defaults, still calibrated for the old day-scaled magnitudes — fixed in the same PR.

All acceptance criteria met. Manual end-to-end verification (PR #1661) PASSED: AUD/AONIA curve snapshot showed plausible, moving rates (4.17-4.28%) across every tenor 1M-10Y, confirming the original "~150-250% blown-up, frozen far point" symptom is gone. Full build green, rat test suite green, schema validation clean, three independent @claude review passes found no correctness issues.

Emacs 29.3 (Org mode 9.6.15)