Task: Fix day-scaled kappa/sigma calibration across short-rate processes

Table of Contents

This page documents a task in the IR Rates synthetic data: dataset seeding, index cleanup, dual-curve, quoting conventions story. It captures the goal, current status, acceptance, and any notes or results.

1. Goal

Make the "day-scaled default kappa/sigma" claim from the tick-batch-publishing task actually true — for every short-rate process, not just the one that first surfaced the bug. Manual verification of the published synthetic.ir_curve_configs.basic dataset produced unrealistic (~150-250) published rates despite day-scaled kappa=0.5/365.

Confirmed root cause on reading the code (not just the ad hoc verification): every process's one-tick transition treats kappa as a discrete per-tick AR(1) decay (exp(-kappa)), not as an annualised kappa*dt. Day-scaling only divided kappa by 365 and left sigma untouched. A correct discretisation of a continuous-time SDE under dt = 1/365 needs both kappa_dt = kappa*dt and sigma_dt = sigma*sqrt(dt) — omitting the sigma scaling means per-tick variance stays roughly annual-sized regardless of how small kappa gets, which is why every process's simulated rates move around far more than a single day's move should. This is shared, not Vasicek-specific:

  • vasicek_process composes hull_white_process directly (it is Hull-White with constant theta, see its own docstring) — one fix covers both.
  • cir_process::next_stochastic()'s per-tick noncentral-chi-squared scale c = sigma^2*(1-e^{-kappa})/(4*kappa) has the identical shape: as kappa -> 0, c -> sigma^2/4, i.e. it does not shrink with a smaller day-scaled kappa either.

Split out of Seed realistic IR curve sample data because this is a quant/numerics bug in the existing process recursions, independent of sourcing new vintage data or wiring it into the feed.

2. Status

Field Value
State DONE
Parent story IR Rates synthetic data: dataset seeding, index cleanup, dual-curve, quoting conventions
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-07-22

3. Acceptance

  • hull_white_process=/=vasicek_process=/=cir_process gain an explicit dt (year-fraction per tick) parameter, defaulting to 1.0 (full backward compatibility); kappa=/=theta=/=sigma=/ =initial_rate stay in their natural annualised units, never pre-scaled by callers.
  • discount_factor()'s per-tick bond-time accumulation is dt-aware for both Hull-White's recursion and CIR's closed form (not just the state-transition side) — this is the actual source of the ~150-250% published-rate bug, not sigma scaling.
  • process_factory::make_yield_curve_process() and every ores.synthetic call site thread dt through explicitly (1/365 for the IR curve feed's day-per-tick convention), replacing the SQL- embedded kappa/365=/=sigma/sqrt(365) scaling in synthetic_ir_curve_configs_basic_populate.sql=/ =_realistic_populate.sql, which revert to plain annualised parameters.
  • Root cause and the corrected recursion are documented in doc/knowledge/domain/vasicek_process.org, hull_white_process.org, cir_process.org, including the QuantLib cross-check and attribution.
  • The published synthetic.ir_curve_configs.basic dataset produces realistic rates end-to-end (verified via the same manual verification path that surfaced the bug) for at least a 2Y entry, for each process type the dataset exercises.
  • Comprehensive test coverage in vasicek_process_tests.cpp=/=hull_white_process_tests.cpp=/ =cir_process_tests.cpp (see Plan): flat-curve reproduction across a dt sweep (year/day/hour/minute), cross-dt consistency, degenerate small-kappa branch sanity, and default-dt backward-compatibility (existing assertions unchanged).

4. Plan

Cross-checked against QuantLib (/mnt/development/Development/OreStudio/Engine.remote/QuantLib): ql/processes/ornsteinuhlenbeckprocess.cpp, ql/processes/coxingersollrossprocess.cpp, ql/models/shortrate/onefactormodels/vasicek.cpp, hullwhite.cpp. Confirms the fix direction and supersedes the sigma*sqrt(dt) framing above — the actual numeric root cause is more specific and doesn't need caller-side sigma pre-scaling at all once dt is explicit (see the parent story's * Decisions for the full writeup):

  • QuantLib keeps speed=(kappa) and =dt always separate: exp(-speed*dt), variance = sigma^2*(1-exp(-2*speed*dt))/(2*speed) — never a pre-scaled composite kappa.
  • Applying that to hull_white_process: the only change needed in next() is decay = exp(-kappa_ * dt_) (was exp(-kappa_)); the existing var = (1-decay^2)/(2*kappa_) formula is already correct once decay itself is dt-aware (dividing by annualised kappa_ was never the bug — the missing *dt in the exponent was).
  • The actual, more damaging bug is in discount_factor(): its next_b = 1.0 + b*decay treats every tick as one full year of bond-time regardless of what a tick represents. Fix: next_b = dt_ + b*decay. For a 730-daily-tick 2Y entry this was integrating exp(-730*rate_) instead of exp(-2*rate_) — the actual source of the ~150-250% published rates, not sigma at all.
  • cir_process: next_stochastic()'s decay=/=c=/=lambda need the same exp(-kappa_*dt_) fix; discount_factor()'s closed form needs tau = ticks_ahead * dt_ (was raw ticks_ahead) — CIR already uses a closed form (no iteration), so this is a one-line fix once dt is threaded through, unlike Hull-White's recursion.
  • Degenerate small-kappa branches (Hull-White's kappa < 0=, both processes' driftless/ODE paths) get the same *dt treatment (b + dt_=, rate_ + sigma_*sqrt(dt_)*z=), following QuantLib's speed < sqrt(QL_EPSILON) algebraic-limit pattern rather than a bare < 0= guard.
  • dt defaults to 1.0 (one tick per year) on every constructor — exactly today's behaviour, so this is purely additive for every existing caller/test that doesn't pass it.
  • process_factory::make_yield_curve_process() gains the same dt parameter (default 1.0); ores.synthetic's three call sites (ir_curve_feed.cpp, ir_curve_preview_handler.hpp x2) pass 1.0/365.0, matching ir_curve_template_resolver's already- documented "1 tick == 1 calendar day" convention.
  • synthetic_ir_curve_configs_basic_populate.sql=/ =_realistic_populate.sql revert their kappa/365=/=sigma/sqrt(365) SQL-embedded scaling back to plain, real annualised Vasicek parameters — the scaling now lives in tested C++, not SQL comments.

5. Notes

Manual scenario run surfaced a real gap this fix's original scope missed: IrCurveEditor's (Qt) Process tab – the Simple-mode κ/σ sliders, their tooltips, and a new curve's default κ/θ/σ values – was never updated from the old day-scaled convention. Its slider ranges were hardcoded to [0, 0.02]=/[0, 0.005]= (assuming day-scaled kappa ~0.0007-0.0015, sigma ~0.0004-0.0007), so once the SQL populate scripts reverted to real annualised values (kappa ~0.2-0.55, sigma ~0.006-0.022) the sliders pegged visually at their max even though the underlying stored value was correct. Fixed: slider ranges to [0, 2.0]=/[0, 0.05]=, tooltips to describe annualised (not per-tick) magnitudes with the real seeded ranges, and a new curve's defaults from kappa=0.001,sigma=0.0005 to kappa=0.35,sigma=0.01.

Also flagged during the same run: published rates read as "a tad small" in the Curve Snapshot chart (raw decimal, e.g. 0.03022 rather than 3.022% or 302.2bp). The rates themselves are correct (EUR ESTR hovering ~2.98-3.04%, exactly theta≈3% plus a few bp of realistic vol) – this is a display-scale question, not a calibration bug, and is already the exact scope of the sibling IR rate display/quoting conventions task in this story, so it stays there rather than expanding this task's scope.

6. Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario, run through the QA Validation Runner panel) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
Verify day-scaled kappa/sigma calibration fix for IR curve feeds PASSED All 5 steps passed after the IrCurveEditor slider fix; AUD/AONIA curve snapshot showed plausible, moving rates (4.17-4.28%) across every tenor.

7. PRs

PR Title
#1661 [analytics.quant,synthetic,sql] Fix day-scaled kappa/sigma calibration across short-rate processes

8. Review

Three independent @claude review passes on #1661, all issue-level (no line comments): all three found no correctness/security/ maintainability issues and marked the PR ready to merge. Non-blocking observations only:

# Comment summary File Decision Notes
1 dt=1.0/365.0 duplicated as two separately-named constants (ir_curve_feed.cpp's dt, ir_curve_preview_handler.hpp's preview_dt) ir_curve_feed.cpp, ir_curve_preview_handler.hpp Accepted Consolidated into one ir_curve_feed_dt constant in ir_curve_template_resolver.hpp, next to the "1 tick == 1 calendar day" convention it implements; both call sites now reference it.
2 dt isn't validated by the shared process_parameter_validation layer (Qt client's validation surface), only by each process constructor's own throw process_parameter_validation.hpp/.cpp Declined dt is never user-facing input (fixed 1/365 convention set in code, not an editable config field), so there is nothing for the shared UI-facing validator to validate; the constructor throw is the only meaningful validation site.
3 preview_shape() passes a literal 42 seed rather than deriving one from req, unlike simulate_paths() ir_curve_preview_handler.hpp Declined Not a regression from this PR – matches prior behaviour (the old 4-arg call relied on the constructor's implicit seed=42 default); out of scope for a dt-focused fix.

9. Result

Root-caused and fixed the day-scaling bug shared by every discrete- time short-rate process (Vasicek/Hull-White, CIR) plus OU: the actual bug was discount_factor()'s bond-time accumulation treating every tick as a full year regardless of what a tick represents (+1.0 per tick instead of +dt), not the sigma scaling first suspected — confirmed by cross-checking against QuantLib's own process sources, which always keep speed=/=dt as separate arguments.

Shipped: an explicit dt (year-fraction per tick) constructor parameter on all four processes, defaulting to 1.0 for full backward compatibility; process_factory and every ores.synthetic call site thread dt=1/365 through explicitly; the SQL populate scripts reverted to plain annualised parameters; 191 new dt-tagged Catch2 assertions across all four processes (flat-curve exactness, discretisation convergence, statistical variance scaling, degenerate- branch sanity); three knowledge docs updated with the QuantLib cross- check; and a real regression the fix's own manual verification surfaced — IrCurveEditor's κ/σ sliders/tooltips/defaults, still calibrated for the old day-scaled magnitudes — fixed in the same PR.

All acceptance criteria met. Manual end-to-end verification (PR #1661) PASSED: AUD/AONIA curve snapshot showed plausible, moving rates (4.17-4.28%) across every tenor 1M-10Y, confirming the original "~150-250% blown-up, frozen far point" symptom is gone. Full build green, rat test suite green, schema validation clean, three independent @claude review passes found no correctness issues.

Emacs 29.3 (Org mode 9.6.15)