Story: IR curve bootstrapping + official curve republish

Table of Contents

This page documents a story in Sprint 25. It captures the goal, current status, acceptance criteria, and the tasks that compose it.

Goal

Real desks don't consume raw instrument quotes (deposit/FRA/swap) directly — they bootstrap them into an actual zero/discount curve (interpolation, day-count conventions, multi-curve OIS-discounting framework, per Ametrano & Bianchetti 2013) and that bootstrapped curve is what pricers/risk actually read from. Two pieces of work, previously untracked as concrete stories:

  1. A curve-bootstrapping engine over the raw instrument grid the IR curve family feed publishes (see IR Rates synthetic data generation) — interpolation, day-count conventions, multi-curve framework.
  2. An "official curve" republish/remap step, analogous to ores.marketdata's existing remap of a raw producer tick onto the official tenant-scoped stream (see Market Data Architecture) — so downstream consumers read a real usable curve, not raw ticks.

Filed while designing the IR Rates synthetic data generation story's tick-batch-publishing/curve-snapshot-builder-viewer tasks. Both of those are deliberately scoped to the raw instrument grid only (step 1 of a 3-step pipeline: raw ticks -> bootstrap -> republish) — curve-snapshot-builder-viewer is explicitly a viewer over raw ticks, not the bootstrapped curve a desk would price off of. Promoted from an inbox capture into a real Sprint 24 story now that Sprint 24's mission includes improving IR generation.

Design: derivation lineage, bootstrap config, build order

Working through what a persisted, bootstrapped curve object actually is (see Multicurve Management's "curve template" requirement) surfaced a gap this story must close rather than inherit: reusing the existing market_series=/=market_observations tables for bootstrapped output (the same pattern the CRM's computed crosses would eventually use, and the same shape ores.synthetic's raw-tick storage already uses) is architecturally correct — it is the established pattern here, applied a third time, not a new one — but doing so without a lineage mechanism would leave "was this point observed or computed, and by what" unanswerable except by guessing from the source string, which is exactly CRM's current soft spot (CRM's derived crosses are pull-only today — see crm_handler's "never broadcast the full derived set as ticks" — so this gap is latent there, not yet realised in storage; this story should not add a second, curve-specific version of the same weakness).

Three design decisions this story now commits to, detailed in the tasks below:

  • Shared, generic lineage — a derivation_kind marker on market_series (sentinel 'OBSERVED', matching the existing curve_role'NONE'= convention on instrument_codes rather than a nullable column) plus a per-observation observation_lineage side table, deliberately generic so CRM can reuse it unchanged whenever it starts persisting derived crosses — not this story's job to change CRM itself, but its job not to build something CRM would later have to duplicate.
  • Bootstrap config as refdata, not marketdata — recipe/config entities consistently live in ores.refdata in this codebase (crm_topology_config, curve_role, tenor); the bootstrap config follows that placement, reusing tenor and curve_role directly rather than inventing parallel concepts.
  • Build order as data — the Funding-before-Projection dependency (Multi-Curve Construction) is encoded via a self-referencing discount_curve_config_id on the bootstrap config (nil-uuid sentinel for Funding, required for Projection), so a Projection curve's bootstrap run has an explicit, queryable dependency rather than an implicit ordering assumption.

Every nullable field considered during design was replaced with either a sentinel paired to a discriminator column (curve_family_role FUNDING -> nil discount_curve_config_id; 'OBSERVED' -> nil derivation_config_id) or, where the null represented a genuinely avoidable gap (output_series_id before first publish), removed by minting the output series row at config-creation time instead.

Status

Field Value
State STARTED
Parent sprint Sprint 25
Now Not yet started.
Waiting on Nothing.
Next Break the story into tasks: bootstrapping engine first, then official-curve republish.
Last touched 2026-07-21

Acceptance

  • A curve-bootstrapping engine exists over the raw instrument grid published by the IR curve family feed: interpolation, day-count conventions, multi-curve (OIS-discounting) framework.
  • An "official curve" republish/remap step exists, analogous to ores.marketdata's existing raw-producer-tick remap onto the official tenant-scoped stream, so downstream consumers read a real usable curve rather than raw ticks.
  • A shared, generic derivation-lineage mechanism exists on market_series=/a new =observation_lineage table, such that any published series or observation can be queried for "was this observed or derived, and by what config/version/source" without guessing from the source string — usable unchanged by CRM's own derived-cross publishing whenever that lands (not this story's scope to change CRM itself).
  • The bootstrap recipe (source series, pillar list, interpolation method, day-count convention, curve family role, discount-curve dependency) is a first-class, inspectable ores.refdata entity, not implicit in code or ad hoc per bootstrap run.
  • The Funding-before-Projection build order is enforced/encoded as data (a config's discount-curve dependency), not left as an implicit ordering assumption in the engine.

Tasks

Task State Start End Description
FOMC-dated OIS short end with flat-forward interpolation ABANDONED 2026-08-09   Bootstrap the SOFR OIS short end off FOMC-meeting-dated pillars with a flat-forward (step) interpolation method, transitioning to a continuous method (log-linear/spline) at the long-end swap pillars.
Shared derivation lineage: market_series marker + observation lineage table DONE 2026-08-04 2026-08-04 A derivation_kind/derivation_config_id/derivation_config_version marker on market_series (sentinel 'OBSERVED', not nullable) plus a new observation_lineage side table for per-observation provenance – foundational for both the curve bootstrap output and, later, CRM's own derived-cross publishing.
IR curve bootstrap config: refdata entities for the bootstrap recipe DONE 2026-08-05 2026-08-05 New ores.refdata entities (bootstrap config + pillar children, mirroring ir_curve_template_entries' shape but source-independent) recording how a curve is bootstrapped: source series, curve family role (Funding/Projection), the discount_curve_config_id build-order link, interpolation method, day-count convention, and split tenor.
IR curve bootstrapping engine: inverse solve over the raw instrument grid DONE 2026-08-05 2026-08-05 Given a bootstrap config and the raw pillar rates already published by the IR curve family feed, solve tenor-by-tenor for discount factors that reprice each pillar – the inverse of curve_instrument_pricer's forward rate functions – applying the configured interpolation method, day-count convention, and Funding-before-Projection build order.
Official curve republish: write bootstrapped output as a lineage-stamped market_series DONE 2026-08-05 2026-08-06 Publish the bootstrapper's output curve into its pre-minted output_series_id via market_observations, stamping the observation_lineage table per observation – the republish/remap half of the story's goal, a transform rather than a passthrough.
Curve review/sign-off UI: approve a bootstrapped generation before it goes official BACKLOG     A human review gate between a curve bootstrap run and its publish to output_series_id – QC checks, diff vs prior generation, pillar-quote override, and an explicit approve action – so pricers only ever read an approved generation, matching how production trading systems gate bootstrapped curves before desks consume them.
Curve builder UI: bootstrap config CRUD + a guided Save/Bootstrap/Publish workbench DONE 2026-08-06 2026-08-08 Non-modal Curve Builder Workbench under the Market Data menu (Conventions/Pillars/Build & Diagnostics tabs) guiding a user through defining a bootstrap recipe, previewing its computed output, and explicitly publishing it – three distinct actions, not a standard save-and-close detail dialog – plus the backend compute/publish split and analytics.quant building blocks (forward_rate_calculator, curve_health_checker) its diagnostics tab needs.
Curated curve bootstrap templates for novice-friendly curve building BACKLOG     A first-class reference-data template entity (config + pillar rows, mirroring ir_curve_bootstrap_config/pillar's own shape) that admins curate, with a Start from Template… picker in CurveBuilderWorkbench.

Decisions

Curve Builder Workbench: three explicit actions, never fused

Save (persist recipe) / Bootstrap (compute preview) / Publish (make live) stay three separate, explicit UI actions calling three separate backend paths, not two or one – confirmed mid-build as a genuine domain distinction, not just a UI nicety. The backend split this required (curve_republish_service::compute() alongside republish()) is also the foundation the still-BACKLOG sign-off/review task needs, since it cannot gate "publish" behind an approval step while compute and publish stay fused.

Menu placement: Market Data, not Reference Data, by purpose not backend ownership

ir_curve_bootstrap_config's backend stays in ores.refdata (matching crm_topology_config, tenor, other recipe/config entities), but its Qt surface registers under Market Data's shared menu. General rule worth keeping for future entities: config/convention entities that are looked up and referenced stay Reference Data; entities whose primary purpose is producing or triggering the production of a market data series surface under Market Data regardless of backend component ownership.

Curated curve templates deferred, not skipped

A novice cannot build a curve from a blank Pillars tab with no guidance on which tenors/roles a real curve needs – identified mid-build as a genuinely separate, larger piece of work (a curated, admin-maintained template entity), not something to retrofit into this task's own scope. Shipped a cheap "New from Existing…" clone-a-config stopgap now; filed the proper version as its own follow-on task.

market_series identifier filtering is a known, deliberately deferred gap

Filtering market_series by currency/index (needed so the Workbench can constrain which raw series a curve bootstraps from) currently means substring-matching a free-text qualifier column – already caused one real bug (a floating_index_type code's dash-vs-slash convention silently diverging from market_series.qualifier's own). The oresmd migration in flight elsewhere does not resolve this by itself (it only standardises qualifier generation, still as a plain string, not a queryable typed field). Captured, not fixed here: strongly-typed market_series index/currency filter fields.

Out of scope

  • Modelling the raw instrument grid itself and its publishing pipeline — already delivered by IR Rates synthetic data generation (Sprint 23, DONE).
  • Changing CRM itself to persist derived crosses — CRM's derived rates are pull-only today (crm_handler's explicit "never broadcast the full derived set as ticks" architecture decision), so there is nothing to retrofit yet. This story's shared derivation-lineage table is deliberately generic so that whenever CRM persistence does land, it becomes a schema-only follow-on (stamp derivation_kind = 'CRM_DERIVATION', reuse derived_rate::as_of as source_as_of) — not a task this story needs to do.

References

  • ab-notebook: reproducible interest rate curve bootstrapping — Ballabio's replication of Ametrano & Bianchetti (2013), the reference case study for what a real bootstrap needs.
  • projects/ores.analytics.quant/include/ores.analytics.quant/service/curve_instrument_pricer.hpp — the pricer functions already deriving par rates from one short-rate draw's discount factors; a bootstrapper would consume the published ticks, not this pricer directly.
  • Bootstrapping a SOFR curve in ORE — worked example bootstrapping a USD SOFR OIS discounting/forecasting curve from overnight deposits and 1M-30Y OIS quotes, directly against the underlying ORE/QuantLib engine (ore.xml/todaysmarket.xml/curveconfig.xml/conventions.xml, the Python API for curve extraction and zero-rate output, and calibration reports verifying zero-PV repricing of the calibrated swaps) rather than a wrapper library — a concrete reference for piece 1's conventions and multi-curve framework, and for how ORE's own config/quote-string plumbing would map onto this bootstrapper.

See also

Emacs 29.3 (Org mode 9.6.15)