Task: Shared derivation lineage: market_series marker + observation lineage table

Table of Contents

This page documents a task in the IR curve bootstrapping + official curve republish story. It captures the goal, current status, acceptance, and any notes or results.

Goal

Give the system one shared way to answer, for any published series or observation, "was this observed or computed, and by what" — instead of each derivation mechanism (curve bootstrap, and eventually CRM) inventing its own ad hoc provenance. Two parts:

  • A catalog-level marker on market_series: derivation_kind (not null, sentinel 'OBSERVED', matching the curve_role'NONE'= precedent on instrument_codes), derivation_config_id (not null, ores_utility_nil_uuid_fn() sentinel when derivation_kind = 'OBSERVED'), derivation_config_version (not null, default 0). A new small ores_refdata_derivation_kinds_tbl reference table (same shape as curve_roles) seeds at least 'OBSERVED', 'IR_CURVE_BOOTSTRAP', 'CRM_DERIVATION' — extensible for future kinds without a schema migration.
  • A per-observation lineage side table, ores_marketdata_observation_lineage_tbl (tenant_id, party_id, series_id, observation_datetime, point_id [not null, ''' sentinel for scalar series — a stricter rule than market_observations, whose own point_id column is genuinely nullable and only reaches ''' via coalesce(point_id, '') inside index/trigger predicates, not as a column-level convention; this new table chooses not to carry that nullability forward], derivation_config_id, derivation_config_version, source_as_of, source_series_ids uuid[], valid_from, valid_to), written only alongside a derived observation — never a column on market_observations_tbl itself, which is a TimescaleDB hypertable explicitly documented as carrying no audit columns because tick-level volumes make that impractical. A row's existence is the "derived" marker; no nullable columns. party_id is included in the key because market_observations_tbl itself carries party_id as part of its own current-row uniqueness index — omitting it here would leave the lineage table unable to uniquely identify the observation it describes whenever a series isn't 1:1 with a single party. valid_from=/=valid_to mirror market_observations's own bitemporal treatment: a bootstrap or CRM derivation can rerun and produce a new generation at the same (series_id, observation_datetime, point_id) natural key (the soft-update trigger closes the prior generation's market_observations row rather than overwriting it), so the lineage row for each generation must be equally versioned — otherwise a rerun would either collide or silently overwrite the prior generation's provenance, defeating "which config produced this" for any generation but the latest.

Deliberately generic, not curve-specific: this is what lets a future CRM change (persisting derived crosses — currently pull-only, out of scope here) stamp the exact same table with derivation_kind = 'CRM_DERIVATION' and reuse derived_rate::as_of (already computed today, currently discarded once the pull-reply is sent) as source_as_of, rather than inventing a second lineage mechanism.

Status

Field Value
State DONE
Parent story IR curve bootstrapping + official curve republish
Now Nothing.
Waiting on Nothing.
Next Pick up ir-curve-bootstrap-config, the next task in this story.
Last touched 2026-08-04

Acceptance

  • market_series carries derivation_kind=/=derivation_config_id=/ =derivation_config_version, all not null; every existing series defaults to 'OBSERVED' + nil-uuid + 0 with no behaviour change to today's raw-tick/FX-spot publishing.
  • ores_refdata_derivation_kinds_tbl exists, seeded with 'OBSERVED', 'IR_CURVE_BOOTSTRAP', 'CRM_DERIVATION', validated the same way curve_roles is (soft FK + validate function).
  • ores_marketdata_observation_lineage_tbl exists, is never written on the raw-tick ingest hot path, and supports a query "for this tenant/party/series + observation_datetime (+ point_id), which config/version/source produced it" in one lookup — including when more than one generation exists at the same natural key (proven by a test that reruns a derivation over the same tenor/point and asserts both generations' lineage remain independently queryable).
  • No column added to market_observations_tbl itself.

Plan

(Implementation strategy. Written when work starts; key decisions are distilled into the parent story's * Decisions at close, but the plan itself stays — it is the historical record of what we did.)

Notes

Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario, run through the QA Validation Runner panel) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

PRs

PR Title
#1835 [marketdata,refdata,codegen] Shared derivation lineage: schema + entities
#1822 [agile] IR curve bootstrapping: lineage + bootstrap-config design

Review

# Comment summary File Decision Notes
1 discount_curve_config_id self-reference has no cycle/chain protection (raised independently by all 4 review passes) task_ir-curve-bootstrap-config.org Accepted Added strict two-tier check constraint (referenced config must itself be FUNDING) plus explicit self-reference guard, per Multi-Curve Construction's own stated no-chaining model
2 observation_lineage key omits party_id and has no tie to a specific market_observations generation task_shared-derivation-lineage.org Accepted Added party_id to the key and valid_from/valid_to bitemporal columns mirroring market_observations' own soft-update convention, so reruns don't collide or overwrite prior generations' lineage
3 No task explicitly owns enforcing Funding-before-Projection ordering at run time task_ir-curve-bootstrapping-engine.org Accepted Added an explicit ownership note and acceptance criterion; mechanism (dependency check/queue/DAG walk) left as an implementation decision for that task's Plan
4 Pillar table's unique-active-row index not called out explicitly task_ir-curve-bootstrap-config.org Accepted Added explicit acceptance criterion mirroring ir_curve_template_entries' (tenant_id, party_id, config_id, sequence_index) index
5 point_id sentinel described as "matching" market_observations' convention, but that column is nullable there task_shared-derivation-lineage.org Accepted Reworded to describe it as a stricter rule than market_observations, not a literal match
6 derivation_config_version not enforced non-zero for derived series (check constraint only required derivation_config_id) ores.marketdata.market_series.org Accepted Added derivation_config_version <> 0 to the non-OBSERVED branch of the check constraint
7 Generated unique-index identifier for observation_lineages exceeds PostgreSQL's 63-byte NAMEDATALEN limit ores.marketdata.observation_lineage.org Accepted Overrode natural_keys_composite_name to a short explicit name (party_series_obs_point) via the top-level Flags drawer
8 is_required_timestamp codegen fix was applied to the tangled .mustache files directly, not their literate .org tangle sources – next tangle run would silently drop it (confirmed: CI's drift check failed) ores.cpp.repository.entity_header.org, ores.cpp.repository.mapper_impl.org Accepted Ported the same is_required_timestamp blocks into both .org sources; re-tangled and confirmed byte-identical output to the already-committed .mustache files
9 market_series.derivation_kind declared as FK-validated in the Validations table but the insert trigger never called the validation function (pre-existing gap also affecting asset_class, inherited by copying its structure) ores.marketdata.market_series.org Accepted Root cause: Validations table was nested under * SQL instead of the required top-level * Insert trigger heading the codegen parser looks for; moved it, which wires validation for both asset_class and derivation_kind

Result

Shipped in two PRs: #1822 (design, docs-only, merged) and #1835 (implementation, merged).

Schema

  • New ores.refdata.derivation_kind entity (simple-lookup profile), seeded OBSERVED=/=IR_CURVE_BOOTSTRAP=/=CRM_DERIVATION, validated the same way curve_roles is.
  • market_series gained derivation_kind=/=derivation_config_id=/ =derivation_config_version, all not null, paired by a check constraint (OBSERVED <-> nil-uuid/zero; any other kind requires all three of derivation_config_id <> nil, derivation_config_version <> 0, and a valid derivation_kind).
  • New ores.marketdata.observation_lineage entity: standard bitemporal (not a hypertable), keyed on tenant/party/series/observation_datetime/point_id, with a custom read_latest_by_observation repository method.
  • source_series_ids modelled as jsonb rather than a native uuid[] array (no existing precedent for array-typed columns in this codegen framework; jsonb already has precedent elsewhere and keeps this task schema-only, not a codegen-array-support project).

Codegen

Fixed a genuine, previously-latent template gap surfaced while building this: a required (NOT NULL) plain timestamp with time zone column (observation_lineage.source_as_of) had no working entity-layer representation – it fell through to a raw std::chrono::time_point field, which sqlgen cannot serialise, failing at compile time. Added the missing is_required_timestamp path (entity type, both mapper directions, and the ores.platform/time/datetime.hpp include gate) mirroring the existing natural-key std::string treatment, in both the tangled .mustache files and their literate .org tangle sources (missed first pass – caught by the drift CI check going red, see review round below).

Review

Two rounds, nine findings total across both PRs, all accepted and fixed – see the * Review table above for the full list. Notable ones: a real check-constraint gap (derived series could carry derivation_config_version = 0, a version that can never exist), a generated index name exceeding PostgreSQL's 63-byte identifier limit, the codegen fix landing only in tangled output and not its tangle source, and a pre-existing structural mistake in market_series.org (the Validations table nested under the wrong heading, silently skipping the insert trigger's validation call for both asset_class and the new derivation_kind) that got fixed for both columns while touching that section anyway.

Verified

  • Local build clean (linux-clang-debug-make) after every round.
  • compass db recreate + the SQL schema validator clean after every round (one pre-existing-pattern RLS warning shared with market_observations' own sibling gap, not introduced by this task).
  • ctest -R "marketdata|refdata": 8/8 test binaries passed both rounds (one unrelated, pre-existing environmental failure in ores.refdata.service.tests second round – config_parser_tests.cpp asserts a hardcoded NATS default but reads this environment's .env-configured value; not a file this task touched).
  • CI green on the final round: check, classify, claude-review, cmake-sources-drift, drift, pr-gate, refdata-codegen-drift, roundtrip, site/site.

Follow-ons not covered by this task

  • The acceptance criterion asking for a test proving "more than one generation exists at the same natural key, both independently queryable" was not written – read_latest_by_observation exists and is exercised by the build/tests, but no dedicated rerun test was added. Worth a task/note when official-curve-republish (the task that will actually rerun a derivation) lands, since that is where this behaviour first gets exercised for real.
  • The generated-identifier-length issue (finding 7) is a pre-existing codegen pattern risk beyond this one table – worth a capture to hash/shorten long generated identifiers in core.py generally, not just override this one table's name.

Emacs 29.3 (Org mode 9.6.15)