Lineage and provenance
Table of Contents
1. Summary
observation_lineage, declared in
ores.marketdata.api/domain/observation_lineage.hpp, carries per
observation provenance for a derived observation. A row exists only for a
derived observation, so the existence of a row is itself the derived
marker. The common OBSERVED case writes none.
2. Detail
2.1. What one row records
series_idandobservation_datetimepoint at themarket_observationrow this lineage describes.oresmd_urimatches that observation row's own URI.derivation_config_idandderivation_config_versionname the config that produced the observation.source_as_ofis the as of timestamp of the upstream source data this derivation read.source_series_idsis a serialised JSON array of the upstreammarket_seriesids the observation was derived from.
2.2. How source_as_of fixes a rerun
A rerun of a derivation over the same natural key closes the prior generation's lineage row and inserts a new one, which is the same soft update convention every other bitemporal entity in this codebase receives. Because each row carries the upstream as of timestamp it read, a reader can tell which upstream vintage each generation used instead of guessing from the current values.
2.3. Why it is not a hypertable
The lineage table is a standard bitemporal entity with GIST exclusion and
the soft update and delete triggers. It is not a hypertable, because
lineage rows are written only for the minority of observations that are
derived. The tick level volume argument that makes market_observations_tbl
a hypertable with no audit columns does not apply here, so this table does
carry the audit columns.
2.4. The axis this is not
derivation_kind separates a published observation from a computed one.
source_kind on the source of an observation says whether a live adapter, a
replay or a simulator produced the input. The two axes are independent, and a
simulated quote that a bootstrap turns into a curve appears under both. The
source kind note states the second axis.
2.5. The fact the code shape does not show
The table is deliberately generic and not curve specific. CRM's own derived
cross publishing can reuse it unchanged whenever it starts persisting,
stamping derivation_kind to CRM_DERIVATION exactly as the IR curve
bootstrapper stamps it to IR_CURVE_BOOTSTRAP.
3. See also
- Market Data Architecture — the structure note that orders this cluster, and where to read this page in it.
- Curve Point Provenance, the curve specific view of provenance, which this note must not restate.
- Series and observation, the tables a lineage row points at.