Lineage and provenance

Table of Contents

1. Summary

observation_lineage, declared in ores.marketdata.api/domain/observation_lineage.hpp, carries per observation provenance for a derived observation. A row exists only for a derived observation, so the existence of a row is itself the derived marker. The common OBSERVED case writes none.

2. Detail

2.1. What one row records

  • series_id and observation_datetime point at the market_observation row this lineage describes.
  • oresmd_uri matches that observation row's own URI.
  • derivation_config_id and derivation_config_version name the config that produced the observation.
  • source_as_of is the as of timestamp of the upstream source data this derivation read.
  • source_series_ids is a serialised JSON array of the upstream market_series ids the observation was derived from.

2.2. How source_as_of fixes a rerun

A rerun of a derivation over the same natural key closes the prior generation's lineage row and inserts a new one, which is the same soft update convention every other bitemporal entity in this codebase receives. Because each row carries the upstream as of timestamp it read, a reader can tell which upstream vintage each generation used instead of guessing from the current values.

2.3. Why it is not a hypertable

The lineage table is a standard bitemporal entity with GIST exclusion and the soft update and delete triggers. It is not a hypertable, because lineage rows are written only for the minority of observations that are derived. The tick level volume argument that makes market_observations_tbl a hypertable with no audit columns does not apply here, so this table does carry the audit columns.

2.4. The axis this is not

derivation_kind separates a published observation from a computed one. source_kind on the source of an observation says whether a live adapter, a replay or a simulator produced the input. The two axes are independent, and a simulated quote that a bootstrap turns into a curve appears under both. The source kind note states the second axis.

2.5. The fact the code shape does not show

The table is deliberately generic and not curve specific. CRM's own derived cross publishing can reuse it unchanged whenever it starts persisting, stamping derivation_kind to CRM_DERIVATION exactly as the IR curve bootstrapper stamps it to IR_CURVE_BOOTSTRAP.

3. See also

Emacs 29.3 (Org mode 9.6.15)