ores.marketdata.observation_lineage

1. Flags
2. Columns
3. SQL
- 3.1. Flags
- 3.2. Checks
4. C++
5. See also

Per-observation provenance for a bare market_observation row whose source is not the market's own quote: which derivation config/version produced it, and which upstream source series/as-of it read, or – for a point an operator keyed by hand – which kind of point it is, who keyed it and why. Written never for the common quoted case, and never as a column on market_observations_tbl itself (a TimescaleDB hypertable explicitly documented as carrying no audit columns because tick-level volumes make that impractical).

point_source_kind names the two kinds that carry an annex row: derived for a point a derivation created, and manual for one keyed by hand. Quoted is the absence of a row, so a quoted tick keeps writing one row to the hot hypertable and nothing else; the invariant this rests on is that a point a derivation created always has an annex row.

Deliberately generic, not curve-specific: this is what lets CRM's own derived-cross publishing (currently pull-only – see the CRM architecture decision to never broadcast the full derived set as ticks) reuse this table unchanged whenever it starts persisting, stamping derivation_kind to 'CRM_DERIVATION' exactly as the IR curve bootstrapper stamps it to 'IR_CURVE_BOOTSTRAP'.

Standard bitemporal entity (GIST exclusion, soft-update/delete triggers), not a hypertable: unlike market_observations, lineage rows are written only for the minority of observations that are derived, so tick-level volume concerns that justify market_observations's own hypertable/no-audit-columns treatment do not apply here. A rerun of a derivation over the same tenor/point natural key closes the prior generation's lineage row and inserts a new one – the same soft-update convention every other bitemporal entity in this codebase already gets, mirroring (not literally reusing) how market_observations itself handles a rerun.

1. Flags

2. Columns

2.1. id

Surrogate UUID uniquely identifying this lineage row.

2.2. party_id

Party that owns the observation this lineage row describes. Included in the natural key because market_observations_tbl itself carries party_id as part of its own current-row uniqueness index – omitting it here would leave this table unable to uniquely identify the observation it describes whenever a series isn't 1:1 with a single party.

Set server-side from the authenticated session. Enforced by RLS.

ctx.generate_uuid()

2.3. series_id

Reference to ores_marketdata_market_series_tbl(id) – the derived series this observation belongs to.

2.4. observation_datetime

The market_observation row's own observation_datetime this lineage row describes.

ctx.past_timepoint()

2.5. oresmd_uri

The datum's canonical oresmd URI, matching the market_observation row's own oresmd_uri. Both are not null; the empty string is this column's default, and the only writer in the tree sets a real pillar.

std::string("")

2.6. point_source_kind

Which kind of point this annex row describes: derived for a point a derivation created, manual for one keyed by hand. The two values are validated by a check constraint. A point with no annex row is quoted, which is the absence of a row rather than a third stored value.

std::string("derived")

2.7. derivation_config_id

The derivation config (soft reference; table depends on the owning market_series.derivation_kind) that produced this observation. Null for a manual point, which was not produced by any derivation.

ctx.generate_uuid()

2.8. derivation_config_version

The derivation config's version this observation was produced under.

faker::number::integer(1, 10)

2.9. source_as_of

The upstream source data's as-of timestamp this derivation read (e.g. the raw instrument grid's as-of for a curve bootstrap, or the oldest contributing driver tick's timestamp for a CRM triangulation). Null for a manual point, which read no upstream source.

ctx.past_timepoint()

2.10. source_series_ids

Serialised JSON array of the upstream market_series ids this observation was derived from (e.g. the raw series for a Funding curve bootstrap; the raw series plus the discount curve's own output series for a Projection curve bootstrap; the walked driver series for a CRM triangulation). Empty ([]) for a manual point, which read no upstream series; the check admits the empty array while still refusing a value that is not a JSON array.

std::string(R"(["00000000-0000-0000-0000-000000000001"])")

3. SQL

3.1. Flags

The top-level * Flags drawer's natural_keys_composite_name: party_series_obs_point overrides the default auto-concatenated natural-key index name. It was added when that default spelled the coordinate point_id and came to 62 characters before the observation_lineages_=/=_uniq_idx prefix and suffix, over PostgreSQL's 63-byte NAMEDATALEN limit. The column is now oresmd_uri and the default would fit, but the override stays so the index name does not change under a database that already carries it.

3.2. Checks

expression
jsonb_array_length("source_series_ids") >= 0
"point_source_kind" in ('derived', 'manual')

4. C++

4.1. Flags

4.2. Repository

4.2.1. read_latest_by_observation

Reads the current lineage row for one observation, identified by series_id, observation_datetime, and oresmd_uri – the "which config/version/source produced this" lookup the shared derivation-lineage design calls for, in one round trip.

std::optional<domain::observation_lineage> read_latest_by_observation(context ctx,
    const boost::uuids::uuid& series_id,
    std::chrono::system_clock::time_point observation_datetime,
    const std::string& oresmd_uri);

#include "ores.platform/time/datetime.hpp"
#include <boost/uuid/uuid_io.hpp>
#include <optional>

std::optional<domain::observation_lineage>
observation_lineage_repository::read_latest_by_observation(context ctx,
    const boost::uuids::uuid& series_id,
    std::chrono::system_clock::time_point observation_datetime,
    const std::string& oresmd_uri) {
    using ores::platform::time::datetime;
    static constexpr auto max = db_timestamp::max();
    const auto tid = ctx.tenant_id().to_string();
    const auto sid = boost::uuids::to_string(series_id);
    const auto odt_str = datetime::to_iso8601_utc(observation_datetime);

    const auto query = query_builder::read<std::vector<observation_lineage_entity>> |
                       query_builder::where("tenant_id"_c == tid && "series_id"_c == sid &&
                             "observation_datetime"_c == odt_str &&
                             "oresmd_uri"_c == oresmd_uri &&
                             "valid_to"_c == max) |
                       query_builder::order_by("id"_c);
    const auto results = query_builder::execute_read_query<observation_lineage_entity, domain::observation_lineage>(
        ctx, query,
        [](const auto& entities) { return observation_lineage_mapper::map(entities); },
        lg(), "Reading latest observation lineage by observation");
    if (results.empty())
        return std::nullopt;
    return results.front();
}

4.2.2. read_latest_for_points

Reads the current lineage rows of the points a caller states, and nothing else. A point with no row is quoted, so the answer is shorter than the request when any is. The read is bounded by the stated points, not by the series, which is what lets a snapshot ask for the provenance of the points it placed.

/**
 * @brief The natural key of a market observation within one series, which is the
 * point an annex row describes.
 */
struct point_key {
    std::chrono::system_clock::time_point observation_datetime;
    std::string oresmd_uri;
};

/**
 * @brief The current annex rows of the stated points, one per point that has one.
 *
 * A point with no row is quoted, so the answer is shorter than @p points when any
 * is. The query takes the stated URIs and the span of the stated instants, which
 * is a cross-product, and the rows are then narrowed to the exact points, so a
 * wide span can read more rows than it returns. The rows are the current ones,
 * the same generation the as-of observation read uses for its values.
 */
std::vector<domain::observation_lineage> read_latest_for_points(context ctx,
    const boost::uuids::uuid& series_id,
    const std::vector<point_key>& points);

#include <algorithm>
#include <set>
#include <string>
#include <utility>
#include <vector>

std::vector<domain::observation_lineage>
observation_lineage_repository::read_latest_for_points(context ctx,
    const boost::uuids::uuid& series_id,
    const std::vector<point_key>& points) {
    using ores::platform::time::datetime;
    if (points.empty())
        return {};
    static constexpr auto max = db_timestamp::max();
    const auto tid = ctx.tenant_id().to_string();
    const auto sid = boost::uuids::to_string(series_id);

    // The query is a per-column cross-product of the stated URIs and the stated
    // time span, so the rows it returns are narrowed to the exact points below.
    std::vector<std::string> uris;
    uris.reserve(points.size());
    auto first = points.front().observation_datetime;
    auto last = first;
    for (const auto& point : points) {
        uris.push_back(point.oresmd_uri);
        first = std::min(first, point.observation_datetime);
        last = std::max(last, point.observation_datetime);
    }
    std::ranges::sort(uris);
    uris.erase(std::unique(uris.begin(), uris.end()), uris.end());

    const auto query = query_builder::read<std::vector<observation_lineage_entity>> |
                       query_builder::where("tenant_id"_c == tid && "series_id"_c == sid &&
                             "oresmd_uri"_c.in(uris) &&
                             "observation_datetime"_c >= datetime::to_iso8601_utc(first) &&
                             "observation_datetime"_c <= datetime::to_iso8601_utc(last) &&
                             "valid_to"_c == max) |
                       query_builder::order_by("id"_c);
    const auto rows = query_builder::execute_read_query<observation_lineage_entity, domain::observation_lineage>(
        ctx, query,
        [](const auto& entities) { return observation_lineage_mapper::map(entities); },
        lg(), "Reading latest observation lineages for points");

    std::set<std::pair<std::string, std::chrono::system_clock::time_point>> stated;
    for (const auto& point : points)
        stated.emplace(point.oresmd_uri, point.observation_datetime);

    std::vector<domain::observation_lineage> result;
    for (const auto& row : rows)
        if (stated.contains({row.oresmd_uri, row.observation_datetime}))
            result.push_back(row);
    return result;
}

4.2.3. read_manual_for_series

Reads the current annex rows of one series that name a manual point. The annex is cold and holds only the derived and manual points, and the manual ones are the few an operator keyed by hand, so the read is small. A write path asks it which coordinates an operator owns before it writes an automatic value.

/**
 * @brief The current annex rows of the series that name a manual point.
 */
std::vector<domain::observation_lineage> read_manual_for_series(context ctx,
    const boost::uuids::uuid& series_id);

std::vector<domain::observation_lineage>
observation_lineage_repository::read_manual_for_series(context ctx,
    const boost::uuids::uuid& series_id) {
    static constexpr auto max = db_timestamp::max();
    const auto tid = ctx.tenant_id().to_string();
    const auto sid = boost::uuids::to_string(series_id);

    const auto query = query_builder::read<std::vector<observation_lineage_entity>> |
                       query_builder::where("tenant_id"_c == tid && "series_id"_c == sid &&
                             "point_source_kind"_c == std::string("manual") &&
                             "valid_to"_c == max) |
                       query_builder::order_by("id"_c);
    return query_builder::execute_read_query<observation_lineage_entity, domain::observation_lineage>(
        ctx, query,
        [](const auto& entities) { return observation_lineage_mapper::map(entities); },
        lg(), "Reading the manual annex rows of a series");
}

4.3. Domain includes

#include <boost/uuid/uuid.hpp>
#include <string>

4.4. Entity includes

#include <string>

4.5. Conventions

5. See also

Market Series – carries the catalog-level derivation_kind marker this table's rows detail.
Market Observation – the table each lineage row describes one generation of.