A quote series is keyed by item, provider and party

Table of Contents

1. Summary

Two tables hold the stored result of the pipeline. A market_series catalogues a quote series: one internal item from one provider for one party. A market_observation records one value of that series that a person or a process states. The series key is (item, provider, party). It holds no source, so a party can change its source and keep one series. A live tick is not an observation. It lives in the shared tick table, on a stream that every party of the provider reads, and its own read returns it after checking the party's subscription.

2. Detail

2.1. What each one holds

market_series holds oresmd_uri, provider, party_id, series_subclass, and the asset classes in market_series_asset_classes. The key is unique. The table keeps no decomposed copy of the key, so a reader that wants a part of it reads it through the codec.

market_observation holds series_id, observation_datetime, oresmd_uri, value and source_id. The observation_datetime is the vendor time. The source_id records the source that produced the value and is information only. The URI is the row's only identity. The ORE key is not stored, because ore_key_codec writes it from the datum.

2.2. A tick is not an observation

The ingest writes each tick once, as a point of its source's stream, and writes no observation row. A party's series points at the stream. See A source tick is stored once and entitled when it is read for the store and The tick ingest pipeline: freshness first for the lanes that write it. The observation table holds the points that people and processes state: an ORE import, a hand-keyed point and a derived point. The list of observations returns observations and never a tick. A client that wants ticks calls the history read. See The history call returns every tick of a range in one reply, or refuses the range.

2.3. The tick store is shared

A tick is an observation about a source, not a row of a party. The store holds one row for each tick of a source, on a stream named by an id derived from the provider code and the series URI, and a party's series names the same stream. The row carries the stream, the source, the time, the coordinate and the value, and no tenant or party. So the store does not grow with the number of parties, and two parties of one provider read the same row.

The history read is therefore the only door to the ticks. It resolves the party's series, checks that the party holds a subscription to the provider with display or non-display, and refuses a party with none or with the derived right alone. See One shared tick or one copy for each party. The coordinate dictionary is shared for the same reason: every party of a stream must agree on which number means which datum.

2.4. Why the provider is in the key

Two providers quote the same item with different values. A party that subscribes to both holds two series, one for each, and a composite can then blend them. See A composite attributes its use to the providers of its inputs.

2.5. Why the source is not in the key

The source is a route, and routes change. A key that held it would split one price history in two on each switch. See A source switch changes one route and nothing else.

2.6. The soft update rule

market_observations_tbl is a TimescaleDB hypertable partitioned by observation_datetime with 30 day chunks. A correction replaces the previous value through the soft update trigger. GIST exclusion and DELETE rules do not work on hypertables, so a partial unique index and the soft update and soft delete trigger pair enforce uniqueness.

2.7. Why observations carry no audit columns

The observation type has no version, modified_by, performed_by, change_reason_code or change_commentary. Tick-level volume makes them impractical. valid_from and valid_to carry the transaction time.

2.8. Why valid_from is projected onto the wire

Codegen drops the audit tail for a :no_audit_columns: true entity. The database column exists and every as-of read selects it. read_as_of_records returns each point with the valid_from of its row. The curve_snapshot.get reply carries the record times as a parallel array, so a consumer can report how stale a term structure is. Two ages are readable. The read's instant minus a point's observation_datetime is the age of the market state. Now minus valid_from is how long ago the value arrived.

2.9. The derivation marker on the series

market_series.derivation_kind defaults to OBSERVED. A derived series carries a named mechanism such as IR_CURVE_BOOTSTRAP, with derivation_config_id and derivation_config_version. This is a catalogue marker, and it differs from the per-observation lineage row in Lineage and provenance.

3. See also

Emacs 29.3 (Org mode 9.6.15)