Market Data Architecture

The decoupling layer is the typed NATS contract published by ores.marketdata.api: the tick payload schema, the observation/series schemas, and the subject conventions. A producer "implements a feed" purely by publishing conformant ticks; it shares only this API package. Depending on ores.marketdata.api is depending on the interface. Forbidden: marketdata consuming a producer as a library, marketdata calling a producer per tick, or a producer writing marketdata's tables directly.

A producer is a source, not "the" market data. It:

• owns its own "how to produce" configuration and CRUD/UI for it;
• generates autonomously (its config carries enabled=/=auto);
• publishes ticks to its own namespace: <producer>.v1.tick.<ore-key-dots> (e.g. synthetic.v1.tick.fx.rate.eur.usd);
• never claims the official subject and never writes the marketdata store.

New producers slot in by (a) publishing conformant ticks on their namespace and (b) having a feed config point at them — no consumer or marketdata code change.

Owned by ores.marketdata. For each official series it records the active source binding: the producer + source channel, a reference to the producer's config (e.g. synthetic config abc), and enabled=/=auto. Switching a series from synthetic to a vendor is a feed-config edit — point the binding at bloomberg.v1.tick.… instead of synthetic.v1.tick.…. This is the source-agnostic surface the control panel manages.

On start (and on feed-config / control changes) the marketdata service:

1. reads its feed registry;
2. subscribes to each bound source channel (<producer>.v1.tick.<key>);
3. on each incoming tick: persists the observation into its store (sole writer), stamped with the tenant and the source name (provenance);
4. remaps — re-publishes the tick onto the official stream marketdata.v1.tick.<tenant>.<key>.

There is no per-tick request/reply and no orchestration of producers: pure pub/sub. Persistence policy (dedup, downsample, batch) lives here, uniformly across all producers.

A feed config binds an official series to one active source, but more than one source can legitimately be available for the same pair (two synthetic producers, or a vendor plus synthetic). Where the authority must reconcile several sources into the one official spot, that combination is spot rate blending — a weighted/consensus average, robust median, or governed waterfall — and is a responsibility of ores.marketdata, not the producers. Blending is situational, not always-on: FX spot is an observable Level 1 input, so the authority mostly passes a single source through and blends only at the margin (stale, illiquid, or disputed feeds, or where governance requires defensibility against more than one source). Downstream of blending sits a lighter-weight spot verification (tolerance check vs reference sources) and, for the EOD/IPV cuts, an approval step — all owned by the authority and its governance, not the producer. This is deliberately distinct from the no-arbitrage derivation in the CRM below: blending picks/combines observed rates for a pair; the CRM computes derived crosses from drivers.

See Spot rate blending and its cluster: blending methods, why spot is observable, and spot rate governance (verification, approval, local vs global).

The "remap" step is more than a passthrough for FX. Producers feed driver rates (the most-liquid legs); the marketdata authority runs the cross-rates matrix (CRM) — a no-arbitrage spanning tree — to compute the derived rates by triangulation and enforce consistency, then publishes the full official set (remapped drivers + computed crosses). Correlations and derived volatility surfaces are owned and computed here too. So producers stay simple (drivers); ores.marketdata owns derivation, consistency and correlation. See Cross-rates matrix (CRM) for the full treatment.

Charts and other consumers subscribe only to marketdata.v1.tick.<tenant>.<key> and read history from the observation store. They are source-independent: they cannot tell (and must not care) whether the data originated synthetic or vendor.

Two distinct, separately-owned configs:

• Synthetic generation config (ores.synthetic): the recipe — how to generate. A named container with typed sub-configs per market-data type (FX spot GMM now; vol surface, IR later). Modelled per Polymorphic types over NATS — one typed struct + subject per concrete sub-config type, with a discriminator on the container; never an untyped JSON blob.
• Feed config (ores.marketdata): the binding — which source feeds which official series. References a producer + its config + enabled/auto.

Built on the ORE key (see Market Data Identifier: <series_type>/<metric>/<qualifier>[/<point_id>] → lowercase, '/'→'.'):

• Producer (raw source) stream: <producer>.v1.tick.<ore-key-dots>.
• Official (authoritative) stream: marketdata.v1.tick.<tenant>.<ore-key-dots> (tenant-scoped, because storage and entitlement are tenant-scoped).
• Read/store APIs: marketdata.v1.series.*, marketdata.v1.observations.*.

For marketdata to persist + remap unambiguously, a producer tick must carry enough identity to resolve the official series and tenant — the feed/config id (or tenant + ore_key) plus the source name. Marketdata resolves series_id + tenant from the feed binding, writes the observation with source = <source name>, and stamps the official tick with the same source. (Today's fx_spot_tick carries only ore_key/datetime/mid; it needs the feed identity + source added.) Provenance is therefore via the source name — market data stays tenant-scoped; the source name ties each observation to the producing config (see Multi-Tenancy Architecture for tenant/party scoping).

Two independent levers:

• A producer decides whether it is generating (its config's enabled=/=auto); it self-manages its loop.
• A feed config decides whether marketdata consumes + officialises a source (its enabled=/=auto).

Start/stop is a generic, source-agnostic control contract the owning party subscribes to; the control panel speaks the registry + control contract and needs no producer-specific knowledge.

Importing an ORE market.txt vintage (import_service, distinct from the live subscribe/persist/remap path above — a batch operation, not a tick consumer) checks each FX/RATE spot quote's currency order against ores.refdata's currency_pair reference data, and corrects a reversed key before persisting (see the task and the pure decision logic in fx_quote_convention_checker, ores.ore.core). The reversed-key case is real, not hypothetical: some vendored ORE example market.txt files store GBP/USD under the key FX/RATE/USD/GBP.

currency_pair is tenant- and party-scoped — each tenant can define its own currency-pair conventions — so this lookup cannot run as the marketdata service's own identity (which has no tenant) and must instead borrow the importing user's tenant/party: the On-Behalf-Of half of Service-to-Service Auth: Client Credentials vs. On-Behalf-Of Delegation, not a one-off. import_handler extracts the caller's bearer token from the inbound import_market_data_request message and delegates it on the service's own nats_client before calling refdata.v1.currency_pairs.list, exactly as feed_controller already does for market-data lookups in the vintage guard. Best-effort: if the fetch fails (refdata unreachable, etc.), the import proceeds with no reversed-key correction rather than failing.

Today (PoC): ores.synthetic publishes straight to marketdata.v1.tick.* and calls marketdata.v1.observations.save per tick; the series + observations are written under the system tenant from a boot-time hardcoded bootstrap; there is no persisted config and the feed is started by a raw NATS request.

Target (this doc): synthetic publishes to synthetic.v1.tick.* only; persisted synthetic + feed configs drive generation under the correct tenant; marketdata subscribes → persists → remaps to the official tenant-scoped stream; consumers use the official stream. See FX spot synthetic data PoC: architecture for the vertical-slice that seeded this and the PoC story for the migration tasks.