Task: Wire the CRM into ores.marketdata's ingest path (service wiring)

Table of Contents

This page documents a task in the Cross-rates matrix (CRM) story. It captures the goal, current status, acceptance, and any notes or results.

Goal

Wire the ores.analytics.quant CRM engine into ores.marketdata, consuming the config entities from the prior task (crm_topology_config, crm_driver_pair, crm_enabled_derived_pair), following the Market Data Architecture doc's "drivers in, derived rates out" section – without the naive design of republishing every derived pair as an official tick on every driver movement (an O(n^2) fan-out for no real benefit, since triangulation is cheap and deterministic and anyone with the driver ticks + topology can derive the same value locally).

Status

Field Value
State DONE
Parent story Cross-rates matrix (CRM)
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-07-12

Acceptance

  • ores.marketdata.service holds one service::rate_engine per (tenant_id, party_id) – built from that party's crm_topology_config + enabled crm_driver_pair rows, refreshed on config change the same way feed_ingest_loop::refresh() already reconciles feed-binding subscriptions.
  • Driver ticks keep flowing through feed_ingest_loop exactly as today (persist + remap 1:1, unchanged hot path); additionally, if the tick's pair matches a crm_driver_pair edge for that tenant/party, it also feeds rate_engine::update() for that party's engine. No new ticks are published as a side effect of this – the engine is updated, nothing is broadcast.
  • Two NATS request/reply endpoints – both pull-only, no push/subscribe path in this task, per the architecture decision to avoid the tick-fan-out problem entirely rather than merely throttle it:
    • marketdata.v1.crm.rate: a single rate (direct or derived, indistinguishable from the caller's point of view – the caller never says which) by tenant, party, base and quote code; the service looks up that party's rate_engine and calls rate().
    • marketdata.v1.crm.rates: all currently-configured pairs for a (tenant, party) in one call – the union of that party's enabled crm_driver_pair edges and crm_enabled_derived_pair rows, resolved via a single rate_engine::rates() batch call (one atomic snapshot load, not N single-pair round trips) so a consumer wanting "give me the whole matrix" doesn't have to enumerate pairs itself or pay for N separate requests.
  • Carried forward from the prior task's review (see its * Review, items #1-#2) – fix before/alongside this task's own SQL touches it, since both matter once something actually reads "the" config for a party:
    • crm_driver_pair=/=crm_enabled_derived_pair's config_id FK trigger validates tenant_id=+=id against crm_topology_configs_tbl but never checks party_id matches the parent config's party_id – a driver pair for party B could silently point at party A's config within the same tenant.
    • No constraint prevents more than one enabled = true crm_topology_config per (tenant, party), though the modeling doc documents "at most one active rate_engine per (tenant, party)" as the invariant – crm_ingest_bridge (below) needs this to actually hold to know which config to build from.

Plan

(Implementation strategy. Written when work starts; key decisions are distilled into the parent story's * Decisions at close, but the plan itself stays — it is the historical record of what we did.)

Service wiring (carried forward from the prior task's design)

A new class (working name crm_ingest_bridge, alongside feed_ingest_loop in ores.marketdata.service/app) owns a std::map<std::pair<tenant_id, party_id>, service::rate_engine>. On refresh() (same trigger as feed_ingest_loop: NATS notify on config table changes) it rebuilds any (tenant, party) whose crm_topology_config or crm_driver_pair rows changed, via topology_builder::build + constructing a fresh rate_engine – config changes are rare, so a full rebuild (not an incremental patch) is the right cost/complexity trade-off, consistent with topology_builder being a rare, one-shot operation by design.

feed_ingest_loop gains a hook: after persisting + remapping a driver tick as it already does, it also checks whether the tick's ore_key resolves to a crm_driver_pair edge for that tick's tenant/party and, if so, calls crm_ingest_bridge::update() (which just delegates to that party's rate_engine::update()) – no change to what gets published, only an additional local update.

Notes

Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario, run through the QA Validation Runner panel) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

PRs

PR Title
#1530 [marketdata] Wire the CRM engine into the ingest path

Review

Comment summary File Decision Notes
       

Result

Wired ores.analytics.quant into ores.marketdata.service:

  • crm_ingest_bridge (service/app): one rate_engine per (tenant, party), keyed off an atomically-swapped shared_ptr<const map<...>> so refresh() (rare) and update()=/=rate()=/=rates() (hot path, any number of threads) never race – refresh replaces the whole engine set wholesale rather than mutating existing engines, matching topology_builder's own "rare, one-shot" design. Built from crm_topology_config + enabled crm_driver_pair rows via topology_builder::build; enabled crm_enabled_derived_pair rows are folded into each engine's "configured pairs" list for the batch endpoint, without affecting the topology (they're never edges).
  • feed_ingest_loop gained an optional crm_bridge_ hook: after its existing persist+remap, an FX/RATE tick is also offered to the bridge as a candidate driver update – silently ignored if the (tenant, party) has no CRM configured or the pair isn't one of its driver edges, so a CRM lookup failure can never break the tick's normal path.
  • Two new NATS request/reply endpoints, marketdata.v1.crm.rate (single pair) and marketdata.v1.crm.rates (every currently-configured pair for a party in one rate_engine::rates() batch call – added per request during this task, since the original design only had the single-pair endpoint), served by crm_handler. Both pull-only; no push/subscribe path, per the story's architecture decision.
  • application.cpp: constructs the bridge, calls an initial refresh(), subscribes crm_topology_config_changed_event=/ =crm_driver_pair_changed_event to trigger re-refresh(), and registers the two new endpoints alongside the existing codegen ones.
  • Added read_latest_all_tenants to all three CRM entities (same pattern as feed_binding's) so the bridge can enumerate every (tenant, party) at startup/refresh – codegen-generated, not hand-SQL.
  • Filed a second codegen-gap capture (codegen_generic_fk_crosscheck_and_partial_unique.org) for the two SQL fixes carried forward from the prior task's review (cross-party FK validation, single-enabled-config uniqueness) – neither is fixable without either duplicating the book/portfolio one-off hack or hand-writing SQL codegen doesn't produce anywhere else, so both stay gaps rather than patches. crm_ingest_bridge::refresh() carries an application-level safeguard instead: if more than one crm_topology_config is enabled for a (tenant, party), the first wins and a warning is logged, rather than crashing or silently merging.
  • New crm_protocol.hpp (request/response message types) in ores.marketdata.api.

5 new integration test cases in ores.marketdata.service.tests (app_crm_ingest_bridge_tests.cpp, using the DB-backed test fixture, real repository writes): refresh builds a working engine end-to-end (unavailable before ticking, fresh + correctly triangulated after); rate() returns nullopt (not unavailable) when no CRM is configured at all; update() on a non-edge pair doesn't throw; rates() returns every configured pair (drivers + enabled derived) in one call; the ambiguous-multiple-enabled-configs case doesn't crash.

Verified: full local build (all targets) clean; full test suite 71/71 suites, 0 failures; db recreate clean; validate_schemas.sh --strict zero warnings.

Added a follow-up task, CRM cross-rates matrix UI panel, for a Qt panel consuming marketdata.v1.crm.rates (manual reload only, no auto-refresh/push), per a reference mockup.

Emacs 29.3 (Org mode 9.6.15)