Task: Wire the CRM into ores.marketdata's ingest path (service wiring)
Table of Contents
This page documents a task in the Cross-rates matrix (CRM) story. It captures the goal, current status, acceptance, and any notes or results.
Goal
Wire the ores.analytics.quant CRM engine into ores.marketdata,
consuming the config entities from the prior task
(crm_topology_config, crm_driver_pair, crm_enabled_derived_pair),
following the Market Data Architecture doc's "drivers in, derived rates
out" section – without the naive design of republishing every derived
pair as an official tick on every driver movement (an O(n^2) fan-out for
no real benefit, since triangulation is cheap and deterministic and
anyone with the driver ticks + topology can derive the same value
locally).
Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Cross-rates matrix (CRM) |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-07-12 |
Acceptance
ores.marketdata.serviceholds oneservice::rate_engineper (tenant_id, party_id) – built from that party'scrm_topology_config+ enabledcrm_driver_pairrows, refreshed on config change the same wayfeed_ingest_loop::refresh()already reconciles feed-binding subscriptions.- Driver ticks keep flowing through
feed_ingest_loopexactly as today (persist + remap 1:1, unchanged hot path); additionally, if the tick's pair matches acrm_driver_pairedge for that tenant/party, it also feedsrate_engine::update()for that party's engine. No new ticks are published as a side effect of this – the engine is updated, nothing is broadcast. - Two NATS request/reply endpoints – both pull-only, no push/subscribe
path in this task, per the architecture decision to avoid the
tick-fan-out problem entirely rather than merely throttle it:
marketdata.v1.crm.rate: a single rate (direct or derived, indistinguishable from the caller's point of view – the caller never says which) by tenant, party, base and quote code; the service looks up that party'srate_engineand callsrate().marketdata.v1.crm.rates: all currently-configured pairs for a (tenant, party) in one call – the union of that party's enabledcrm_driver_pairedges andcrm_enabled_derived_pairrows, resolved via a singlerate_engine::rates()batch call (one atomic snapshot load, not N single-pair round trips) so a consumer wanting "give me the whole matrix" doesn't have to enumerate pairs itself or pay for N separate requests.
- Carried forward from the prior task's review (see its
* Review, items #1-#2) – fix before/alongside this task's own SQL touches it, since both matter once something actually reads "the" config for a party:crm_driver_pair=/=crm_enabled_derived_pair'sconfig_idFK trigger validatestenant_id=+=idagainstcrm_topology_configs_tblbut never checksparty_idmatches the parent config'sparty_id– a driver pair for party B could silently point at party A's config within the same tenant.- No constraint prevents more than one
enabled = truecrm_topology_configper (tenant, party), though the modeling doc documents "at most one active rate_engine per (tenant, party)" as the invariant –crm_ingest_bridge(below) needs this to actually hold to know which config to build from.
Plan
(Implementation strategy. Written when work starts; key decisions
are distilled into the parent story's * Decisions at close, but the
plan itself stays — it is the historical record of what we did.)
Service wiring (carried forward from the prior task's design)
A new class (working name crm_ingest_bridge, alongside
feed_ingest_loop in ores.marketdata.service/app) owns a
std::map<std::pair<tenant_id, party_id>, service::rate_engine>. On
refresh() (same trigger as feed_ingest_loop: NATS notify on config
table changes) it rebuilds any (tenant, party) whose crm_topology_config
or crm_driver_pair rows changed, via topology_builder::build +
constructing a fresh rate_engine – config changes are rare, so a full
rebuild (not an incremental patch) is the right cost/complexity trade-off,
consistent with topology_builder being a rare, one-shot operation by
design.
feed_ingest_loop gains a hook: after persisting + remapping a driver
tick as it already does, it also checks whether the tick's ore_key
resolves to a crm_driver_pair edge for that tick's tenant/party and, if
so, calls crm_ingest_bridge::update() (which just delegates to that
party's rate_engine::update()) – no change to what gets published, only
an additional local update.
Notes
Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario, run
through the QA Validation Runner panel) that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
PRs
| PR | Title |
|---|---|
| #1530 | [marketdata] Wire the CRM engine into the ingest path |
Review
| Comment summary | File | Decision | Notes |
|---|---|---|---|
Result
Wired ores.analytics.quant into ores.marketdata.service:
crm_ingest_bridge(service/app): onerate_engineper (tenant, party), keyed off an atomically-swappedshared_ptr<const map<...>>sorefresh()(rare) andupdate()=/=rate()=/=rates()(hot path, any number of threads) never race – refresh replaces the whole engine set wholesale rather than mutating existing engines, matchingtopology_builder's own "rare, one-shot" design. Built fromcrm_topology_config+ enabledcrm_driver_pairrows viatopology_builder::build; enabledcrm_enabled_derived_pairrows are folded into each engine's "configured pairs" list for the batch endpoint, without affecting the topology (they're never edges).feed_ingest_loopgained an optionalcrm_bridge_hook: after its existing persist+remap, an FX/RATE tick is also offered to the bridge as a candidate driver update – silently ignored if the (tenant, party) has no CRM configured or the pair isn't one of its driver edges, so a CRM lookup failure can never break the tick's normal path.- Two new NATS request/reply endpoints,
marketdata.v1.crm.rate(single pair) andmarketdata.v1.crm.rates(every currently-configured pair for a party in onerate_engine::rates()batch call – added per request during this task, since the original design only had the single-pair endpoint), served bycrm_handler. Both pull-only; no push/subscribe path, per the story's architecture decision. application.cpp: constructs the bridge, calls an initialrefresh(), subscribescrm_topology_config_changed_event=/ =crm_driver_pair_changed_eventto trigger re-refresh(), and registers the two new endpoints alongside the existing codegen ones.- Added
read_latest_all_tenantsto all three CRM entities (same pattern asfeed_binding's) so the bridge can enumerate every (tenant, party) at startup/refresh – codegen-generated, not hand-SQL. - Filed a second codegen-gap capture
(
codegen_generic_fk_crosscheck_and_partial_unique.org) for the two SQL fixes carried forward from the prior task's review (cross-party FK validation, single-enabled-config uniqueness) – neither is fixable without either duplicating the book/portfolio one-off hack or hand-writing SQL codegen doesn't produce anywhere else, so both stay gaps rather than patches.crm_ingest_bridge::refresh()carries an application-level safeguard instead: if more than onecrm_topology_configis enabled for a (tenant, party), the first wins and a warning is logged, rather than crashing or silently merging. - New
crm_protocol.hpp(request/response message types) inores.marketdata.api.
5 new integration test cases in ores.marketdata.service.tests
(app_crm_ingest_bridge_tests.cpp, using the DB-backed test fixture,
real repository writes): refresh builds a working engine end-to-end
(unavailable before ticking, fresh + correctly triangulated after);
rate() returns nullopt (not unavailable) when no CRM is configured
at all; update() on a non-edge pair doesn't throw; rates() returns
every configured pair (drivers + enabled derived) in one call; the
ambiguous-multiple-enabled-configs case doesn't crash.
Verified: full local build (all targets) clean; full test suite 71/71
suites, 0 failures; db recreate clean; validate_schemas.sh --strict
zero warnings.
Added a follow-up task, CRM cross-rates matrix UI panel, for a Qt panel
consuming marketdata.v1.crm.rates (manual reload only, no
auto-refresh/push), per a reference mockup.