Story: Cross-rates matrix (CRM)
Table of Contents
This page documents a story in Sprint 23. It captures the goal, current status, acceptance criteria, and the tasks that compose it.
Future work under the Market data generation epic. Domain model: Cross-rates matrix (CRM) and its linked notes.
Goal
Implement the FX cross-rates matrix in ores.marketdata so derived rates are
computed from drivers and the matrix is always arbitrage-free — the consistency
engine the architecture relies on.
Status
| Field | Value |
|---|---|
| State | DONE |
| Parent sprint | Sprint 23 |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-07-18 |
Acceptance
- Driver/derived spanning-tree topology with cycle detection (never set both sides).
- Derived rates computed by triangulation through the pivot; direct preferred.
- Risk recentering (star-shaped on the aggregation currency) supported.
- A management UI to view/edit drivers and inspect derived rates.
Tasks
| Task | State | Start | End | Description |
|---|---|---|---|---|
| Model the driver/derived spanning-tree topology | DONE | 2026-07-11 | 2026-07-12 | Graph structure over currency pairs for the CRM: driver/derived edges with cycle detection so a pair is never set on both sides simultaneously. |
| Implement triangulation/derivation | DONE | 2026-07-12 | 2026-07-12 | Runtime rate_engine: consumes a stream of driver_quote ticks (possibly cross-thread) and serves batched derived-rate reads via an atomically-swapped immutable snapshot, computing derived rates by triangulation through the pivot and propagating staleness. |
| Implement risk recentering | DONE | 2026-07-12 | 2026-07-12 | Re-root the CRM into a star-shaped matrix around a chosen aggregation currency without changing any derived rate's value: rebuild topology as a star, seed it by pulling current derived rates from the source engine for each currency. |
| Fix rate_engine concurrent update/rates crash (SIGSEGV/SIGABRT) | DONE | 2026-07-12 | 2026-07-12 | rate_engine's documented single-producer/multi-reader concurrency contract crashes reproducibly under load: SIGSEGV in immer node refcounting during update(), and SIGSEGV/SIGABRT in crm_topology::currency_id_for's unordered_map lookup during rates(), consistent with heap corruption. |
| Wire the CRM into ores.marketdata's ingest path | DONE | 2026-07-12 | 2026-07-12 | Add persisted CRM topology config (codegened entities), wire ores.marketdata.service to run one rate_engine per tenant/party fed by driver ticks, and expose derived rates via a request/reply NATS endpoint – drivers stay the only high-frequency published ticks, avoiding an O(n^2) tick fan-out. |
| Wire the CRM into ores.marketdata's ingest path (service wiring) | DONE | 2026-07-12 | 2026-07-13 | Wire ores.marketdata.service to run one rate_engine per tenant/party fed by driver ticks, and expose derived rates via a request/reply NATS endpoint – drivers stay the only high-frequency published ticks, avoiding an O(n^2) tick fan-out. Follow-up to the CRM config entities task. |
| CRM cross-rates matrix UI panel | DONE | 2026-07-13 | 2026-07-14 | A Qt panel showing the full cross-rates matrix for a party via marketdata.v1.crm.rates, with a manual reload button (no auto-refresh/push) – covers every currently-configured pair. |
| Support multiple concurrently-enabled named CRM configs per party | DONE | 2026-07-13 | 2026-07-13 | Run and query several concurrently-enabled crm_topology_config rows per (tenant, party) by name (e.g. majors, exotics), replacing the current first-enabled-wins stopgap. |
| Publish realistic named CRM synthetic bundles (majors, exotics) via DQ | DONE | 2026-07-13 | 2026-07-13 | Seed demo parties with two realistic, concurrently-enabled named CRMs via the DQ synthetic data pipeline so the UI panel can be exercised against real data. |
| Fix OSX build: std::atomic<std::shared_ptr<…>> not trivially copyable (crm_ingest_bridge) | DONE | macOS CI (Xcode 16.2 libc++) fails to build crm_ingest_bridge: std::atomic<std::shared_ptr<const engines_map>> hits libc++'s generic atomic<T> template, which static_asserts T is trivially copyable – shared_ptr isn't. Linux/clang (libstdc++) accepts this because it implements the dedicated atomic<shared_ptr> specialization from P0718; the macOS libc++ toolchain in CI apparently does not (or not for this map value type). Landed with PR #1530's crm_ingest_bridge and reproduced again on the multi-named-CRM engine work. | ||
| Investigate valgrind 'possibly lost' in immer's thread-local free-list pool (rate_engine) | DONE | 2026-07-15 | 2026-07-15 | Valgrind reports 11,544 bytes in 39 blocks 'possibly lost' in ores.analytics.quant.tests, allocated via immer::cpp_heap/free_list_heap/thread_local_free_list_storage during rate_engine::update()'s immer::vector_transient::set() (rbtree.hpp assoc_mut/ensure_mutable_tail), reached from rate_engine_tests.cpp's worker-thread update path. Determine whether this is a real leak in rate_engine/vertex_state's use of immer, or the well-known thread-local-allocator-pool false positive valgrind gives for pools still holding cached (not leaked) memory when a worker thread exits before the pool is reclaimed. |
| Codegen full-stack CRM topology entities (config, driver pair, derived pair) with refdata UI | DONE | 2026-07-13 | 2026-07-14 | Generate domain/repository/service/HTTP/shell/Qt layers for crm_topology_config, crm_driver_pair, and crm_enabled_derived_pair, wired into the reference-data menu, so the DQ-published CRM bundles are viewable and testable outside raw SQL ahead of the (still-blocked) cross-rates matrix UI panel. |
| Market Data > FX menu with named-CRM quick-open shortcuts | ABANDONED | 2026-07-14 | Superseded by a picker-on-open redesign of the matrix panel itself – see its Result. | |
| Move CRM matrix's numeric logic (inversion, deltas) out of the Qt client | DONE | 2026-07-15 | 2026-07-15 | The cross-rates matrix UI panel currently computes inverted rates and %-change deltas client-side; this logic belongs in ores.analytics.quant (self-contained, unit-tested), exposed via NATS protocol additions (current+previous price with delta, an inverted-quote flag), with FX spot convention fields taken as caller-supplied input, not a class dependency. |
| Export CRM cross-rates matrix as CSV and ORE XML | DONE | 2026-07-15 | 2026-07-15 | Add an export button to the CRM cross-rates matrix UI panel to save the currently displayed rates as CSV, and as an ORE-format market data export (matching the existing ExportOre convention used elsewhere in the app). |
| Add ores.refdata.client with a currency_pair_convention cache | DONE | 2026-07-15 | 2026-07-16 | New thin, DB-free NATS-only component housing a currency_pair_convention_cache built on the nats-event-cache codegen facet, so every UI surface (Qt now, shell/Wt later) shares one cache instead of each hand-rolling its own. |
| Add convention-aware CRM rate formatter and wire into Qt matrix | DONE | 2026-07-16 | 2026-07-16 | Stateless plain C++ formatter (ores.marketdata.client/presentation) mapping raw CRM rates + a resolved currency_pair_convention into display-ready strings, replacing CrmCrossRatesMatrixMdiWindow's hand-formatting. Depends on the currency_pair_convention_cache task for its convention lookups. |
| Add CRM support to ores.shell | DONE | 2026-07-16 | 2026-07-17 | Shell command(s) to view a party's CRM cross-rates matrix (or a single pair), reusing the resolved-rates NATS request and, once available, the convention-aware display formatter – so ores.shell isn't stuck re-deriving inversion/delta/formatting itself. |
| Rename CRM 'inverted' concept to 'reciprocal' across code, protocol, UI, and docs | DONE | 2026-07-17 | 2026-07-18 | Rename the CRM 1/rate backfill concept from 'inverted' to 'reciprocal' end-to-end – UI labels/tooltips, the inverted field on crm_rate_item/get_crm_rates_request, crm_client/crm_rate_display_service/crm_rate_formatter identifiers, Qt button/handler names, the shell –inverted flag, ores.analytics.quant's rate_inverter, and every doc/recipe/task that names it – to match FX market convention and distinguish it from derived/triangulated (cross-rate) computation, which is a different concept entirely. |
Planned tasks (not yet scaffolded as task docs), following the domain model's concept map:
- Model the driver/derived spanning-tree topology (CRM graph topology and spanning tree): the graph structure over currency pairs, cycle detection so a pair is never set on both sides (driver and derived simultaneously).
- Implement triangulation/derivation (Triangulation and cross rates, spot rate derivation mechanics): compute a derived rate from the spanning tree via its pivot currency, preferring a direct driver-to-driver quote over a multi-hop path when both exist.
- Implement risk recentering (CRM risk: recentering and artefacts): re-root the tree star-shaped on a chosen aggregation currency without changing the derived rates' actual values.
- Wire the CRM into the
ores.marketdataauthority's ingest path (see Market Data Architecture): producers publish driver rates only; the CRM computes and publishes the derived set on the official tenant-scoped stream. - Management UI: view/edit drivers, inspect derived rates and the current spanning-tree structure.
Decisions
- Auto-refresh is allowed for the CRM cross-rates matrix UI (and other
trading-related UIs), on a client-side polling timer – this does not
reopen the earlier "never broadcast the derived set" decision. That
decision (see
the service-wiring task's
"compute locally where possible, pull on demand otherwise" note) is
about the server never pushing/broadcasting derived ticks to every
subscriber – it says nothing about a client choosing to automate its
own on-demand pulls. A UI timer that calls
marketdata.v1.crm.ratesevery N seconds is exactly the same request/response the manual Reload button already makes, just automated – no new server-side fan-out, no NATS subscription. Traders actively watching a screen (CRM matrix, and likely other trading-related panels going forward) need it to update without a manual click; that's a real, common requirement this story initially under-scoped by conflating "no server broadcast" with "no client polling". Follow the existingServiceDashboardMdiWindowauto-refresh pattern (checkable toolbar action + configurable interval, off by default) rather than inventing a new one. - The CRM lives in a new, dependency-light component –
ores.analytics.quant– not inores.analytics.core(which is persistence/CRUD for pricing-engine entities, unrelated). It links onlyBoost::graph=/=Boost::boostandimmer; no database, service, messaging, or refdata coupling. Everything domain-specific (currency codes, spot days, short-term rates) is a caller-supplied parameter, not fetched by the library. - Two-phase interface: a rare, one-shot
topology_builder::build()that validates and returns an immutablecrm_topology, and a continuousrate_engine(later task) that consumes driver-rate ticks and serves derived-rate reads. - Ambiguous input (more than one path between two currencies) is always
rejected at build time via incremental union-find cycle detection
(
boost::disjoint_sets) – never silently resolved. This is a deliberate departure fromQuantLib::ExchangeRateManager, whose own docs admit undefined behaviour when multiple chains exist. - The runtime engine will be thread-safe without being multi-threaded
itself: no internal threads, concurrency safety via an
atomically-swapped immutable snapshot (
immer::atom), so updates and batched reads never lock against each other. - QuantLib is deliberately not introduced as a dependency for this story (not currently vendored anywhere in the repo); spot/calendar handling stays a caller-supplied parameter.
- Rates are represented as cumulative log-rate-from-the-pivot per vertex
(not per-pair), so any derived rate is one subtraction and one
update()only touches the subtree below the changed edge –crm_topologyexposesparent()to support this (anedge_to_parent()accessor was added alongside it but never used, and was removed in review). Staleness is carried as a per-vertex timestamp alongside the log-rate, recomputed in the same subtree walk rather than as a separate pass. The pivot's ownas_ofis seeded to the maximum time_point (never the limiting factor), not construction time – an early version wrongly used construction time, which floored every other vertex's freshness at that instant forever; caught by the recentering task's tests. rate_engine::update()is single-producer by design (concurrent writers are not supported/needed);rate()=/=rates()are safe from any number of concurrent reader threads. This is documented on the class, not enforced at compile time.- Risk recentering (
service::recenter) is a point-in-time snapshot transform, not a live/continuous one: it builds a fresh star-shapedcrm_topologyaround the aggregation currency (reusingtopology_builderunchanged – a star can never conflict), and seeds the new engine with each currency's current rate and originalas_ofpulled from the source engine, so recentering never fabricates freshness.rate_enginegained an explicit move constructor (loading the moved-fromimmer::atom's value into a fresh one) sincerecentermust return a new engine by value andimmer::atomitself disallows copy/move. - The intermittent SIGSEGV/SIGABRT reported against
rate_engine's concurrency test traced to a test bug, not a production one: Catch2'sREQUIRE=/=CHECKmacros are not thread-safe, and the test's reader threads were calling them directly. A standalone ThreadSanitizer build (compiled ad hoc withclang++ -fsanitize=thread, since TSAN isn't wired into any CMake preset) found zero races inrate_engine=/=crm_topology=/=immer::atomitself under the exact conditions that reliably crashed the real binary – that null result is what redirected the investigation to the vcpkg-prebuiltlibCatch2.a(not TSAN-instrumented) and its documented not-thread-safe assertions. No production code changed; only the test collects results viastd::atomic<int>and asserts on the main thread after joining. - Inversion (1/rate fallback) and %-change-vs-previous are served-side
concerns, not part of
rate_engineitself:rate_inverteris a stateless pure function over a batch of already-computedderived_rate=s, and =rate_delta_trackeris a separate, thin stateful decorator (mutex-guarded, keyed by the displayed (base, quote) pair) layered on top ofrate_enginerather than folded into it. This keepsrate_engine's own immer-snapshot-per-batch concurrency model untouched, and keeps "what did this specific (tenant, party, crm_name) scope last see" – a serving-time/session concept – decoupled from the market/topology engine itself. Onerate_delta_trackerinstance is owned per named engine incrm_ingest_bridge, reset on everyrefresh()the same way the engine's own accumulated state is. - Convention-aware display formatting (precision, base/quote canonical
ordering) is explicitly out of this story's numeric-logic-extraction
scope: it was never implemented anywhere (client or server) to begin
with, so there was nothing to extract. Filed as a separate follow-up
task for a plain C++ presentation overlay in
ores.marketdata.client, since it is net-new functionality with its own design questions (server- vs. client-side convention caching). - The FX pair convention cache this formatter needs lives in a new
ores.refdata.clientcomponent, not insideores.qtorores.marketdata.client–clientis a documented, catalogue-level specialist sub-component role (projects/modeling/component_architecture.org, "consumer-facing API without coupling consumers to core internals"), used the same wayparty_cacheusesores.iam.corefor its own single-consumer case. The general placement rule (producer never caches itself; single consumer gets the cache in its own component; multiple UI-layer consumers share a thin*.clientcomponent) is captured as its own knowledge doc, Where should an entity-mirror cache live?, for futurecached_bydecisions to reference rather than re-litigate. - Generalised the
nats-event-cachecodegen facet'scached_byflag to accept the existingcomponent.subcomponentdotted form (cached_by: refdata.client) instead of hardcoding a.coresubcomponent – no new model flag, purely derived, verified byte-identical regeneration of the one prior consumer (party_cache). Also fixed two facet-level bugs this new second consumer surfaced thatparty_cachehad only ever avoided by accident: a missingreflectors.hppinclude needed forrfl::jsonon anytenant_id=/=uuid-bearing entity, andload()silently swallowing failures instead of returning them (now[[nodiscard]] std::string, empty on success) so a UI consumer isn't stuck tailing logs to find out why nothing loaded. - The convention-aware CRM rate formatter
(
ores.marketdata.client/presentation/crm_rate_formatter) needs only a resolveddecimal_placesint, not the fullcurrency_pair_conventionorcurrency.base_precedence: a CRM cell's (base, quote) orientation is already fixed upstream byrate_inverter=/=crm_rate_view, andbase_precedenceonly matters when a convention is captured (deciding which direction becomes the storedpair_code), not when it's looked up for an already-oriented pair. "Base/quote reordering" is instead a direction-agnostic lookup concern, handled byCrmCrossRatesMatrixMdiWindowtrying the cell's pair-code in both directions againstcurrency_pair_convention_cachebefore falling back to the formatter's own fixed default precision.
Out of scope
- Correlation management and volatility-surface driving (Correlation management, Volatility surface driving) — these reuse the same driver/derived pattern but for a different asset class; separate future stories once the CRM mechanism itself is proven on FX spot.
- Spot rate governance workflow (local/global approval, Spot rate governance) — an operational process layered on top of the CRM, not part of building the CRM itself.