Task: One feed ingest loop over a unified subject scheme
Table of Contents
This page documents a task in the Feed lifecycle harmonization: asset-class-agnostic feed control-plane story. It captures the goal, current status, acceptance, and any notes or results.
Goal
One ingest loop replaces feed_ingest_loop and curve_feed_ingest_loop over a unified subject scheme — the kind is part of the tick subject, one JetStream stream — with uniform republish and observation handling for every feed kind.
Status
| Field | Value |
|---|---|
| State | DONE |
| Parent story | Feed lifecycle harmonization: asset-class-agnostic feed control-plane |
| Now | Nothing. |
| Waiting on | Nothing. |
| Next | Nothing. |
| Last touched | 2026-08-11 |
Acceptance
- One ingest loop subscribes to the unified tick subjects; the curve subject family and the second JetStream stream are retired.
- The tick wire type carries the kind; persistence (market_observation rows) and republish behave uniformly (FX republish semantics applied to every kind, or the difference is an explicit per-kind flag on the interface).
- Existing observation data and consumers (curve_republish_resolver point-id lookups) keep working.
Plan
One subject family, one wire convention, one ingest loop.
- Wire scheme:
synthetic.v1.tick.<kind>.<source_name>— the kind is the factory kind string (fx_spot,ir_curve), so the factory vocabulary and the subject scheme are one and the same. The sandbox prefix (synthetic.v1.sandbox.tick.) and the republishedmarketdata.v1.tick.scheme are unchanged. New shared headerores.marketdata.api/domain/tick_subjects.hppholds the prefix, the kind tokens and the subject builder — the wire contract; both ores.synthetic (producers) and ores.marketdata (ingest, Qt clients) already depend on ores.marketdata.api. - Streams: the existing
synthetic_ticksstream (synthetic.v1.tick.>) already filters both kinds — no stream change needed. =synthetic_curve_ticksis retired: both services stop ensuring it and no producer publishes into it. ores.nats has no delete-stream API, so the stale stream on an already-running server lingers inert (no producers, no consumers) until the server store is reset. - Producers:
synthetic_producer_subject(fx_spot_feed.hpp) emitssynthetic.v1.tick.fx_spot.<token>for bound feeds; sandboxed feeds unchanged. ir_curve_feed emitssynthetic.v1.tick.ir_curve.<source>. Wire payloads are unchanged — the kind rides in the subject, so every existing decoder (Qt tick chart, fx_spot_subscription) keeps working. - One ingest loop:
curve_feed_ingest_loopis deleted;feed_ingest_loopbecomes the single loop with one wildcard subscriptionsynthetic.v1.tick.>and kind-token dispatch:fx_spot: identity from the binding cache (refresh() rebuilds it from feed_bindings on the existing feed_binding_changed_event trigger; the per-source subscribe/unsubscribe lifecycle becomes a cache lifecycle). An arriving tick whose source has no enabled binding is dropped with a one-time warn — the same outcome as today's structural unreachability (no subscription), now explicit. Persistence, CRM bridge and republish unchanged in substance.ir_curve: the self-describing persistence path is unchanged; republish is added (FX republish semantics applied to every kind): the raw payload goes tomarketdata.v1.tick.<tenant>.<series_type>.<metric>.<qualifier>lowercased, through the same ore-key subject conversion.- CRM stays FX-only: currency driver pairs are FX-shaped, guarded by the existing series-type check.
- Consumers: MarketSimulatorWindow's
synthetic_subject()emits the kind-prefixed subject (is_curve_familymaps to their_curvetoken); its tick-chart decode (extract_tick_value) is already kind-agnostic. Republished-stream consumers (fx_spot_subscription, Qt grid/chart windows) are untouched — same subjects, same payloads. - Docs: the subject references in
ir_curve_tick.hpp,feed_binding.hpp,market_observation.hpp,fx_spot_feed.hpp,ir_curve_feed.hpp, MarketSimulatorWindow and the sandbox test comment are updated to the unified scheme. - Verification: full build; no SQL change so no DB recreate;
compass build rat.
Notes
Test Scenarios
Manual QA scenarios (scaffolded via compass add test_scenario, run
through the QA Validation Runner panel) that verify this task. Link
new ones here as they're created; the scenario doc itself links back
via its "Verifies task" field.
| Scenario | State | Notes |
|---|---|---|
PRs
| PR | Title |
|---|---|
| #1979 | [marketdata,synthetic,qt] Unified feed ingest: kind in the tick subject, one loop, one stream |
Review
Round 1 (claude[bot], comment #5264564353):
| # | Comment summary | File | Decision | Notes |
|---|---|---|---|---|
| 1 | ir_curve republishes even when persist failed | feed_ingest_loop.cpp | Fixed | Republish gated on persist success for both kinds. |
| 2 | stats_ keyed by plain source_name, colliding across kinds | feed_ingest_loop.cpp | Fixed | Keyed by (kind, source_name). |
| 3 | refresh() no longer prunes stats_ | feed_ingest_loop.cpp | Fixed | Prunes fx_spot stats of sources that left the bindings. |
| 4 | fresh random_generator per tick | feed_ingest_loop.cpp | Fixed | Reused member generator. |
| 5 | missing algorithm include | feed_ingest_loop.cpp | Fixed | Include added. |
| 6 | duplicated persist logic, already drifted | feed_ingest_loop.cpp | Fixed | One persist_tick_observation helper; uniform asset-class derivation. |
| 7 | kind dispatch hardcoded; tokens duplicated without a tie | tick_subjects.hpp, feed_controller_tests.cpp | Fixed | Doc reworded; static_assert ties + ir_curve subject-shape test. |
| 8 | isIrCurveSourceName free-text lookup | MarketSimulatorWindow.cpp | Not changed | Pre-existing client-side resolution; the client has no kind on the wire. |
| 9 | class doc narrates edit history | feed_ingest_loop.hpp | Fixed | Reworded to state the current design. |
Result
Shipped: one ingest loop over a unified tick subject scheme.
- Wire contract: new shared header
ores.marketdata.api/domain/tick_subjects.hpp(prefixes, kind tokensfx_spot=/=ir_curve,synthetic_tick_subject()builder). Both producers emitsynthetic.v1.tick.<kind>.<source_name>;synthetic_producer_subject(fx_spot_feed.hpp) andir_curve_feednow share one subject derivation instead of two families. Sandbox prefix andmarketdata.v1.tick.republished scheme unchanged; payloads unchanged — every existing decoder keeps working. - One loop:
curve_feed_ingest_loopdeleted (files, cmake entries);feed_ingest_loopis the single loop with one wildcardsynthetic.v1.tick.>subscription and kind-token dispatch. The per-source subscribe/unsubscribe lifecycle became a binding cache (refresh() rebuilds it from enabled feed_bindings; the existing feed_binding_changed_event trigger still drives refresh). fx_spot ticks get identity from the cache; unbound ticks are dropped with a one-time warn (the old structural unreachability, now explicit). ir_curve persistence is the unchanged self-describing path, with republish added: the raw payload goes tomarketdata.v1.tick.<tenant>.<series_type>.<metric>.<qualifier>lowercased, through the same ore-key subject conversion as FX — uniform republish per the plan (FX semantics applied to every kind, no per-kind flag needed). CRM bridge stays FX-only, guarded by the existing series-type check. - Streams: both services stop ensuring
synthetic_curve_ticks(no delete-stream API — a stale stream on an already-running server lingers inert).synthetic_ticks'ssynthetic.v1.tick.>filter already covers both kinds. - Consumers: MarketSimulatorWindow's
synthetic_subject()now emits the kind-prefixed subject via the shared kind tokens; its tick-chart decode was already kind-agnostic. Republished-stream consumers (fx_spot_subscription, Qt grid/chart) untouched. - Docs: subject references updated in the modeling orgs
(
feed_binding,market_observation,ir_curve_generation_config) and the generated headers (ir_curve_tick,feed_binding,market_observation),curve_snapshot_protocol.hpp,curve_republish_service.hpp,ir_curve_feed.hpp/cpp,fx_spot_feed.hppand thesynthetic_producer_subjectsandbox test (whose bound-subject expectation now asserts thefx_spottoken).
Acceptance met:
- One ingest loop subscribes to the unified tick subjects; the curve
subject family and the second JetStream stream are retired — the
loop's single wildcard covers both kinds;
synthetic_curve_ticksis no longer ensured or produced into;curve_feed_ingest_loopis gone. - The tick wire type carries the kind (the factory kind string rides in the subject); persistence and republish are uniform — FX republish semantics applied to every kind with no per-kind flag.
- Existing observation data and consumers keep working — payloads and
the republished
marketdata.v1.tick.scheme are unchanged, so curve_republish_resolver point-id lookups and the Qt tick chart work as before.
Verification: full build green (exit 0, 100%); no SQL change, so no DB
recreate; compass build rat green (exit 0) after the documented iam
CD-residue cleanup (memory ABF849AE).