Task: One feed ingest loop over a unified subject scheme

Table of Contents

This page documents a task in the Feed lifecycle harmonization: asset-class-agnostic feed control-plane story. It captures the goal, current status, acceptance, and any notes or results.

Goal

One ingest loop replaces feed_ingest_loop and curve_feed_ingest_loop over a unified subject scheme — the kind is part of the tick subject, one JetStream stream — with uniform republish and observation handling for every feed kind.

Status

Field Value
State DONE
Parent story Feed lifecycle harmonization: asset-class-agnostic feed control-plane
Now Nothing.
Waiting on Nothing.
Next Nothing.
Last touched 2026-08-11

Acceptance

  • One ingest loop subscribes to the unified tick subjects; the curve subject family and the second JetStream stream are retired.
  • The tick wire type carries the kind; persistence (market_observation rows) and republish behave uniformly (FX republish semantics applied to every kind, or the difference is an explicit per-kind flag on the interface).
  • Existing observation data and consumers (curve_republish_resolver point-id lookups) keep working.

Plan

One subject family, one wire convention, one ingest loop.

  • Wire scheme: synthetic.v1.tick.<kind>.<source_name> — the kind is the factory kind string (fx_spot, ir_curve), so the factory vocabulary and the subject scheme are one and the same. The sandbox prefix (synthetic.v1.sandbox.tick.) and the republished marketdata.v1.tick. scheme are unchanged. New shared header ores.marketdata.api/domain/tick_subjects.hpp holds the prefix, the kind tokens and the subject builder — the wire contract; both ores.synthetic (producers) and ores.marketdata (ingest, Qt clients) already depend on ores.marketdata.api.
  • Streams: the existing synthetic_ticks stream (synthetic.v1.tick.>) already filters both kinds — no stream change needed. =synthetic_curve_ticks is retired: both services stop ensuring it and no producer publishes into it. ores.nats has no delete-stream API, so the stale stream on an already-running server lingers inert (no producers, no consumers) until the server store is reset.
  • Producers: synthetic_producer_subject (fx_spot_feed.hpp) emits synthetic.v1.tick.fx_spot.<token> for bound feeds; sandboxed feeds unchanged. ir_curve_feed emits synthetic.v1.tick.ir_curve.<source>. Wire payloads are unchanged — the kind rides in the subject, so every existing decoder (Qt tick chart, fx_spot_subscription) keeps working.
  • One ingest loop: curve_feed_ingest_loop is deleted; feed_ingest_loop becomes the single loop with one wildcard subscription synthetic.v1.tick.> and kind-token dispatch:
    • fx_spot: identity from the binding cache (refresh() rebuilds it from feed_bindings on the existing feed_binding_changed_event trigger; the per-source subscribe/unsubscribe lifecycle becomes a cache lifecycle). An arriving tick whose source has no enabled binding is dropped with a one-time warn — the same outcome as today's structural unreachability (no subscription), now explicit. Persistence, CRM bridge and republish unchanged in substance.
    • ir_curve: the self-describing persistence path is unchanged; republish is added (FX republish semantics applied to every kind): the raw payload goes to marketdata.v1.tick.<tenant>.<series_type>.<metric>.<qualifier> lowercased, through the same ore-key subject conversion.
    • CRM stays FX-only: currency driver pairs are FX-shaped, guarded by the existing series-type check.
  • Consumers: MarketSimulatorWindow's synthetic_subject() emits the kind-prefixed subject (is_curve_family maps to the ir_curve token); its tick-chart decode (extract_tick_value) is already kind-agnostic. Republished-stream consumers (fx_spot_subscription, Qt grid/chart windows) are untouched — same subjects, same payloads.
  • Docs: the subject references in ir_curve_tick.hpp, feed_binding.hpp, market_observation.hpp, fx_spot_feed.hpp, ir_curve_feed.hpp, MarketSimulatorWindow and the sandbox test comment are updated to the unified scheme.
  • Verification: full build; no SQL change so no DB recreate; compass build rat.

Notes

Test Scenarios

Manual QA scenarios (scaffolded via compass add test_scenario, run through the QA Validation Runner panel) that verify this task. Link new ones here as they're created; the scenario doc itself links back via its "Verifies task" field.

Scenario State Notes
     

PRs

PR Title
#1979 [marketdata,synthetic,qt] Unified feed ingest: kind in the tick subject, one loop, one stream

Review

Round 1 (claude[bot], comment #5264564353):

# Comment summary File Decision Notes
1 ir_curve republishes even when persist failed feed_ingest_loop.cpp Fixed Republish gated on persist success for both kinds.
2 stats_ keyed by plain source_name, colliding across kinds feed_ingest_loop.cpp Fixed Keyed by (kind, source_name).
3 refresh() no longer prunes stats_ feed_ingest_loop.cpp Fixed Prunes fx_spot stats of sources that left the bindings.
4 fresh random_generator per tick feed_ingest_loop.cpp Fixed Reused member generator.
5 missing algorithm include feed_ingest_loop.cpp Fixed Include added.
6 duplicated persist logic, already drifted feed_ingest_loop.cpp Fixed One persist_tick_observation helper; uniform asset-class derivation.
7 kind dispatch hardcoded; tokens duplicated without a tie tick_subjects.hpp, feed_controller_tests.cpp Fixed Doc reworded; static_assert ties + ir_curve subject-shape test.
8 isIrCurveSourceName free-text lookup MarketSimulatorWindow.cpp Not changed Pre-existing client-side resolution; the client has no kind on the wire.
9 class doc narrates edit history feed_ingest_loop.hpp Fixed Reworded to state the current design.

Result

Shipped: one ingest loop over a unified tick subject scheme.

  • Wire contract: new shared header ores.marketdata.api/domain/tick_subjects.hpp (prefixes, kind tokens fx_spot=/=ir_curve, synthetic_tick_subject() builder). Both producers emit synthetic.v1.tick.<kind>.<source_name>; synthetic_producer_subject (fx_spot_feed.hpp) and ir_curve_feed now share one subject derivation instead of two families. Sandbox prefix and marketdata.v1.tick. republished scheme unchanged; payloads unchanged — every existing decoder keeps working.
  • One loop: curve_feed_ingest_loop deleted (files, cmake entries); feed_ingest_loop is the single loop with one wildcard synthetic.v1.tick.> subscription and kind-token dispatch. The per-source subscribe/unsubscribe lifecycle became a binding cache (refresh() rebuilds it from enabled feed_bindings; the existing feed_binding_changed_event trigger still drives refresh). fx_spot ticks get identity from the cache; unbound ticks are dropped with a one-time warn (the old structural unreachability, now explicit). ir_curve persistence is the unchanged self-describing path, with republish added: the raw payload goes to marketdata.v1.tick.<tenant>.<series_type>.<metric>.<qualifier> lowercased, through the same ore-key subject conversion as FX — uniform republish per the plan (FX semantics applied to every kind, no per-kind flag needed). CRM bridge stays FX-only, guarded by the existing series-type check.
  • Streams: both services stop ensuring synthetic_curve_ticks (no delete-stream API — a stale stream on an already-running server lingers inert). synthetic_ticks's synthetic.v1.tick.> filter already covers both kinds.
  • Consumers: MarketSimulatorWindow's synthetic_subject() now emits the kind-prefixed subject via the shared kind tokens; its tick-chart decode was already kind-agnostic. Republished-stream consumers (fx_spot_subscription, Qt grid/chart) untouched.
  • Docs: subject references updated in the modeling orgs (feed_binding, market_observation, ir_curve_generation_config) and the generated headers (ir_curve_tick, feed_binding, market_observation), curve_snapshot_protocol.hpp, curve_republish_service.hpp, ir_curve_feed.hpp/cpp, fx_spot_feed.hpp and the synthetic_producer_subject sandbox test (whose bound-subject expectation now asserts the fx_spot token).

Acceptance met:

  • One ingest loop subscribes to the unified tick subjects; the curve subject family and the second JetStream stream are retired — the loop's single wildcard covers both kinds; synthetic_curve_ticks is no longer ensured or produced into; curve_feed_ingest_loop is gone.
  • The tick wire type carries the kind (the factory kind string rides in the subject); persistence and republish are uniform — FX republish semantics applied to every kind with no per-kind flag.
  • Existing observation data and consumers keep working — payloads and the republished marketdata.v1.tick. scheme are unchanged, so curve_republish_resolver point-id lookups and the Qt tick chart work as before.

Verification: full build green (exit 0, 100%); no SQL change, so no DB recreate; compass build rat green (exit 0) after the documented iam CD-residue cleanup (memory ABF849AE).

Emacs 29.3 (Org mode 9.6.15)