Story: Feed lifecycle harmonization: asset-class-agnostic feed control-plane

Table of Contents

This page documents a story in Sprint 25. It captures the goal, current status, acceptance criteria, and the tasks that compose it.

Goal

The feed control-plane is asset-class-agnostic. One folder-scoped start/stop request walks the folder subtree once, server-side, and starts every feed under it regardless of asset class; one per-config start/stop/list is keyed by config_id and resolved server-side; one lifecycle controller, one auto-start, and one ingest loop serve every feed kind. ores.shell has feed verbs for every asset class (including the currently missing IR curve commands). The seam is one common feed interface plus a per-kind factory: a new asset class plugs in with a config table, a producer behind the common interface, and one factory registration — the lifecycle machinery is not touched again.

Status

Field Value
State STARTED
Parent sprint Sprint 25
Now Tasks decomposed (8); the seam task is the entry point.
Waiting on Nothing.
Next Start task: feed producer seam.
Last touched 2026-08-11

Acceptance

  • Folder-scoped start/stop walks the folder subtree once, server-side, and starts every feed under it — FX and IR alike. A folder containing both kinds starts both.
  • No client-side enumeration remains: the Qt MarketSimulatorWindow folder branch and per-config IR loop, and the tenant_handler two-path provisioning, are deleted; the clients send the folder/config request and the server does the rest.
  • Per-config start/stop/list works for every asset class, keyed by config_id, resolved server-side with permission checks (the IR curve pattern). The FX market_feed_config_handler client-supplied-params path is deleted.
  • ores.shell has feed verbs for every asset class: folder start/stop and per-config start/stop/list, including the IR curve commands that are missing today. No shell command enumerates asset classes.
  • One lifecycle controller, one auto-start with uniform startability (enabled + auto_start + enabled container; auto_start is added to the FX config for uniform semantics), and one ingest loop over a unified subject scheme (kind in the subject, one JetStream stream).
  • The seam is a factory: a new asset class is a config table, a producer implementing the common feed interface, and one factory registration. Lifecycle, control-plane, auto-start, ingest, shell, and client code do not change for a new asset class.

Analysis

The folder-cascade defect is a symptom of lifecycle machinery built per asset class. The original FX PoC architecture doc (fx_spot_synthetic_poc_architecture.org) specified one market_feed_config with a feed_type discriminator and a feed manager acquiring producers via a registered factory (open design question 8). The PoC resolution — synthetic service calls start/stop directly — was adopted as permanent, and every subsequent piece was re-implemented per kind instead of behind that seam.

The config tables are already ~80% common (id, tenant_id, party_id, config_id to the market_data_generation_config container, source_name, process_type, ticks_per_hour, price_source, vintage_source/date, enabled, folder_id, audit) plus per-kind children (gmm_component vs ir_curve_template_entry + parameter values) and per-kind series identity (ore_key vs qualifier+role). The duplication lives in the machinery:

  • feed_controller vs curve_feed_controller (identical running map, thread-per-feed, start/stop/list/join; IR adds qualifier+role conflict, FX adds binding auto-creation, sandbox subject, status loop).
  • auto_start_enabled_feeds vs auto_start_enabled_ir_curve_feeds (identical walk, different filter semantics — auto_start exists on IR only).
  • market_feed_config_handler vs ir_curve_feed_config_handler — opposite designs: FX is a stateless pass-through with no auth ("no JWT auth or DB context is required"); IR resolves everything server-side from config_id. The IR pattern is the correct one.
  • The folder cascade resolves fx_spot_generation_config rows only; the IR pass would have been a third copy of the build sequence.
  • feed_ingest_loop vs curve_feed_ingest_loop (different subjects, different streams, FX republishes and IR does not).
  • Vintage availability, permission gating, and the subject/stream scheme are likewise duplicated per kind.
  • The producer seam exists half-built: IFxSpotFeed is implemented by FX only; IR has no interface and no factory exists.

Target: one feed description at three levels — identity/lifecycle common data (per-kind tables, kept; optionally a UNION view for a single-query cascade), asset-class data per-kind children, producer math per-kind code behind the common interface. The factory is the clear setup the codebase never had: config table + producer + one registration.

Decisions

  • Code-first: config tables stay per-kind. The common feed description lives in the interface and factory, not in a shared table. A UNION view over the config tables is acceptable if a single-query cascade is wanted later. No schema migration.
  • Scope: the seam and the harmonized lifecycle. Order: seam (interface + factory) and folder cascade first — that fixes the reported defect; then controller, auto-start, and per-config control-plane collapse; then ores.shell verbs; then client-side enumeration removal; ingest unification last.
  • All control-plane verbs (start/stop/list, folder, auto-start) are asset-class agnostic by construction. A new asset class plugs in via the factory.
  • The seam's ownership split (task 050DF944): IFeed — identity and lifecycle (source_name, qualifier, role, conflict key, start, stop, publish count) — lives in ores.marketdata.api; producers and the kind-to-builder factory register in ores.synthetic.api (namespace ores::synthetic::feed). The interface, the producer classes and the factory are exported (ORES_*_API_EXPORT): the controllers and the folder cascade dynamic_cast on IFeed across the shared-library boundary, and the IR builder's vintage_data_missing_error is caught by the service, so both need single, unified RTTI.
  • start() takes no callback; feeds publish to NATS themselves. conflict_key() is the feed's own (qualifier, role) join with a unit separator — the IR conflict semantics, expressed by the feed.
  • Folder cascade (task 9E2344CF): the FX pass drops the old ctrl_->start() vintage check (vintage_data_missing skipped) — the cascade aligns with the auto-start path it replaces, and the vintage check stays in the client-supplied-params path until the per-config control-plane task deletes it.
  • Folder cascade permissions: start/stop under folder requires both fx_spot_generation_configs:read and ir_curve_generation_configs:read — the cascade touches both config families, so the gate is uniform, not per-kind.
  • One controller (task 1D756D47): feed_controller absorbs curve_feed_controller (deleted). The conflict rule is uniform across all registration paths (add(), start(IFeed), and the FX ad-hoc start()), via the feed's own conflict_key(), reported with the holding source_name (running_source_name_for_conflict_key()). feeds_conflict covers FX too: two FX feeds on the same pair conflict — the old permissiveness was pre-seam behaviour, and the seed reality (one config per pair per theme) makes the uniform rule safe; the empty-qualifier guard exempts unparseable ORE keys.
  • FX extras stay FX-shaped (task 1D756D47): binding auto-creation and vintage availability live only in the client-supplied-params start() overload (deleted by the per-config control-plane task); binding_mode is stored per running feed purely as that overload's restart gate, so add() keeps the parameter (two overloads) — dropping it would let a later ad-hoc restart create a bound feed_binding for a sandboxed feed. The status loop is uniform: source_name + publish_count per feed.
  • Per-kind list scoping (task 1D756D47, review round): IFeed exposes kind() — the factory kind string, defined in each producer header (fx_spot_feed_kind, ir_curve_feed_kind) — and feed_controller::list(kind) scopes the running snapshot by it. The per-kind control-plane handlers list only their own kind, preserving the pre-collapse list behaviour.
  • Uniform startability (task E1F81BE2): auto_start is a common lifecycle column — the FX config gains it (codegen model + DQ artefact, SQL default false, published via coalesce), the IR config already had it. The gate for auto-start-at-boot is uniform: config enabled + auto_start + enabled container. The IR gate gains the enabled-container requirement; every publish-created container is enabled, so boot behaviour is unchanged. The FX seeds set auto_start=true on every row (all enabled under enabled containers) to keep the boot-started set identical. One auto_start_feeds walk replaces the two per-kind walks.
  • Per-config control-plane (task FD3C0B97): one kind-agnostic start_feed_request{config_id} / stop_feed_request{config_id, source_name} / list_feeds_request{} on synthetic.v1.feed_configs.start|stop|list serves every asset class. Kind resolution probes the per-kind config repositories by config_id — FX first, then IR — since the market_data_generation_config container carries no kind discriminator; the probe is the resolution (FX wins on the invalid row set naming one id in both families). The per-kind control-plane handlers are deleted, validate with them — Qt's Validate Vintage moves to get_vintage_validity, which already serves both kinds.
  • The per-config verbs share the uniform gate and error shape of the rest of the harmonized lifecycle: resolved-kind read permission, config enabled + enabled container, and the same missing/disabled/ already-running/qualifier-conflict (with the holding source_name)/ vintage-data-missing messages for both kinds. List returns every running feed and is gated on both read permissions, like the folder cascade. The FX config path starts with gmm_initial_price as-is — the client-supplied-params vintage check dies with the deleted handler; vintage availability stays queryable via get_vintage_validity.
  • Shell token resolution (task 26D67D72): resolve_feed probes the FX config family first, then the IR curve family — the server's own kind resolution in feed_config_handler probes the repositories in the same order — and returns a resolved_feed{config_id, source_name, optional ore_key}. The shell never knows the feed kind; the request is always keyed by config_id and the server resolves it. The list feeds verb reuses list_feeds_request{} (no message field; failure prints a generic message). validate-vintage stays FX-shaped: vintage validity entries are keyed by fx config id, so only FX feeds can match one; the ore key renders only when present.
  • Client-side enumeration removal (task 60AA1609): the Qt Market Simulator and tenant_handler provisioning issue the folder request only — the per-config IR loop with its client-side auto_start filter is deleted. Folder starts no longer exclude auto_start=false curves client-side: the cascade's uniform behaviour (every feed under the folder, both kinds) applies everywhere, matching the shell. Explicit Feed-leaf and folder-less selections still go through the per-config request.
  • Unified feed ingest (task F550327A): the kind is part of the tick subject — synthetic.v1.tick.<kind>.<source_name>, the kind token being the factory kind string (fx_spot, ir_curve) — so the factory vocabulary and the subject scheme are one and the same. The wire contract (prefixes, tokens, subject builder) lives in the new shared header ores.marketdata.api/domain/tick_subjects.hpp, which both services and the Qt client depend on. One feed_ingest_loop with a single synthetic.v1.tick.> wildcard and kind-token dispatch replaces both loops: the per-source subscribe/unsubscribe lifecycle became a binding cache rebuilt by refresh() (still driven by feed_binding_changed_event); an unbound fx_spot tick is dropped with a one-time warn — the old structural unreachability, now explicit. Republish is uniform — FX semantics applied to every kind (raw payload to the lowercased marketdata.v1.tick.<tenant>.<series> subject), no per-kind flag; CRM stays FX-only. synthetic_curve_ticks is retired by stop-ensuring in both services (ores.nats has no delete-stream API; a stale stream is inert). Payloads unchanged, so every existing decoder and the curve_republish_resolver point-id lookups keep working.

References

Tasks

Task State Start End Description
The feed seam: common producer interface + per-kind factory DONE 2026-08-11 2026-08-11 Build the seam the original FX PoC architecture doc specified (design question 8, never implemented): generalize IFxSpotFeed into a common feed interface (source_name, qualifier, role, start, stop) implemented by both fx_spot_feed and ir_curve_feed, plus a factory mapping kind to producer builder, registered per asset class.
Folder cascade covers every asset class via the factory DONE 2026-08-11 2026-08-11 The start/stop feeds under folder request walks the folder subtree once and starts every feed under it, dispatching through the factory to the per-kind producer. No client-side enumeration. Fixes the reported defect: today the cascade resolves only fx_spot_generation_config rows; the IR pass would have been a third copy of the build sequence.
One lifecycle controller for all feed kinds DONE 2026-08-11 2026-08-11 Collapse feed_controller and curve_feed_controller into one controller: running map keyed by source_name, thread-per-feed, start/stop/list/join. Conflict semantics (IR qualifier plus role) expressed through the feed's own conflict key on the common interface, not a second class. FX extras (feed-binding auto-creation, sandbox subject, status loop) become uniform behaviour.
One auto-start with uniform startability semantics DONE 2026-08-11 2026-08-11 Collapse auto_start_enabled_feeds and auto_start_enabled_ir_curve_feeds into one boot-time walk with uniform startability: enabled, auto_start (added to the FX config for uniform semantics) and an enabled container. Skip-with-warn and running-count reporting become common code.
One per-config control-plane keyed by config_id, server-side resolution DONE 2026-08-11 2026-08-11 One start/stop/list handler for all kinds, following the IR pattern: the client sends config_id, the server resolves config plus children plus refdata context, checks permissions, and starts the producer via the factory. Delete the FX market_feed_config_handler client-supplied-params path (the no-auth anti-pattern).
ores.shell: feed verbs for every asset class DONE 2026-08-11 2026-08-12 ores.shell gains the missing IR curve feed commands and every control-plane verb works regardless of asset class: folder start/stop (one request, server-side cascade) and per-config start/stop/list keyed by config_id. No shell command enumerates asset classes; new asset classes plug in via the factory.
Delete client-side feed enumeration DONE 2026-08-12 2026-08-12 Remove the tree-walk-and-fan-out re-implementations in clients now that the server cascades folders itself: the Qt MarketSimulatorWindow folder branch plus per-config IR loop (and its auto_start client-side filter) and the start_synthetic_theme_feeds two-path provisioning in tenant_handler. Clients send the folder/config request; the server does the rest.
One feed ingest loop over a unified subject scheme DONE 2026-08-12 2026-08-12 Collapse feed_ingest_loop and curve_feed_ingest_loop into one persistence path: the kind is part of the tick subject instead of a second subject family and a second JetStream stream. Uniform republish and observation handling.

Out of scope

  • Schema unification (a shared synthetic_feed base table with a kind discriminator). Decided against: per-kind tables remain; the common shape lives in the seam.
  • The per-kind producer math itself (process classes, tick derivation, series-identity mapping) stays per-kind code behind the common interface — that is the accepted cost.
  • The FOMC bootstrap segment (task FOMC bootstrap segment) is blocked on this story's folder-cascade and control-plane tasks, but its content is not part of this story.

Emacs 29.3 (Org mode 9.6.15)