FX spot synthetic data PoC: architecture

The market data domain is split across three sub-components (see ores.marketdata infrastructure inventory for the full inventory):

@startuml
!include <tupadr3/common>
left to right direction

rectangle "ores.marketdata.api" {
  class market_series
  class market_observation
  class market_fixing
  class protocols <<nats protocol structs>>
}

rectangle "ores.marketdata.core" {
  class market_series_service
  class market_observations_service
  class market_series_repository
  class market_observations_repository
}

rectangle "ores.marketdata.service" {
  class registrar
  note right: lib + exe\nNATS entry point
}

database "TimescaleDB" {
  table ores_marketdata_series
  table ores_marketdata_observations
}

protocols --> market_series_service : NATS req/rep
market_series_service --> market_series_repository
market_observations_service --> market_observations_repository
market_series_repository --> ores_marketdata_series
market_observations_repository --> ores_marketdata_observations
@enduml

Key architectural invariants established by this stack:

• All data scoped by tenant_id; services derive tenant_id from their IAM context.
• market_series is the catalog; market_observation is the time series. Corrections are bi-temporal (insert trigger closes previous row, opens a fresh one).
• All NATS operations are request-reply — no push/fan-out subjects exist for market data today.
• FX spot uses is_scalar=true on its series; observation point_id is always null.
• The full ORE key FX/RATE/EUR/USD decomposes to series_type"FX", =metric"RATE", =qualifier"EUR/USD"=.

ores.qt.mktdata contains:

• MarketDataController — owns the two list windows, wires drill-down signals.
• MarketSeriesMdiWindow — paginated table of all market series, filter by asset class.
• MarketFixingsMdiWindow — fixings series list.

There is no chart display in this plugin today. The onShowMarketObservations signal drills into an observation list window (tabular, not a time-series chart).

ClientManager provides the subscription infrastructure:

subscribeToEvent(subject)     →  internally calls nats::client::subscribe()
                               parses payload as entity_change_event (JSON/rfl)
                               emits notificationReceived(eventType, ts, ids, tenantId, payloadType=0, payload={})

The existing notificationReceived signal carries payloadType (always 0 today) and payload (always empty today). These fields exist for future extension.

The existing pattern is designed for entity-change notifications, not for streaming tick data. The payload fields are unused; the subscriber calls back into a service to fetch the changed entity by ID.

From ores.nats::client:

• publish(subject, data) — fire-and-forget (core NATS, no durability).
• js_publish(subject, data) — durable JetStream publish; blocks on server ACK.
• subscribe(subject, handler) — fan-out subscription; all subscribers receive every message.
• queue_subscribe(subject, group, handler) — competing-consumer; one subscriber per message; used by all server-side handlers.
• request_sync(subject, data) — synchronous request-reply; used by Qt clients.

Each service runs with an IAM identity from the service registry. The IAM context provides ctx.tenant_id() which is used as the tenant_id on all DB records the service writes. The synthetic entry is already registered:

psql_var  = synthetic_service
iam_role  = SyntheticService
email     = synthetic_service@system.ores

One complete tick cycle in sequence:

1. tick_clock fires (interval determined by configured tick mode and parameters).
2. gmm_generator draws a new log-return from the fitted Gaussian mixture and applies it to the current spot price, producing a new value string and the current UTC observation_date.
3. tick_loop sends save_market_observations_request to marketdata.v1.observations.save with a single market_observation (series_id of EUR/USD, observation_date, point_id=null, value, source"SYNTHETIC"=). Awaits acknowledgement.
4. On success, tick_loop serialises an fx_spot_tick (rfl::json) and publishes it to the per-ORE-key subject, e.g. marketdata.v1.tick.fx.rate.eur.usd (core NATS, fire-and-forget — the tick is already durable in the DB).
5. SyntheticFxSpotChartWindow (in the Qt client) receives the callback from its fx_spot_subscription member (ores.marketdata.client), which has already deserialised the payload into fx_spot_tick. The window appends (datetime, mid) to the live chart series.

The DB write (step 3) is the source of truth; the fan-out (step 4) is a convenience notification for live display. A chart client that reconnects after a gap fetches historical observations from marketdata.v1.observations.list to backfill the gap.

Both types live in ores.marketdata.api so they are shared by the service (publisher) and ores.marketdata.client (subscriber) without coupling either to ores.synthetic.

Tick fan-out subjects follow the ORE canonical key mapping defined in Market data identifiers: lowercase the ORE key and replace / with ., then prefix with marketdata.v1.tick..

Feed configuration is a two-step operation: (1) save the market_feed_config entity; (2) save the type-specific params struct linked by config_id. Start and stop are runtime commands on the persisted config ID — they change status only, not config. See Polymorphic types over NATS for the full pattern rationale.

Feed configuration follows the polymorphic types over NATS pattern (see Polymorphic types over NATS): one NATS subject per concrete params type; a discriminator enum on the containing entity identifies which subject to use; no JSON blobs.

Tick subscriptions are handled by the new ores.marketdata.client library, not by Qt code directly. This library is reusable by Qt, Wt, and shell without modification.

// In ores.marketdata.client
class fx_spot_subscription {
public:
    using handler = std::function<void(const fx_spot_tick&)>;

    // ore_key: "FX/RATE/EUR/USD" — converted internally to NATS subject
    fx_spot_subscription(nats::client&, std::string ore_key, handler);
    ~fx_spot_subscription(); // auto-unsubscribes
};

The library handles: NATS subject derivation from ORE key (lowercase, / → .), rfl::json deserialization into fx_spot_tick, and thread marshalling (callbacks delivered on the NATS client thread; callers are responsible for queuing to their own event loop if needed).

The Qt chart window holds an fx_spot_subscription member and re-subscribes on ClientManager::loggedIn and reconnected. It passes clientManager_->nats_client() to the subscription constructor. No direct NATS wiring lives in Qt code.

Proposed structure following the project's three-layer convention:

projects/ores.synthetic/
  service/
    CMakeLists.txt
    include/ores.synthetic.service/
      ...
    src/
      CMakeLists.txt        ← add_library(.lib) + add_executable(.exe)
      main.cpp
      fx_spot_feed.cpp/.hpp         ← FxSpotFeed (thin wrapper over IStochasticProcess)
      processes/
        fixed_process.cpp/.hpp
        ramp_process.cpp/.hpp
        oscillator_process.cpp/.hpp
        sawtooth_process.cpp/.hpp
        step_process.cpp/.hpp
        gbm_process.cpp/.hpp
        ou_process.cpp/.hpp
        gmm_process.cpp/.hpp
        gjr_garch_process.cpp/.hpp
        regime_switching_process.cpp/.hpp
        jump_diffusion_process.cpp/.hpp
        heston_process.cpp/.hpp
      process_factory.cpp/.hpp      ← constructs IStochasticProcess from feed_type + params
      registrar.cpp/.hpp            ← registers NATS handlers for feed config CRUD + start/stop
    tests/

IStochasticProcess lives in ores.marketdata.api (not ores.synthetic) because a future calibration service needs to instantiate processes without depending on the synthetic service executable. All process implementations live in ores.synthetic.service.

For the first pass, a single-layer ores.synthetic.service (no separate ores.synthetic.api or ores.synthetic.core) is sufficient.

The service binary links against:

• ores.marketdata.api.lib — for IStochasticProcess, IFxSpotFeed, fx_spot_tick, and NATS protocol subjects.
• ores.nats.lib — for client::publish and client::queue_subscribe.
• ores.service.lib — for the service lifecycle base.
• ores.iam.core.lib — for IAM context (ctx.tenant_id()).

The tick clock is the configurable time source for the generation loop. Three modes for the PoC:

The tick loop runs on its own std::thread (not the ASIO executor) and uses the NATS client::request_sync method for the DB write (synchronous, blocks until acknowledged). Fan-out publish is fire-and-forget and does not block.

Known PoC limitation — tick clock drift under DB writes: because request_sync blocks per tick, the effective period is sleep_for(period) + DB round-trip. For Fixed mode this causes actual tick rate to fall below target under DB load. For Poisson and regime-switching modes, serialising a blocking call through the drawn inter-arrival time biases the distribution — the true inter-arrival time is longer than drawn. A production implementation should decouple DB writes from the tick clock (separate thread or async write queue); for the PoC, drift is an acceptable limitation.

The approach document specifies GMM for return generation, with parameters fitted to historical returns. For the PoC:

• Parameters are statically seeded (hardcoded in the start request or compiled in).
• The generator maintains a running spot price and draws log-returns from a \(K=3\) Gaussian mixture with pre-set means and standard deviations.
• The returned value is the new spot price formatted to 5 decimal places.
• The standard library std::normal_distribution is sufficient; no external stats library is required for the PoC.

Live GMM calibration to historical data is explicitly out of scope.

The existing market_observation table uses observation_date: year_month_day (date only) as its financial valid-time. The bi-temporal insert trigger matches rows on (series_id, observation_date, point_id): a second tick for the same calendar date closes the previous row and replaces it. At 12 ticks/hour only one observation per day survives in the live view — all earlier intraday ticks are overwritten.

Decision: update the market_observation schema to replace the date field with a full timestamp field so each intraday tick is a distinct financial valid-time point.

Required changes:

The migration is a breaking change to ores.marketdata and must be delivered as a dedicated task ahead of any synthetic service work that writes intraday observations. All other callers of the market_observation API (import service, the Qt observation list window, the report writer) must be updated in the same PR.

Existing imported data (currently stored as EOD dates) can be migrated by converting each observation_date to midnight UTC of that date.